object DeltaConfigs extends DeltaConfigsBase
- Alphabetic
- By Inheritance
- DeltaConfigs
- DeltaConfigsBase
- DeltaLogging
- DatabricksLogging
- DeltaProgressReporter
- Logging
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- val AUTO_OPTIMIZE: DeltaConfig[Option[Boolean]]
Whether this table will automatically optimize the layout of files during writes.
Whether this table will automatically optimize the layout of files during writes.
- Definition Classes
- DeltaConfigsBase
- val CHANGE_DATA_FEED: DeltaConfig[Boolean]
Enable change data feed output.
Enable change data feed output. When enabled, DELETE, UPDATE, and MERGE INTO operations will need to do additional work to output their change data in an efficiently readable format.
- Definition Classes
- DeltaConfigsBase
- val CHECKPOINT_INTERVAL: DeltaConfig[Int]
How often to checkpoint the delta log.
How often to checkpoint the delta log.
- Definition Classes
- DeltaConfigsBase
- val CHECKPOINT_RETENTION_DURATION: DeltaConfig[CalendarInterval]
The shortest duration we have to keep checkpoint files around before deleting them.
The shortest duration we have to keep checkpoint files around before deleting them. Note that we'll never delete the most recent checkpoint. We may keep checkpoint files beyond this duration until the next calendar day.
- Definition Classes
- DeltaConfigsBase
- val CHECKPOINT_WRITE_STATS_AS_JSON: DeltaConfig[Boolean]
When enabled, we will write file statistics in the checkpoint in JSON format as the "stats" column.
When enabled, we will write file statistics in the checkpoint in JSON format as the "stats" column.
- Definition Classes
- DeltaConfigsBase
- val CHECKPOINT_WRITE_STATS_AS_STRUCT: DeltaConfig[Option[Boolean]]
When enabled, we will write file statistics in the checkpoint in the struct format in the "stats_parsed" column.
When enabled, we will write file statistics in the checkpoint in the struct format in the "stats_parsed" column. We will also write partition values as a struct as "partitionValues_parsed".
- Definition Classes
- DeltaConfigsBase
- val COLUMN_MAPPING_MAX_ID: DeltaConfig[Long]
Maximum columnId used in the schema so far for column mapping.
Maximum columnId used in the schema so far for column mapping. Internal property that cannot be set by users.
- Definition Classes
- DeltaConfigsBase
- val COLUMN_MAPPING_MODE: DeltaConfig[DeltaColumnMappingMode]
- Definition Classes
- DeltaConfigsBase
- val DATA_SKIPPING_NUM_INDEXED_COLS: DeltaConfig[Int]
The number of columns to collect stats on for data skipping.
The number of columns to collect stats on for data skipping. A value of -1 means collecting stats for all columns. Updating this conf does not trigger stats re-collection, but redefines the stats schema of table, i.e., it will change the behavior of future stats collection (e.g., in append and OPTIMIZE) as well as data skipping (e.g., the column stats beyond this number will be ignored even when they exist).
- Definition Classes
- DeltaConfigsBase
- val ENABLE_EXPIRED_LOG_CLEANUP: DeltaConfig[Boolean]
Whether to clean up expired checkpoints and delta logs.
Whether to clean up expired checkpoints and delta logs.
- Definition Classes
- DeltaConfigsBase
- val ENABLE_FULL_RETENTION_ROLLBACK: DeltaConfig[Boolean]
If true, a delta table can be rolled back to any point within LOG_RETENTION.
If true, a delta table can be rolled back to any point within LOG_RETENTION. Leaving this on requires converting the oldest delta file we have into a checkpoint, which we do once a day. If doing that operation is too expensive, it can be turned off, but the table can only be rolled back CHECKPOINT_RETENTION_DURATION ago instead of LOG_RETENTION ago.
- Definition Classes
- DeltaConfigsBase
- val IS_APPEND_ONLY: DeltaConfig[Boolean]
Whether this Delta table is append-only.
Whether this Delta table is append-only. Files can't be deleted, or values can't be updated.
- Definition Classes
- DeltaConfigsBase
- val LOG_RETENTION: DeltaConfig[CalendarInterval]
The shortest duration we have to keep delta files around before deleting them.
The shortest duration we have to keep delta files around before deleting them. We can only delete delta files that are before a compaction. We may keep files beyond this duration until the next calendar day.
- Definition Classes
- DeltaConfigsBase
- val MIN_READER_VERSION: DeltaConfig[Int]
The protocol reader version modelled as a table property.
The protocol reader version modelled as a table property. This property is *not* stored as a table property in the
Metadataaction. It is stored as its own action. Having it modelled as a table property makes it easier to upgrade, and view the version.- Definition Classes
- DeltaConfigsBase
- val MIN_WRITER_VERSION: DeltaConfig[Int]
The protocol reader version modelled as a table property.
The protocol reader version modelled as a table property. This property is *not* stored as a table property in the
Metadataaction. It is stored as its own action. Having it modelled as a table property makes it easier to upgrade, and view the version.- Definition Classes
- DeltaConfigsBase
- val RANDOMIZE_FILE_PREFIXES: DeltaConfig[Boolean]
Whether to use a random prefix in a file path instead of partition information.
Whether to use a random prefix in a file path instead of partition information. This is required for very high volume S3 calls to better be partitioned across S3 servers.
- Definition Classes
- DeltaConfigsBase
- val RANDOM_PREFIX_LENGTH: DeltaConfig[Int]
Whether to use a random prefix in a file path instead of partition information.
Whether to use a random prefix in a file path instead of partition information. This is required for very high volume S3 calls to better be partitioned across S3 servers.
- Definition Classes
- DeltaConfigsBase
- val SAMPLE_RETENTION: DeltaConfig[CalendarInterval]
The shortest duration we have to keep delta sample files around before deleting them.
The shortest duration we have to keep delta sample files around before deleting them.
- Definition Classes
- DeltaConfigsBase
- val SYMLINK_FORMAT_MANIFEST_ENABLED: DeltaConfig[Boolean]
- Definition Classes
- DeltaConfigsBase
- val TOMBSTONE_RETENTION: DeltaConfig[CalendarInterval]
The shortest duration we have to keep logically deleted data files around before deleting them physically.
The shortest duration we have to keep logically deleted data files around before deleting them physically. This is to prevent failures in stale readers after compactions or partition overwrites.
Note: this value should be large enough: - It should be larger than the longest possible duration of a job if you decide to run "VACUUM" when there are concurrent readers or writers accessing the table. - If you are running a streaming query reading from the table, you should make sure the query doesn't stop longer than this value. Otherwise, the query may not be able to restart as it still needs to read old files.
- Definition Classes
- DeltaConfigsBase
- val TRANSACTION_ID_RETENTION_DURATION: DeltaConfig[Option[CalendarInterval]]
The shortest duration within which new Snapshots will retain transaction identifiers (i.e.
The shortest duration within which new Snapshots will retain transaction identifiers (i.e. SetTransactions). When a new Snapshot sees a transaction identifier older than or equal to the specified TRANSACTION_ID_RETENTION_DURATION, it considers it expired and ignores it.
- Definition Classes
- DeltaConfigsBase
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def buildConfig[T](key: String, defaultValue: String, fromString: (String) => T, validationFunction: (T) => Boolean, helpMessage: String, minimumProtocolVersion: Option[Protocol] = None, userConfigurable: Boolean = true, alternateConfs: Seq[DeltaConfig[T]] = Seq.empty): DeltaConfig[T]
- Attributes
- protected
- Definition Classes
- DeltaConfigsBase
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def getMilliSeconds(i: CalendarInterval): Long
- Definition Classes
- DeltaConfigsBase
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def isValidIntervalConfigValue(i: CalendarInterval): Boolean
For configs accepting an interval, we require the user specified string must obey:
For configs accepting an interval, we require the user specified string must obey:
- Doesn't use months or years, since an internal like this is not deterministic. - The microseconds parsed from the string value must be a non-negative value.
The method returns whether a CalendarInterval satisfies the requirements.
- Definition Classes
- DeltaConfigsBase
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logConsole(line: String): Unit
- Definition Classes
- DatabricksLogging
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def mergeGlobalConfigs(sqlConfs: SQLConf, tableConf: Map[String, String]): Map[String, String]
Table properties for new tables can be specified through SQL Configurations using the
sqlConfPrefix.Table properties for new tables can be specified through SQL Configurations using the
sqlConfPrefix. This method checks to see if any of the configurations exist among the SQL configurations and merges them with the user provided configurations. User provided configs take precedence.- Definition Classes
- DeltaConfigsBase
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def normalizeConfigKey(propKey: Option[String]): Option[String]
Normalize the specified property key if the key is for a Delta config.
Normalize the specified property key if the key is for a Delta config.
- Definition Classes
- DeltaConfigsBase
- def normalizeConfigKeys(propKeys: Seq[String]): Seq[String]
Normalize the specified property keys if the key is for a Delta config.
Normalize the specified property keys if the key is for a Delta config.
- Definition Classes
- DeltaConfigsBase
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def parseCalendarInterval(s: String): CalendarInterval
Convert a string to CalendarInterval.
Convert a string to CalendarInterval. This method is case-insensitive and will throw IllegalArgumentException when the input string is not a valid interval.
TODO Remove this method and use
CalendarInterval.fromCaseInsensitiveStringinstead when upgrading Spark. This is a fork version ofCalendarInterval.fromCaseInsensitiveStringwhich will be available in the next Spark release (See SPARK-27735).- Definition Classes
- DeltaConfigsBase
- Exceptions thrown
IllegalArgumentExceptionif the string is not a valid internal.
- def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit
Used to record the occurrence of a single event or report detailed, operation specific statistics.
Used to record the occurrence of a single event or report detailed, operation specific statistics.
- path
Used to log the path of the delta table when
deltaLogis null.
- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A
Used to report the duration as well as the success or failure of an operation on a
deltaLog.Used to report the duration as well as the success or failure of an operation on a
deltaLog.- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A
Used to report the duration as well as the success or failure of an operation on a
tahoePath.Used to report the duration as well as the success or failure of an operation on a
tahoePath.- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
- Definition Classes
- DatabricksLogging
- def recordFrameProfile[T](group: String, name: String)(thunk: => T): T
- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = null, silent: Boolean = true)(thunk: => S): S
- Definition Classes
- DatabricksLogging
- def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
- Definition Classes
- DatabricksLogging
- def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
- Definition Classes
- DatabricksLogging
- def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
- Definition Classes
- DatabricksLogging
- val sqlConfPrefix: String
A global default value set as a SQLConf will overwrite the default value of a DeltaConfig.
A global default value set as a SQLConf will overwrite the default value of a DeltaConfig. For example, user can run: set spark.databricks.delta.properties.defaults.randomPrefixLength = 5 This setting will be populated to a Delta table during its creation time and overwrites the default value of delta.randomPrefixLength.
We accept these SQLConfs as strings and only perform validation in DeltaConfig. All the DeltaConfigs set in SQLConf should adopt the same prefix.
- Definition Classes
- DeltaConfigsBase
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- def validateConfigurations(configurations: Map[String, String]): Map[String, String]
Validates specified configurations and returns the normalized key -> value map.
Validates specified configurations and returns the normalized key -> value map.
- Definition Classes
- DeltaConfigsBase
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- def withDmqTag[T](thunk: => T): T
- Attributes
- protected
- Definition Classes
- DeltaLogging
- def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: => T): T
Report a log to indicate some command is running.
Report a log to indicate some command is running.
- Definition Classes
- DeltaProgressReporter