object DeltaColumnMapping extends DeltaColumnMappingBase
- Alphabetic
- By Inheritance
- DeltaColumnMapping
- DeltaColumnMappingBase
- DeltaLogging
- DatabricksLogging
- DeltaProgressReporter
- Logging
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- val COLUMN_MAPPING_METADATA_ID_KEY: String
- Definition Classes
- DeltaColumnMappingBase
- val COLUMN_MAPPING_METADATA_PREFIX: String
- Definition Classes
- DeltaColumnMappingBase
- val COLUMN_MAPPING_PHYSICAL_NAME_KEY: String
- Definition Classes
- DeltaColumnMappingBase
- val DELTA_INTERNAL_COLUMNS: Set[String]
This list of internal columns (and only this list) is allowed to have missing column mapping metadata such as field id and physical name because they might not be present in user's table schema.
This list of internal columns (and only this list) is allowed to have missing column mapping metadata such as field id and physical name because they might not be present in user's table schema.
These fields, if materialized to parquet, will always be matched by their display name in the downstream parquet reader even under column mapping modes.
For future developers who want to utilize additional internal columns without generating column mapping metadata, please add them here.
This list is case-insensitive.
- Attributes
- protected
- Definition Classes
- DeltaColumnMappingBase
- val MIN_PROTOCOL_VERSION: Protocol
- Definition Classes
- DeltaColumnMappingBase
- val MIN_READER_VERSION: Int
- Definition Classes
- DeltaColumnMappingBase
- val MIN_WRITER_VERSION: Int
- Definition Classes
- DeltaColumnMappingBase
- val PARQUET_FIELD_ID_METADATA_KEY: String
- Definition Classes
- DeltaColumnMappingBase
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def assignColumnIdAndPhysicalName(newMetadata: Metadata, oldMetadata: Metadata, isChangingModeOnExistingTable: Boolean): Metadata
For each column/field in a Metadata's schema, assign id using the current maximum id as the basis and increment from there, and assign physical name using UUID
For each column/field in a Metadata's schema, assign id using the current maximum id as the basis and increment from there, and assign physical name using UUID
- newMetadata
The new metadata to assign Ids and physical names
- oldMetadata
The old metadata
- isChangingModeOnExistingTable
whether this is part of a commit that changes the mapping mode on a existing table
- returns
new metadata with Ids and physical names assigned
- Definition Classes
- DeltaColumnMappingBase
- def assignPhysicalNames(schema: StructType): StructType
- Definition Classes
- DeltaColumnMappingBase
- def checkColumnIdAndPhysicalNameAssignments(schema: StructType, mode: DeltaColumnMappingMode): Unit
- Definition Classes
- DeltaColumnMappingBase
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- def createPhysicalAttributes(output: Seq[Attribute], referenceSchema: StructType, columnMappingMode: DeltaColumnMappingMode): Seq[Attribute]
Create a list of physical attributes for the given attributes using the table schema as a reference.
Create a list of physical attributes for the given attributes using the table schema as a reference.
- output
the list of attributes (potentially without any metadata)
- referenceSchema
the table schema with all the metadata
- columnMappingMode
column mapping mode of the delta table, which determines which metadata to fill in
- Definition Classes
- DeltaColumnMappingBase
- def createPhysicalSchema(schema: StructType, referenceSchema: StructType, columnMappingMode: DeltaColumnMappingMode, checkSupportedMode: Boolean = true): StructType
Create a physical schema for the given schema using the Delta table schema as a reference.
Create a physical schema for the given schema using the Delta table schema as a reference.
- schema
the given logical schema (potentially without any metadata)
- referenceSchema
the schema from the delta log, which has all the metadata
- columnMappingMode
column mapping mode of the delta table, which determines which metadata to fill in
- checkSupportedMode
whether we should check of the column mapping mode is supported
- Definition Classes
- DeltaColumnMappingBase
- def dropColumnMappingMetadata(schema: StructType): StructType
- Definition Classes
- DeltaColumnMappingBase
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- def findMaxColumnId(schema: StructType): Long
- Definition Classes
- DeltaColumnMappingBase
- def generatePhysicalName: String
- Definition Classes
- DeltaColumnMappingBase
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def getColumnId(field: StructField): Int
- Definition Classes
- DeltaColumnMappingBase
- def getColumnMappingMetadata(field: StructField, mode: DeltaColumnMappingMode): Metadata
Gets the required column metadata for each column based on the column mapping mode.
Gets the required column metadata for each column based on the column mapping mode.
- Definition Classes
- DeltaColumnMappingBase
- def getPhysicalName(field: StructField): String
- Definition Classes
- DeltaColumnMappingBase
- def getPhysicalNameFieldMap(schema: StructType): Map[Seq[String], StructField]
Returns a map of physicalNamePath -> field for the given
schema, where physicalNamePath is the [$parentPhysicalName, ..., $fieldPhysicalName] list of physical names for every field (including nested) in theschema.Returns a map of physicalNamePath -> field for the given
schema, where physicalNamePath is the [$parentPhysicalName, ..., $fieldPhysicalName] list of physical names for every field (including nested) in theschema.Must be called after
checkColumnIdAndPhysicalNameAssignments, so that we know the schema is valid.- Definition Classes
- DeltaColumnMappingBase
- def hasColumnId(field: StructField): Boolean
- Definition Classes
- DeltaColumnMappingBase
- def hasPhysicalName(field: StructField): Boolean
- Definition Classes
- DeltaColumnMappingBase
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def isColumnMappingUpgrade(oldMode: DeltaColumnMappingMode, newMode: DeltaColumnMappingMode): Boolean
- Definition Classes
- DeltaColumnMappingBase
- def isDropColumnOperation(newMetadata: Metadata, currentMetadata: Metadata): Boolean
Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a DROP COLUMN operation.
Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a DROP COLUMN operation.
We detect DROP COLUMNS by checking if any physical name in
currentSchemais missing innewSchema.- Definition Classes
- DeltaColumnMappingBase
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isInternalField(field: StructField): Boolean
- Definition Classes
- DeltaColumnMappingBase
- def isRenameColumnOperation(newMetadata: Metadata, currentMetadata: Metadata): Boolean
Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a RENAME COLUMN operation.
Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a RENAME COLUMN operation.
We detect RENAME COLUMNS by checking if any two columns with the same physical name have different logical names
- Definition Classes
- DeltaColumnMappingBase
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logConsole(line: String): Unit
- Definition Classes
- DatabricksLogging
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit
Used to record the occurrence of a single event or report detailed, operation specific statistics.
Used to record the occurrence of a single event or report detailed, operation specific statistics.
- path
Used to log the path of the delta table when
deltaLogis null.
- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A
Used to report the duration as well as the success or failure of an operation on a
deltaLog.Used to report the duration as well as the success or failure of an operation on a
deltaLog.- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A
Used to report the duration as well as the success or failure of an operation on a
tahoePath.Used to report the duration as well as the success or failure of an operation on a
tahoePath.- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
- Definition Classes
- DatabricksLogging
- def recordFrameProfile[T](group: String, name: String)(thunk: => T): T
- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = null, silent: Boolean = true)(thunk: => S): S
- Definition Classes
- DatabricksLogging
- def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
- Definition Classes
- DatabricksLogging
- def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
- Definition Classes
- DatabricksLogging
- def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
- Definition Classes
- DatabricksLogging
- def renameColumns(schema: StructType): StructType
Recursively renames columns in the given schema with their physical schema.
Recursively renames columns in the given schema with their physical schema.
- Definition Classes
- DeltaColumnMappingBase
- def requiresNewProtocol(metadata: Metadata): Boolean
- Definition Classes
- DeltaColumnMappingBase
- def satisfyColumnMappingProtocol(protocol: Protocol): Boolean
- Definition Classes
- DeltaColumnMappingBase
- def setColumnMetadata(schema: StructType, mode: DeltaColumnMappingMode): StructType
Prepares the table schema, to be used by the readers and writers of the table.
Prepares the table schema, to be used by the readers and writers of the table.
In the new Delta protocol that supports column mapping, we persist various column mapping metadata in the serialized schema of the Delta log. This method performs the necessary transformation and filtering on these metadata based on the column mapping mode set for the table.
- schema
the raw schema directly deserialized from the Delta log, with various column mapping metadata.
- mode
column mapping mode of the table
- returns
the table schema for the readers and writers. Columns will need to be renamed by using the
renameColumnsfunction.
- Definition Classes
- DeltaColumnMappingBase
- val supportedModes: Set[DeltaColumnMappingMode]
- Definition Classes
- DeltaColumnMappingBase
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- def tryFixMetadata(oldMetadata: Metadata, newMetadata: Metadata, isChangingModeOnExistingTable: Boolean): Metadata
- Definition Classes
- DeltaColumnMappingBase
- def verifyAndUpdateMetadataChange(deltaLog: DeltaLog, oldProtocol: Protocol, oldMetadata: Metadata, newMetadata: Metadata, isCreatingNewTable: Boolean): Metadata
If the table is already on the column mapping protocol, we block:
If the table is already on the column mapping protocol, we block:
- changing column mapping config otherwise, we block
- upgrading to the column mapping Protocol through configurations
- Definition Classes
- DeltaColumnMappingBase
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- def withDmqTag[T](thunk: => T): T
- Attributes
- protected
- Definition Classes
- DeltaLogging
- def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: => T): T
Report a log to indicate some command is running.
Report a log to indicate some command is running.
- Definition Classes
- DeltaProgressReporter