Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##: Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  val COLUMN_MAPPING_METADATA_ID_KEY: String
5.  val COLUMN_MAPPING_METADATA_PREFIX: String
6.  val COLUMN_MAPPING_PHYSICAL_NAME_KEY: String
7.  val DELTA_INTERNAL_COLUMNS: Set[String]

   This list of internal columns (and only this list) is allowed to have missing column mapping metadata such as field id and physical name because they might not be present in user's table schema.

   This list of internal columns (and only this list) is allowed to have missing column mapping metadata such as field id and physical name because they might not be present in user's table schema.

   These fields, if materialized to parquet, will always be matched by their display name in the downstream parquet reader even under column mapping modes.

   For future developers who want to utilize additional internal columns without generating column mapping metadata, please add them here.

   This list is case-insensitive.

   Attributes

   protected

8.  val MIN_PROTOCOL_VERSION: Protocol
9.  val MIN_READER_VERSION: Int
10.  val MIN_WRITER_VERSION: Int
11.  val PARQUET_FIELD_ID_METADATA_KEY: String
12.  final def asInstanceOf[T0]: T0

    Definition Classes

    Any

13.  def assignColumnIdAndPhysicalName(newMetadata: Metadata, oldMetadata: Metadata, isChangingModeOnExistingTable: Boolean): Metadata

    For each column/field in a Metadata's schema, assign id using the current maximum id as the basis and increment from there, and assign physical name using UUID

    For each column/field in a Metadata's schema, assign id using the current maximum id as the basis and increment from there, and assign physical name using UUID

    newMetadata

    The new metadata to assign Ids and physical names

    oldMetadata

    The old metadata

    isChangingModeOnExistingTable

    whether this is part of a commit that changes the mapping mode on a existing table

    returns

    new metadata with Ids and physical names assigned

14.  def assignPhysicalNames(schema: StructType): StructType
15.  def checkColumnIdAndPhysicalNameAssignments(schema: StructType, mode: DeltaColumnMappingMode): Unit
16.  def clone(): AnyRef

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.CloneNotSupportedException]) @native()

17.  def createPhysicalAttributes(output: Seq[Attribute], referenceSchema: StructType, columnMappingMode: DeltaColumnMappingMode): Seq[Attribute]

    Create a list of physical attributes for the given attributes using the table schema as a reference.

    Create a list of physical attributes for the given attributes using the table schema as a reference.

    output

    the list of attributes (potentially without any metadata)

    referenceSchema

    the table schema with all the metadata

    columnMappingMode

    column mapping mode of the delta table, which determines which metadata to fill in

18.  def createPhysicalSchema(schema: StructType, referenceSchema: StructType, columnMappingMode: DeltaColumnMappingMode, checkSupportedMode: Boolean = true): StructType

    Create a physical schema for the given schema using the Delta table schema as a reference.

    Create a physical schema for the given schema using the Delta table schema as a reference.

    schema

    the given logical schema (potentially without any metadata)

    referenceSchema

    the schema from the delta log, which has all the metadata

    columnMappingMode

    column mapping mode of the delta table, which determines which metadata to fill in

    checkSupportedMode

    whether we should check of the column mapping mode is supported

19.  def dropColumnMappingMetadata(schema: StructType): StructType
20.  final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

21.  def equals(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef → Any

22.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.Throwable])

23.  def findMaxColumnId(schema: StructType): Long
24.  def generatePhysicalName: String
25.  final def getClass(): Class[_ <: AnyRef]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

26.  def getColumnId(field: StructField): Int
27.  def getColumnMappingMetadata(field: StructField, mode: DeltaColumnMappingMode): Metadata

    Gets the required column metadata for each column based on the column mapping mode.

28.  def getPhysicalName(field: StructField): String
29.  def getPhysicalNameFieldMap(schema: StructType): Map[Seq[String], StructField]

    Returns a map of physicalNamePath -> field for the given schema, where physicalNamePath is the [$parentPhysicalName, ..., $fieldPhysicalName] list of physical names for every field (including nested) in the schema.

    Returns a map of physicalNamePath -> field for the given schema, where physicalNamePath is the [$parentPhysicalName, ..., $fieldPhysicalName] list of physical names for every field (including nested) in the schema.

    Must be called after checkColumnIdAndPhysicalNameAssignments, so that we know the schema is valid.

30.  def hasColumnId(field: StructField): Boolean
31.  def hasPhysicalName(field: StructField): Boolean
32.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

33.  def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Attributes

    protected

    Definition Classes

    Logging

34.  def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes

    protected

    Definition Classes

    Logging

35.  def isColumnMappingUpgrade(oldMode: DeltaColumnMappingMode, newMode: DeltaColumnMappingMode): Boolean
36.  def isDropColumnOperation(newMetadata: Metadata, currentMetadata: Metadata): Boolean

    Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a DROP COLUMN operation.

    Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a DROP COLUMN operation.

    We detect DROP COLUMNS by checking if any physical name in currentSchema is missing in newSchema.

37.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

38.  def isInternalField(field: StructField): Boolean
39.  def isRenameColumnOperation(newMetadata: Metadata, currentMetadata: Metadata): Boolean

    Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a RENAME COLUMN operation.

    Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a RENAME COLUMN operation.

    We detect RENAME COLUMNS by checking if any two columns with the same physical name have different logical names

40.  def isTraceEnabled(): Boolean

    Attributes

    protected

    Definition Classes

    Logging

41.  def log: Logger

    Attributes

    protected

    Definition Classes

    Logging

42.  def logConsole(line: String): Unit

    Definition Classes

    DatabricksLogging

43.  def logDebug(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

44.  def logDebug(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

45.  def logError(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

46.  def logError(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

47.  def logInfo(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

48.  def logInfo(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

49.  def logName: String

    Attributes

    protected

    Definition Classes

    Logging

50.  def logTrace(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

51.  def logTrace(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

52.  def logWarning(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

53.  def logWarning(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

54.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

55.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

56.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

57.  def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    path

    Used to log the path of the delta table when deltaLog is null.

    Attributes

    protected

    Definition Classes

    DeltaLogging

58.  def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Attributes

    protected

    Definition Classes

    DeltaLogging

59.  def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Attributes

    protected

    Definition Classes

    DeltaLogging

60.  def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit

    Definition Classes

    DatabricksLogging

61.  def recordFrameProfile[T](group: String, name: String)(thunk: => T): T

    Attributes

    protected

    Definition Classes

    DeltaLogging

62.  def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = null, silent: Boolean = true)(thunk: => S): S

    Definition Classes

    DatabricksLogging

63.  def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit

    Definition Classes

    DatabricksLogging

64.  def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit

    Definition Classes

    DatabricksLogging

65.  def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit

    Definition Classes

    DatabricksLogging

66.  def renameColumns(schema: StructType): StructType

    Recursively renames columns in the given schema with their physical schema.

67.  def requiresNewProtocol(metadata: Metadata): Boolean
68.  def satisfyColumnMappingProtocol(protocol: Protocol): Boolean
69.  def setColumnMetadata(schema: StructType, mode: DeltaColumnMappingMode): StructType

    Prepares the table schema, to be used by the readers and writers of the table.

    Prepares the table schema, to be used by the readers and writers of the table.

    In the new Delta protocol that supports column mapping, we persist various column mapping metadata in the serialized schema of the Delta log. This method performs the necessary transformation and filtering on these metadata based on the column mapping mode set for the table.

    schema

    the raw schema directly deserialized from the Delta log, with various column mapping metadata.

    mode

    column mapping mode of the table

    returns

    the table schema for the readers and writers. Columns will need to be renamed by using the renameColumns function.

70.  val supportedModes: Set[DeltaColumnMappingMode]
71.  final def synchronized[T0](arg0: => T0): T0

    Definition Classes

    AnyRef

72.  def toString(): String

    Definition Classes

    AnyRef → Any

73.  def tryFixMetadata(oldMetadata: Metadata, newMetadata: Metadata, isChangingModeOnExistingTable: Boolean): Metadata
74.  def verifyAndUpdateMetadataChange(deltaLog: DeltaLog, oldProtocol: Protocol, oldMetadata: Metadata, newMetadata: Metadata, isCreatingNewTable: Boolean): Metadata

    If the table is already on the column mapping protocol, we block:

    If the table is already on the column mapping protocol, we block:

    • changing column mapping config otherwise, we block
    • upgrading to the column mapping Protocol through configurations
75.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

76.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

77.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException]) @native()

78.  def withDmqTag[T](thunk: => T): T

    Attributes

    protected

    Definition Classes

    DeltaLogging

79.  def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: => T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes

    DeltaProgressReporter