Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##: Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  val DELTA_COL_RESOLVER: (String, String) => Boolean
5.  def addColumn(schema: StructType, column: StructField, position: Seq[Int]): StructType

   Add column to the specified position in schema.

   Add column to the specified position in schema.

   position

   A Seq of ordinals on where this column should go. It is a Seq to denote positions in nested columns (0-based). For example: tableSchema: <a:STRUCT<a1,a2,a3>, b,c:STRUCT<c1,c3>> column: c2 position: Seq(2, 1) will return result: <a:STRUCT<a1,a2,a3>, b,c:STRUCT<c1,**c2**,c3>>

6.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

7.  def canChangeDataType(from: DataType, to: DataType, resolver: Resolver, columnMappingMode: DeltaColumnMappingMode, columnPath: Seq[String] = Seq.empty): Option[String]

   Check if the two data types can be changed.

   Check if the two data types can be changed.

   returns

   None if the data types can be changed, otherwise Some(err) containing the reason.

8.  def changeDataType(from: DataType, to: DataType, resolver: Resolver): DataType

   Copy the nested data type between two data types.

9.  def checkFieldNames(names: Seq[String]): Unit

   Verifies that the column names are acceptable by Parquet and henceforth Delta.

   Verifies that the column names are acceptable by Parquet and henceforth Delta. Parquet doesn't accept the characters ' ,;{}()\n\t='. We ensure that neither the data columns nor the partition columns have these characters.

10.  def checkSchemaFieldNames(schema: StructType, columnMappingMode: DeltaColumnMappingMode): Unit

    Check if the schema contains invalid char in the column names depending on the mode.

11.  def clone(): AnyRef

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.CloneNotSupportedException]) @native()

12.  def containsDependentExpression(spark: SparkSession, columnToChange: Seq[String], exprString: String, resolver: Resolver): Boolean

    Will a column change, e.g., rename, need to be populated to the expression.

    Will a column change, e.g., rename, need to be populated to the expression. This is true when the column to change itself or any of its descendent column is referenced by expression. For example:

    • a, length(a) -> true
    • b, (b.c + 1) -> true, because renaming b1 will need to change the expr to (b1.c + 1).
    • b.c, (cast b as string) -> false, because you can change b.c to b.c1 without affecting b.
13.  def dropColumn(schema: StructType, position: Seq[Int]): (StructType, StructField)

    Drop from the specified position in schema and return with the original column.

    Drop from the specified position in schema and return with the original column.

    position

    A Seq of ordinals on where this column should go. It is a Seq to denote positions in nested columns (0-based). For example: tableSchema: <a:STRUCT<a1,a2,a3>, b,c:STRUCT<c1,c2,c3>> position: Seq(2, 1) will return result: <a:STRUCT<a1,a2,a3>, b,c:STRUCT<c1,c3>>

14.  def dropNullTypeColumns(schema: StructType): StructType

    Drops null types from the schema if they exist.

    Drops null types from the schema if they exist. We do not recurse into Array and Map types, because we do not expect null types to exist in those columns, as Delta doesn't allow it during writes.

15.  def dropNullTypeColumns(df: DataFrame): DataFrame

    Drops null types from the DataFrame if they exist.

    Drops null types from the DataFrame if they exist. We don't have easy ways of generating types such as MapType and ArrayType, therefore if these types contain NullType in their elements, we will throw an AnalysisException.

16.  final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

17.  def equals(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef → Any

18.  def fieldNameToColumn(field: String): Column

    converting field name to column type with quoted back-ticks

19.  def fieldToColumn(field: StructField): Column
20.  def filterRecursively(schema: StructType, checkComplexTypes: Boolean)(f: (StructField) => Boolean): Seq[(Seq[String], StructField)]

    Finds StructFields that match a given check f.

    Finds StructFields that match a given check f. Returns the path to the column, and the field.

    checkComplexTypes

    While StructType is also a complex type, since we're returning StructFields, we definitely recurse into StructTypes. This flag defines whether we should recurse into ArrayType and MapType.

21.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.Throwable])

22.  def findColumnPosition(column: Seq[String], schema: StructType, resolver: Resolver = DELTA_COL_RESOLVER): (Seq[Int], Int)

    Returns the given column's ordinal within the given schema and the size of the last schema size.

    Returns the given column's ordinal within the given schema and the size of the last schema size. The length of the returned position will be as long as how nested the column is.

    For ArrayType: accessing the array's element adds a position 0 to the position list. e.g. accessing a.element.y would have the result -> Seq(..., positionOfA, 0, positionOfY)

    For MapType: accessing the map's key adds a position 0 to the position list. e.g. accessing m.key.y would have the result -> Seq(..., positionOfM, 0, positionOfY)

    For MapType: accessing the map's value adds a position 1 to the position list. e.g. accessing m.key.y would have the result -> Seq(..., positionOfM, 1, positionOfY)

    column

    The column to search for in the given struct. If the length of column is greater than 1, we expect to enter a nested field.

    schema

    The current struct we are looking at.

    resolver

    The resolver to find the column.

23.  def findDependentGeneratedColumns(sparkSession: SparkSession, targetColumn: Seq[String], protocol: Protocol, schema: StructType): Seq[StructField]

    Find all the generated columns that depend on the given target column.

24.  def findNestedFieldIgnoreCase(schema: StructType, fieldNames: Seq[String], includeCollections: Boolean = false): Option[StructField]

    Copied verbatim from Apache Spark.

    Copied verbatim from Apache Spark.

    Returns a field in this struct and its child structs, case insensitively. This is slightly less performant than the case sensitive version.

    If includeCollections is true, this will return fields that are nested in maps and arrays.

    fieldNames

    The path to the field, in order from the root. For example, the column nested.a.b.c would be Seq("nested", "a", "b", "c").

25.  def findNullTypeColumn(schema: StructType): Option[String]

    Returns the name of the first column/field that has null type (void).

26.  def findUndefinedTypes(dt: DataType): Seq[DataType]

    Recursively find all types not defined in Delta protocol but used in dt

27.  def findUnsupportedDataTypes(schema: StructType): Seq[UnsupportedDataTypeInfo]

    Find the unsupported data type in a table schema.

    Find the unsupported data type in a table schema. Return all columns that are using unsupported data types. For example, findUnsupportedDataType(struct<a: struct<b: unsupported_type>>) will return Some(unsupported_type, Some("a.b")).

28.  final def getClass(): Class[_ <: AnyRef]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

29.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

30.  def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Attributes

    protected

    Definition Classes

    Logging

31.  def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes

    protected

    Definition Classes

    Logging

32.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

33.  def isReadCompatible(existingSchema: StructType, readSchema: StructType): Boolean

    As the Delta snapshots update, the schema may change as well.

    As the Delta snapshots update, the schema may change as well. This method defines whether the new schema of a Delta table can be used with a previously analyzed LogicalPlan. Our rules are to return false if:

    • Dropping any column that was present in the DataFrame schema
    • Converting nullable=false to nullable=true for any column
    • Any change of datatype
34.  def isTraceEnabled(): Boolean

    Attributes

    protected

    Definition Classes

    Logging

35.  def log: Logger

    Attributes

    protected

    Definition Classes

    Logging

36.  def logConsole(line: String): Unit

    Definition Classes

    DatabricksLogging

37.  def logDebug(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

38.  def logDebug(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

39.  def logError(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

40.  def logError(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

41.  def logInfo(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

42.  def logInfo(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

43.  def logName: String

    Attributes

    protected

    Definition Classes

    Logging

44.  def logTrace(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

45.  def logTrace(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

46.  def logWarning(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

47.  def logWarning(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

48.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

49.  def normalizeColumnNames(baseSchema: StructType, data: Dataset[_]): DataFrame

    Rewrite the query field names according to the table schema.

    Rewrite the query field names according to the table schema. This method assumes that all schema validation checks have been made and this is the last operation before writing into Delta.

50.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

51.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

52.  def prettyFieldName(columnPath: Seq[String]): String

    Pretty print the column path passed in.

53.  def quoteIdentifier(part: String): String
54.  def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    path

    Used to log the path of the delta table when deltaLog is null.

    Attributes

    protected

    Definition Classes

    DeltaLogging

55.  def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Attributes

    protected

    Definition Classes

    DeltaLogging

56.  def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Attributes

    protected

    Definition Classes

    DeltaLogging

57.  def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit

    Definition Classes

    DatabricksLogging

58.  def recordFrameProfile[T](group: String, name: String)(thunk: => T): T

    Attributes

    protected

    Definition Classes

    DeltaLogging

59.  def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = null, silent: Boolean = true)(thunk: => S): S

    Definition Classes

    DatabricksLogging

60.  def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit

    Definition Classes

    DatabricksLogging

61.  def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit

    Definition Classes

    DatabricksLogging

62.  def recordUndefinedTypes(deltaLog: DeltaLog, schema: StructType): Unit

    Record all types not defined in Delta protocol but used in the schema.

63.  def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit

    Definition Classes

    DatabricksLogging

64.  def removeUnenforceableNotNullConstraints(schema: StructType, conf: SQLConf): StructType

    Go through the schema to look for unenforceable NOT NULL constraints.

    Go through the schema to look for unenforceable NOT NULL constraints. By default we'll throw when they're encountered, but if this is suppressed through SQLConf they'll just be silently removed.

    Note that this should only be applied to schemas created from explicit user DDL - in other scenarios, the nullability information may be inaccurate and Delta should always coerce the nullability flag to true.

65.  def reportDifferences(existingSchema: StructType, specifiedSchema: StructType): Seq[String]

    Compare an existing schema to a specified new schema and return a message describing the first difference found, if any:

    Compare an existing schema to a specified new schema and return a message describing the first difference found, if any:

    • different field name or datatype
    • different metadata
66.  final def synchronized[T0](arg0: => T0): T0

    Definition Classes

    AnyRef

67.  def toString(): String

    Definition Classes

    AnyRef → Any

68.  def transformColumns[E](schema: StructType, input: Seq[(Seq[String], E)])(tf: (Seq[String], StructField, Seq[(Seq[String], E)]) => StructField): StructType

    Transform (nested) columns in a schema using the given path and parameter pairs.

    Transform (nested) columns in a schema using the given path and parameter pairs. The transform function is only invoked when a field's path matches one of the input paths.

    E

    the type of the payload used for transforming fields.

    schema

    to transform

    input

    paths and parameter pairs. The paths point to fields we want to transform. The parameters will be passed to the transform function for a matching field.

    tf

    function to apply per matched field. This function takes the field path, the field itself and the input names and payload pairs that matched the field name. It should return a new field.

    returns

    the transformed schema.

69.  def transformColumnsStructs(schema: StructType, colName: Option[String] = None)(tf: (Seq[String], StructType, Resolver) => Seq[StructField]): StructType

    Transform (nested) columns in a schema.

    Transform (nested) columns in a schema. Runs the transform function on all nested StructTypes

    If colName is defined, we also check if the struct to process contains the column name.

    schema

    to transform.

    colName

    Optional name to match for

    tf

    function to apply on the StructType.

    returns

    the transformed schema.

70.  def typeAsNullable(dt: DataType): DataType

    Turns the data types to nullable in a recursive manner for nested columns.

71.  def typeExistsRecursively(dt: DataType)(f: (DataType) => Boolean): Boolean

    Copied over from DataType for visibility reasons.

72.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

73.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

74.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException]) @native()

75.  def withDmqTag[T](thunk: => T): T

    Attributes

    protected

    Definition Classes

    DeltaLogging

76.  def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: => T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes

    DeltaProgressReporter