Packages

t

org.apache.spark.sql.delta

UpdateExpressionsSupport

trait UpdateExpressionsSupport extends CastSupport with SQLConfHelper with AnalysisHelper

Trait with helper functions to generate expressions to update target columns, even if they are nested fields.

Linear Supertypes
AnalysisHelper, SQLConfHelper, CastSupport, AnyRef, Any
Known Subclasses
PreprocessTableMerge, PreprocessTableUpdate
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. UpdateExpressionsSupport
  2. AnalysisHelper
  3. SQLConfHelper
  4. CastSupport
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. case class UpdateOperation(targetColNameParts: Seq[String], updateExpr: Expression) extends Product with Serializable

    Specifies an operation that updates a target column with the given expression.

    Specifies an operation that updates a target column with the given expression. The target column may or may not be a nested field and it is specified as a full quoted name or as a sequence of split into parts.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def cast(child: Expression, dataType: DataType): Cast
    Definition Classes
    CastSupport
  6. def castIfNeeded(fromExpression: Expression, dataType: DataType, allowStructEvolution: Boolean = false): Expression

    Add a cast to the child expression if it differs from the specified data type.

    Add a cast to the child expression if it differs from the specified data type. Note that structs here are cast by name, rather than the Spark SQL default of casting by position.

    fromExpression

    the expression to cast

    dataType

    The data type to cast to.

    allowStructEvolution

    Whether to allow structs to evolve. When this is false (default), struct casting will throw an error if there's any mismatch between column names. For example, (b, c, a) -> (a, b, c) is always a valid cast, but (a, b) -> (a, b, c) is valid only with this flag set.

    Attributes
    protected
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  8. def conf: SQLConf
    Definition Classes
    SQLConfHelper
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  12. def generateUpdateExpressions(targetCols: Seq[NamedExpression], nameParts: Seq[Seq[String]], updateExprs: Seq[Expression], resolver: Resolver, generatedColumns: Seq[StructField]): Seq[Option[Expression]]

    See docs on overloaded method.

    See docs on overloaded method.

    Attributes
    protected
  13. def generateUpdateExpressions(targetCols: Seq[NamedExpression], updateOps: Seq[UpdateOperation], resolver: Resolver, pathPrefix: Seq[String] = Nil, allowStructEvolution: Boolean = false, generatedColumns: Seq[StructField] = Nil): Seq[Option[Expression]]

    Given a list of target-column expressions and a set of update operations, generate a list of update expressions, which are aligned with given target-column expressions.

    Given a list of target-column expressions and a set of update operations, generate a list of update expressions, which are aligned with given target-column expressions.

    For update operations to nested struct fields, this method recursively walks down schema tree and apply the update expressions along the way. For example, assume table target has two attributes a and z, where a is of struct type with 3 fields: b, c and d, and z is of integer type.

    Given an update command:

    • UPDATE target SET a.b = 1, a.c = 2, z = 3

    this method works as follows:

    generateUpdateExpressions(targetCols=[a,z], updateOps=[(a.b, 1), (a.c, 2), (z, 3)]) generateUpdateExpressions(targetCols=[b,c,d], updateOps=[(b, 1),(c, 2)], pathPrefix=["a"]) end-of-recursion -> returns (1, 2, d) -> return ((1, 2, d), 3)

    targetCols

    a list of expressions to read named columns; these named columns can be either the top-level attributes of a table, or the nested fields of a StructType column.

    updateOps

    a set of update operations.

    pathPrefix

    the path from root to the current (nested) column. Only used for printing out full column path in error messages.

    allowStructEvolution

    Whether to allow structs to evolve. When this is false (default), struct casting will throw an error if there's any mismatch between column names. For example, (b, c, a) -> (a, b, c) is always a valid cast, but (a, b) -> (a, b, c) is valid only with this flag set.

    generatedColumns

    the list of the generated columns in the table. When a column is a generated column and the user doesn't provide a update expression, its update expression in the return result will be None. If generatedColumns is empty, any of the options in the return result must be non-empty.

    returns

    a sequence of expression options. The elements in the sequence are options because when a column is a generated column but the user doesn't provide an update expression for this column, we need to generate the update expression according to the generated column definition. But this method doesn't have enough context to do that. Hence, we return a None for this case so that the caller knows it should generate the update expression for such column. For other cases, we will always return Some(expr).

    Attributes
    protected
  14. def generateUpdateExprsForGeneratedColumns(updateTarget: LogicalPlan, generatedColumns: Seq[StructField], updateExprs: Seq[Option[Expression]], finalSchemaExprs: Option[Seq[Attribute]] = None): Seq[Expression]

    Generate update expressions for generated columns that the user doesn't provide a update expression.

    Generate update expressions for generated columns that the user doesn't provide a update expression. For each item in updateExprs that's None, we will find its generation expression from generatedColumns. In order to resolve this generation expression, we will create a fake Project which contains all update expressions and resolve the generation expression with this project. Source columns of a generation expression will also be replaced with their corresponding update expressions.

    For example, given a table that has a generated column g defined as c1 + 10. For the following update command:

    UPDATE target SET c1 = c2 + 100, c2 = 1000

    We will generate the update expression (c2 + 100) + 10 for column g. Note: in this update expression, we should use the old c2 attribute rather than its new value 1000.

    updateTarget

    The logical plan of the table to be updated.

    generatedColumns

    A list of generated columns.

    updateExprs

    The aligned (with finalSchemaExprs if not None, or updateTarget.output otherwise) update actions.

    finalSchemaExprs

    In case of UPDATE in MERGE when schema evolution happened, this is the final schema of the target table. This might not be the same as the output of updateTarget.

    returns

    a sequence of update expressions for all of columns in the table.

    Attributes
    protected
  15. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  16. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  17. def improveUnsupportedOpError(f: => Unit): Unit
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  18. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  19. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  21. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  22. def resolveReferencesForExpressions(sparkSession: SparkSession, exprs: Seq[Expression], planProvidingAttrs: LogicalPlan): Seq[Expression]

    Resolve expressions using the attributes provided by planProvidingAttrs.

    Resolve expressions using the attributes provided by planProvidingAttrs. Throw an error if failing to resolve any expressions.

    Attributes
    protected
    Definition Classes
    AnalysisHelper
  23. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  24. def toDataset(sparkSession: SparkSession, logicalPlan: LogicalPlan): Dataset[Row]
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  25. def toString(): String
    Definition Classes
    AnyRef → Any
  26. def tryResolveReferences(sparkSession: SparkSession)(expr: Expression, planContainingExpr: LogicalPlan): Expression
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  27. def tryResolveReferencesForExpressions(sparkSession: SparkSession, exprs: Seq[Expression], planContainingExpr: LogicalPlan): Seq[Expression]
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  28. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  29. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  30. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from AnalysisHelper

Inherited from SQLConfHelper

Inherited from CastSupport

Inherited from AnyRef

Inherited from Any

Ungrouped