Type Members

1.  class DeltaColumnBuilder extends AnyRef

   :: Evolving ::

   :: Evolving ::

   Builder to specify a table column.

   See DeltaTableBuilder for examples.

   Annotations

   @Evolving()

   Since

   1.0.0

2.  class DeltaMergeBuilder extends AnalysisHelper with Logging

   Builder to specify how to merge data from source DataFrame into the target Delta table.

   Builder to specify how to merge data from source DataFrame into the target Delta table. You can specify any number of whenMatched and whenNotMatched clauses. Here are the constraints on these clauses.

   • whenMatched clauses:
     • There can be at most one update action and one delete action in whenMatched clauses.
     • Each whenMatched clause can have an optional condition. However, if there are two whenMatched clauses, then the first one must have a condition.
     • When there are two whenMatched clauses and there are conditions (or the lack of) such that a row matches both clauses, then the first clause/action is executed. In other words, the order of the whenMatched clauses matter.
     • If none of the whenMatched clauses match a source-target row pair that satisfy the merge condition, then the target rows will not be updated or deleted.
     • If you want to update all the columns of the target Delta table with the corresponding column of the source DataFrame, then you can use the whenMatched(...).updateAll(). This is equivalent to

               whenMatched(...).updateExpr(Map(
                 ("col1", "source.col1"),
                 ("col2", "source.col2"),
                 ...))
       

   • whenNotMatched clauses:
     • This clause can have only an insert action, which can have an optional condition.
     • If the whenNotMatched clause is not present or if it is present but the non-matching source row does not satisfy the condition, then the source row is not inserted.
     • If you want to insert all the columns of the target Delta table with the corresponding column of the source DataFrame, then you can use whenMatched(...).insertAll(). This is equivalent to

               whenMatched(...).insertExpr(Map(
                 ("col1", "source.col1"),
                 ("col2", "source.col2"),
                 ...))
       

   Scala example to update a key-value Delta table with new key-values from a source DataFrame:

   deltaTable
    .as("target")
    .merge(
      source.as("source"),
      "target.key = source.key")
    .whenMatched
    .updateExpr(Map(
      "value" -> "source.value"))
    .whenNotMatched
    .insertExpr(Map(
      "key" -> "source.key",
      "value" -> "source.value"))
    .execute()

   Java example to update a key-value Delta table with new key-values from a source DataFrame:

   deltaTable
    .as("target")
    .merge(
      source.as("source"),
      "target.key = source.key")
    .whenMatched
    .updateExpr(
       new HashMap<String, String>() {{
         put("value", "source.value");
       }})
    .whenNotMatched
    .insertExpr(
       new HashMap<String, String>() {{
        put("key", "source.key");
        put("value", "source.value");
      }})
    .execute();

   Since

   0.3.0

3.  class DeltaMergeMatchedActionBuilder extends AnyRef

   Builder class to specify the actions to perform when a target table row has matched a source row based on the given merge condition and optional match condition.

   Builder class to specify the actions to perform when a target table row has matched a source row based on the given merge condition and optional match condition.

   See DeltaMergeBuilder for more information.

   Since

   0.3.0

4.  class DeltaMergeNotMatchedActionBuilder extends AnyRef

   Builder class to specify the actions to perform when a source row has not matched any target Delta table row based on the merge condition, but has matched the additional condition if specified.

   Builder class to specify the actions to perform when a source row has not matched any target Delta table row based on the merge condition, but has matched the additional condition if specified.

   See DeltaMergeBuilder for more information.

   Since

   0.3.0

5.  class DeltaOptimizeBuilder extends AnalysisHelper

   Builder class for constructing OPTIMIZE command and executing.

   Builder class for constructing OPTIMIZE command and executing.

   Since

   2.0.0

6.  class DeltaTable extends DeltaTableOperations with Serializable

   Main class for programmatically interacting with Delta tables.

   Main class for programmatically interacting with Delta tables. You can create DeltaTable instances using the static methods.

   DeltaTable.forPath(sparkSession, pathToTheDeltaTable)

   Since

   0.3.0

7.  class DeltaTableBuilder extends AnyRef

   :: Evolving ::

   :: Evolving ::

   Builder to specify how to create / replace a Delta table. You must specify the table name or the path before executing the builder. You can specify the table columns, the partitioning columns, the location of the data, the table comment and the property, and how you want to create / replace the Delta table.

   After executing the builder, an instance of DeltaTable is returned.

   Scala example to create a Delta table with generated columns, using the table name:

   val table: DeltaTable = DeltaTable.create()
     .tableName("testTable")
     .addColumn("c1",  dataType = "INT", nullable = false)
     .addColumn(
       DeltaTable.columnBuilder("c2")
         .dataType("INT")
         .generatedAlwaysAs("c1 + 10")
         .build()
     )
     .addColumn(
       DeltaTable.columnBuilder("c3")
         .dataType("INT")
         .comment("comment")
         .nullable(true)
         .build()
     )
     .partitionedBy("c1", "c2")
     .execute()

   Scala example to create a delta table using the location:

   val table: DeltaTable = DeltaTable.createIfNotExists(spark)
     .location("/foo/`bar`")
     .addColumn("c1", dataType = "INT", nullable = false)
     .addColumn(
       DeltaTable.columnBuilder(spark, "c2")
         .dataType("INT")
         .generatedAlwaysAs("c1 + 10")
         .build()
     )
     .addColumn(
       DeltaTable.columnBuilder(spark, "c3")
         .dataType("INT")
         .comment("comment")
         .nullable(true)
         .build()
     )
     .partitionedBy("c1", "c2")
     .execute()

   Java Example to replace a table:

   DeltaTable table = DeltaTable.replace()
     .tableName("db.table")
     .addColumn("c1",  "INT", false)
     .addColumn(
       DeltaTable.columnBuilder("c2")
         .dataType("INT")
         .generatedAlwaysBy("c1 + 10")
         .build()
     )
     .execute();

   Annotations

   @Evolving()

   Since

   1.0.0