case class JdbcTableDataObject(id: DataObjectId, createSql: Option[String] = None, preReadSql: Option[String] = None, postReadSql: Option[String] = None, preWriteSql: Option[String] = None, postWriteSql: Option[String] = None, schemaMin: Option[StructType] = None, table: Table, jdbcFetchSize: Int = 1000, saveMode: SDLSaveMode = SDLSaveMode.Overwrite, allowSchemaEvolution: Boolean = false, connectionId: ConnectionId, jdbcOptions: Map[String, String] = Map(), virtualPartitions: Seq[String] = Seq(), expectedPartitionsCondition: Option[String] = None, metadata: Option[DataObjectMetadata] = None)(implicit instanceRegistry: InstanceRegistry) extends TransactionalSparkTableDataObject with CanHandlePartitions with CanEvolveSchema with CanMergeDataFrame with Product with Serializable

DataObject of type JDBC. Provides details for an action to access tables in a database through JDBC.

id

unique name of this data object

createSql

DDL-statement to be executed in prepare phase, using output jdbc connection. Note that it is also possible to let Spark create the table in Init-phase. See jdbcOptions to customize column data types for auto-created DDL-statement.

preReadSql

SQL-statement to be executed in exec phase before reading input table, using input jdbc connection. Use tokens with syntax %{<spark sql expression>} to substitute with values from DefaultExpressionData.

postReadSql

SQL-statement to be executed in exec phase after reading input table and before action is finished, using input jdbc connection Use tokens with syntax %{<spark sql expression>} to substitute with values from DefaultExpressionData.

preWriteSql

SQL-statement to be executed in exec phase before writing output table, using output jdbc connection Use tokens with syntax %{<spark sql expression>} to substitute with values from DefaultExpressionData.

postWriteSql

SQL-statement to be executed in exec phase after writing output table, using output jdbc connection Use tokens with syntax %{<spark sql expression>} to substitute with values from DefaultExpressionData.

schemaMin

An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.

table

The jdbc table to be read

jdbcFetchSize

Number of rows to be fetched together by the Jdbc driver

saveMode

SDLSaveMode to use when writing table, default is "Overwrite". Only "Append" and "Overwrite" supported.

allowSchemaEvolution

If set to true schema evolution will automatically occur when writing to this DataObject with different schema, otherwise SDL will stop with error.

connectionId

Id of JdbcConnection configuration

jdbcOptions

Any jdbc options according to https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html. Note that some options above set and override some of this options explicitly. Use "createTableOptions" and "createTableColumnTypes" to control automatic creating of database tables.

virtualPartitions

Virtual partition columns. Note that this doesn't need to be the same as the database partition columns for this table. But it is important that there is an index on these columns to efficiently list existing "partitions".

expectedPartitionsCondition

Optional definition of partitions expected to exist. Define a Spark SQL expression that is evaluated against a PartitionValues instance and returns true or false Default is to expect all partitions to exist.

Annotations
@Scaladoc()
Linear Supertypes
Serializable, Serializable, Product, Equals, CanMergeDataFrame, CanEvolveSchema, CanHandlePartitions, TransactionalSparkTableDataObject, CanWriteDataFrame, TableDataObject, SchemaValidation, CanCreateDataFrame, DataObject, AtlasExportable, SmartDataLakeLogger, ParsableFromConfig[DataObject], SdlConfigObject, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. JdbcTableDataObject
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. CanMergeDataFrame
  7. CanEvolveSchema
  8. CanHandlePartitions
  9. TransactionalSparkTableDataObject
  10. CanWriteDataFrame
  11. TableDataObject
  12. SchemaValidation
  13. CanCreateDataFrame
  14. DataObject
  15. AtlasExportable
  16. SmartDataLakeLogger
  17. ParsableFromConfig
  18. SdlConfigObject
  19. AnyRef
  20. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new JdbcTableDataObject(id: DataObjectId, createSql: Option[String] = None, preReadSql: Option[String] = None, postReadSql: Option[String] = None, preWriteSql: Option[String] = None, postWriteSql: Option[String] = None, schemaMin: Option[StructType] = None, table: Table, jdbcFetchSize: Int = 1000, saveMode: SDLSaveMode = SDLSaveMode.Overwrite, allowSchemaEvolution: Boolean = false, connectionId: ConnectionId, jdbcOptions: Map[String, String] = Map(), virtualPartitions: Seq[String] = Seq(), expectedPartitionsCondition: Option[String] = None, metadata: Option[DataObjectMetadata] = None)(implicit instanceRegistry: InstanceRegistry)

    id

    unique name of this data object

    createSql

    DDL-statement to be executed in prepare phase, using output jdbc connection. Note that it is also possible to let Spark create the table in Init-phase. See jdbcOptions to customize column data types for auto-created DDL-statement.

    preReadSql

    SQL-statement to be executed in exec phase before reading input table, using input jdbc connection. Use tokens with syntax %{<spark sql expression>} to substitute with values from DefaultExpressionData.

    postReadSql

    SQL-statement to be executed in exec phase after reading input table and before action is finished, using input jdbc connection Use tokens with syntax %{<spark sql expression>} to substitute with values from DefaultExpressionData.

    preWriteSql

    SQL-statement to be executed in exec phase before writing output table, using output jdbc connection Use tokens with syntax %{<spark sql expression>} to substitute with values from DefaultExpressionData.

    postWriteSql

    SQL-statement to be executed in exec phase after writing output table, using output jdbc connection Use tokens with syntax %{<spark sql expression>} to substitute with values from DefaultExpressionData.

    schemaMin

    An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.

    table

    The jdbc table to be read

    jdbcFetchSize

    Number of rows to be fetched together by the Jdbc driver

    saveMode

    SDLSaveMode to use when writing table, default is "Overwrite". Only "Append" and "Overwrite" supported.

    allowSchemaEvolution

    If set to true schema evolution will automatically occur when writing to this DataObject with different schema, otherwise SDL will stop with error.

    connectionId

    Id of JdbcConnection configuration

    jdbcOptions

    Any jdbc options according to https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html. Note that some options above set and override some of this options explicitly. Use "createTableOptions" and "createTableColumnTypes" to control automatic creating of database tables.

    virtualPartitions

    Virtual partition columns. Note that this doesn't need to be the same as the database partition columns for this table. But it is important that there is an index on these columns to efficiently list existing "partitions".

    expectedPartitionsCondition

    Optional definition of partitions expected to exist. Define a Spark SQL expression that is evaluated against a PartitionValues instance and returns true or false Default is to expect all partitions to exist.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def addFieldIfNotExisting(writeSchema: StructType, colName: String, dataType: DataType): StructType
    Attributes
    protected
    Definition Classes
    CanCreateDataFrame
  5. val allowSchemaEvolution: Boolean

    If set to true schema evolution will automatically occur when writing to this DataObject with different schema, otherwise SDL will stop with error.

    If set to true schema evolution will automatically occur when writing to this DataObject with different schema, otherwise SDL will stop with error.

    Definition Classes
    JdbcTableDataObject → CanEvolveSchema
  6. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  7. def atlasName: String
    Definition Classes
    TableDataObject → DataObject → AtlasExportable
  8. def atlasQualifiedName(prefix: String): String
    Definition Classes
    TableDataObject → AtlasExportable
  9. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  10. val connection: JdbcTableConnection

    Connection defines driver, url and db in central location

  11. val connectionId: ConnectionId
  12. def createReadSchema(writeSchema: StructType)(implicit context: ActionPipelineContext): StructType

    Creates the read schema based on a given write schema.

    Creates the read schema based on a given write schema. Normally this is the same, but some DataObjects can remove & add columns on read (e.g. KafkaTopicDataObject, SparkFileDataObject) In this cases we have to break the DataFrame lineage und create a dummy DataFrame in init phase.

    Definition Classes
    CanCreateDataFrame
    Annotations
    @Scaladoc()
  13. val createSql: Option[String]
  14. def deleteAllData(): Unit
  15. def deletePartitions(partitionValues: Seq[PartitionValues])(implicit context: ActionPipelineContext): Unit

    Delete virtual partitions by "delete from" statement

    Delete virtual partitions by "delete from" statement

    Definition Classes
    JdbcTableDataObject → CanHandlePartitions
    Annotations
    @Scaladoc()
  16. def dropTable(implicit context: ActionPipelineContext): Unit
    Definition Classes
    JdbcTableDataObject → TableDataObject
  17. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. val expectedPartitionsCondition: Option[String]

    Definition of partitions that are expected to exists.

    Definition of partitions that are expected to exists. This is used to validate that partitions being read exists and don't return no data. Define a Spark SQL expression that is evaluated against a PartitionValues instance and returns true or false example: "elements['yourColName'] > 2017"

    returns

    true if partition is expected to exist.

    Definition Classes
    JdbcTableDataObject → CanHandlePartitions
  19. def factory: FromConfigFactory[DataObject]

    Returns the factory that can parse this type (that is, type CO).

    Returns the factory that can parse this type (that is, type CO).

    Typically, implementations of this method should return the companion object of the implementing class. The companion object in turn should implement FromConfigFactory.

    returns

    the factory (object) for this class.

    Definition Classes
    JdbcTableDataObject → ParsableFromConfig
  20. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  21. def getConnection[T <: Connection](connectionId: ConnectionId)(implicit registry: InstanceRegistry, ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

    Handle class cast exception when getting objects from instance registry

    Handle class cast exception when getting objects from instance registry

    Attributes
    protected
    Definition Classes
    DataObject
    Annotations
    @Scaladoc()
  22. def getConnectionReg[T <: Connection](connectionId: ConnectionId, registry: InstanceRegistry)(implicit ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T
    Attributes
    protected
    Definition Classes
    DataObject
  23. def getDataFrame(partitionValues: Seq[PartitionValues] = Seq())(implicit context: ActionPipelineContext): DataFrame
    Definition Classes
    JdbcTableDataObject → CanCreateDataFrame
  24. def getPKduplicates(implicit context: ActionPipelineContext): DataFrame
    Definition Classes
    TableDataObject
  25. def getPKnulls(implicit context: ActionPipelineContext): DataFrame
    Definition Classes
    TableDataObject
  26. def getPKviolators(implicit context: ActionPipelineContext): DataFrame
    Definition Classes
    TableDataObject
  27. def housekeepingMode: Option[HousekeepingMode]

    Configure a housekeeping mode to e.g cleanup, archive and compact partitions.

    Configure a housekeeping mode to e.g cleanup, archive and compact partitions. Default is None.

    Definition Classes
    DataObject
    Annotations
    @Scaladoc()
  28. val id: DataObjectId

    A unique identifier for this instance.

    A unique identifier for this instance.

    Definition Classes
    JdbcTableDataObject → DataObject → SdlConfigObject
  29. def init(df: DataFrame, partitionValues: Seq[PartitionValues], saveModeOptions: Option[SaveModeOptions] = None)(implicit context: ActionPipelineContext): Unit

    Called during init phase for checks and initialization.

    Called during init phase for checks and initialization. If possible dont change the system until execution phase.

    Definition Classes
    JdbcTableDataObject → CanWriteDataFrame
  30. implicit val instanceRegistry: InstanceRegistry
  31. def isDbExisting(implicit context: ActionPipelineContext): Boolean
    Definition Classes
    JdbcTableDataObject → TableDataObject
  32. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  33. def isPKcandidateKey(implicit context: ActionPipelineContext): Boolean
    Definition Classes
    TableDataObject
  34. def isTableExisting(implicit context: ActionPipelineContext): Boolean
    Definition Classes
    JdbcTableDataObject → TableDataObject
  35. val jdbcFetchSize: Int
  36. val jdbcOptions: Map[String, String]
  37. def listPartitions(implicit context: ActionPipelineContext): Seq[PartitionValues]

    Listing virtual partitions by a "select distinct partition-columns" query

    Listing virtual partitions by a "select distinct partition-columns" query

    Definition Classes
    JdbcTableDataObject → CanHandlePartitions
    Annotations
    @Scaladoc()
  38. lazy val logger: Logger
    Attributes
    protected
    Definition Classes
    SmartDataLakeLogger
    Annotations
    @transient()
  39. def mergeDataFrameByPrimaryKey(df: DataFrame, saveModeOptions: SaveModeMergeOptions)(implicit context: ActionPipelineContext): Unit

    Merges DataFrame with existing table data by writing DataFrame to a temp-table and using SQL Merge-statement.

    Merges DataFrame with existing table data by writing DataFrame to a temp-table and using SQL Merge-statement. Table.primaryKey is used as condition to check if a record is matched or not. If it is matched it gets updated (or deleted), otherwise it is inserted. This all is done in one transaction.

    Annotations
    @Scaladoc()
  40. val metadata: Option[DataObjectMetadata]

    Additional metadata for the DataObject

    Additional metadata for the DataObject

    Definition Classes
    JdbcTableDataObject → DataObject
  41. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  42. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  43. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  44. val partitions: Seq[String]

    Definition of partition columns

    Definition of partition columns

    Definition Classes
    JdbcTableDataObject → CanHandlePartitions
  45. def postRead(partitionValues: Seq[PartitionValues])(implicit context: ActionPipelineContext): Unit

    Runs operations after reading from DataObject

    Runs operations after reading from DataObject

    Definition Classes
    JdbcTableDataObject → DataObject
  46. val postReadSql: Option[String]
  47. def postWrite(partitionValues: Seq[PartitionValues])(implicit context: ActionPipelineContext): Unit

    Runs operations after writing to DataObject

    Runs operations after writing to DataObject

    Definition Classes
    JdbcTableDataObject → DataObject
  48. val postWriteSql: Option[String]
  49. def preRead(partitionValues: Seq[PartitionValues])(implicit context: ActionPipelineContext): Unit

    Runs operations before reading from DataObject

    Runs operations before reading from DataObject

    Definition Classes
    JdbcTableDataObject → DataObject
  50. val preReadSql: Option[String]
  51. def preWrite(implicit context: ActionPipelineContext): Unit

    Runs operations before writing to DataObject Note: As the transformed SubFeed doesnt yet exist in Action.preWrite, no partition values can be passed as parameters as in preRead

    Runs operations before writing to DataObject Note: As the transformed SubFeed doesnt yet exist in Action.preWrite, no partition values can be passed as parameters as in preRead

    Definition Classes
    JdbcTableDataObject → DataObject
  52. val preWriteSql: Option[String]
  53. def prepare(implicit context: ActionPipelineContext): Unit

    Prepare & test DataObject's prerequisits

    Prepare & test DataObject's prerequisits

    This runs during the "prepare" operation of the DAG.

    Definition Classes
    JdbcTableDataObject → DataObject
  54. val saveMode: SDLSaveMode
  55. val schemaMin: Option[StructType]

    An optional, minimal schema that a DataObject schema must have to pass schema validation.

    An optional, minimal schema that a DataObject schema must have to pass schema validation.

    The schema validation semantics are: - Schema A is valid in respect to a minimal schema B when B is a subset of A. This means: the whole column set of B is contained in the column set of A.

    • A column of B is contained in A when A contains a column with equal name and data type.
    • Column order is ignored.
    • Column nullability is ignored.
    • Duplicate columns in terms of name and data type are eliminated (set semantics).

    Note: This is mainly used by the functionality defined in CanCreateDataFrame and CanWriteDataFrame, that is, when reading or writing Spark data frames from/to the underlying data container. io.smartdatalake.workflow.action.Actions that work with files ignore the schemaMin attribute if it is defined. Additionally schemaMin can be used to define the schema used if there is no data or table doesn't yet exist.

    Definition Classes
    JdbcTableDataObject → SchemaValidation
  56. def streamingOptions: Map[String, String]
    Definition Classes
    CanWriteDataFrame
  57. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  58. var table: Table
    Definition Classes
    JdbcTableDataObject → TableDataObject
  59. val tableSchema: StructType
    Definition Classes
    TableDataObject
  60. def toStringShort: String
    Definition Classes
    DataObject
  61. def validateSchema(df: DataFrame, schemaExpected: StructType, role: String): Unit

    Validate the schema of a given Spark Data Frame df against a given expected schema.

    Validate the schema of a given Spark Data Frame df against a given expected schema.

    df

    The data frame to validate.

    schemaExpected

    The expected schema to validate against.

    role

    role used in exception message. Set to read or write.

    Definition Classes
    SchemaValidation
    Annotations
    @Scaladoc()
    Exceptions thrown

    SchemaViolationException is the schemaMin does not validate.

  62. def validateSchemaHasPartitionCols(df: DataFrame, role: String): Unit

    Validate the schema of a given Spark Data Frame df that it contains the specified partition columns

    Validate the schema of a given Spark Data Frame df that it contains the specified partition columns

    df

    The data frame to validate.

    role

    role used in exception message. Set to read or write.

    Definition Classes
    CanHandlePartitions
    Annotations
    @Scaladoc()
    Exceptions thrown

    SchemaViolationException if the partitions columns are not included.

  63. def validateSchemaHasPrimaryKeyCols(df: DataFrame, primaryKeyCols: Seq[String], role: String): Unit

    Validate the schema of a given Spark Data Frame df that it contains the specified primary key columns

    Validate the schema of a given Spark Data Frame df that it contains the specified primary key columns

    df

    The data frame to validate.

    role

    role used in exception message. Set to read or write.

    Definition Classes
    CanHandlePartitions
    Annotations
    @Scaladoc()
    Exceptions thrown

    SchemaViolationException if the partitions columns are not included.

  64. def validateSchemaMin(df: DataFrame, role: String): Unit

    Validate the schema of a given Spark Data Frame df against schemaMin.

    Validate the schema of a given Spark Data Frame df against schemaMin.

    df

    The data frame to validate.

    role

    role used in exception message. Set to read or write.

    Definition Classes
    SchemaValidation
    Annotations
    @Scaladoc()
    Exceptions thrown

    SchemaViolationException is the schemaMin does not validate.

  65. val virtualPartitions: Seq[String]
  66. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  67. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  68. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  69. def writeDataFrame(df: DataFrame, partitionValues: Seq[PartitionValues] = Seq(), isRecursiveInput: Boolean = false, saveModeOptions: Option[SaveModeOptions] = None)(implicit context: ActionPipelineContext): Unit

    Write DataFrame to DataObject

    Write DataFrame to DataObject

    df

    the DataFrame to write

    partitionValues

    partition values included in DataFrames data

    isRecursiveInput

    if DataFrame needs this DataObject as input - special treatment might be needed in this case.

    Definition Classes
    JdbcTableDataObject → CanWriteDataFrame
  70. def writeStreamingDataFrame(df: DataFrame, trigger: Trigger, options: Map[String, String], checkpointLocation: String, queryName: String, outputMode: OutputMode = OutputMode.Append, saveModeOptions: Option[SaveModeOptions] = None)(implicit context: ActionPipelineContext): StreamingQuery

    Write Spark structured streaming DataFrame The default implementation uses foreachBatch and this traits writeDataFrame method to write the DataFrame.

    Write Spark structured streaming DataFrame The default implementation uses foreachBatch and this traits writeDataFrame method to write the DataFrame. Some DataObjects will override this with specific implementations (Kafka).

    df

    The Streaming DataFrame to write

    trigger

    Trigger frequency for stream

    checkpointLocation

    location for checkpoints of streaming query

    Definition Classes
    CanWriteDataFrame
    Annotations
    @Scaladoc()

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from CanMergeDataFrame

Inherited from CanEvolveSchema

Inherited from CanHandlePartitions

Inherited from TransactionalSparkTableDataObject

Inherited from CanWriteDataFrame

Inherited from TableDataObject

Inherited from SchemaValidation

Inherited from CanCreateDataFrame

Inherited from DataObject

Inherited from AtlasExportable

Inherited from SmartDataLakeLogger

Inherited from ParsableFromConfig[DataObject]

Inherited from SdlConfigObject

Inherited from AnyRef

Inherited from Any

Ungrouped