Return the ACL definition for the Hadoop path of this DataObject

Callback that enables potential transformation to be applied to df after the data is read.

Default is to validate the schemaMin and not apply any modification.

Callback that enables potential transformation to be applied to df before the data is written.

Default is to validate the schemaMin and not apply any modification.

Check if the input files exist.

Compact partitions using Spark

Return the connection id.

Connection defines path prefix (scheme, authority, base path) and ACL's in central location.

create empty partition

Create an OutputStream for a given path, that the Action can use to write data into.

Creates the read schema based on a given write schema. Normally this is the same, but some DataObjects can remove & add columns on read (e.g. KafkaTopicDataObject, SparkFileDataObject) In this cases we have to break the DataFrame lineage und create a dummy DataFrame in init phase.

Delete all data. This is used to implement SaveMode.Overwrite.

delete all files inside given path recursively

Delete given files. This is used to cleanup files after they are processed.

Delete Hadoop Partitions.

if there is no value for a partition column before the last partition column given, the partition path will be exploded

Delete files inside Hadoop Partitions, but keep partition directory to preserve ACLs

if there is no value for a partition column before the last partition column given, the partition path will be exploded

This is called after all output streams have been written. It is used for e.g. making sure empty partitions are created as well.

Definition of partitions that are expected to exists. This is used to validate that partitions being read exists and don't return no data. Define a Spark SQL expression that is evaluated against a PartitionValues instance and returns true or false example: "elements['yourColName'] > 2017"

Extract partition values from a given file path

Returns the factory that can parse this type (that is, type CO).

Typically, implementations of this method should return the companion object of the implementing class. The companion object in turn should implement FromConfigFactory.

Configure whether io.smartdatalake.workflow.action.Actions should fail if the input file(s) are missing on the file system.

Default is false.

Definition of fileName. Default is an asterix to match everything. This is concatenated with the partition layout to search for files.

The name of the (optional) additional column containing the source filename

Filters only existing partition. Note that partition values to check don't need to have a key/value defined for every partition column.

The Spark-Format provider to be used

Generate all paths for given partition values exploding undefined partitions before the last given partition value. Use case: Reading all files from a given path with spark cannot contain wildcards. If there are partitions without given partition value before the last partition value given, they must be searched with globs.

Handle class cast exception when getting objects from instance registry

Constructs an Apache Spark DataFrame from the underlying file content.

As spark-xml doesn't support reading partitions, SDL needs to handle partitions on its own. This method overwrites standard getDataFrame method of SparkFileDataObject for this purpose.

Example for spark-xml failure: reading partitioned XML-files with results in FileNotFoundException session.read .format("xml") .options(Map("rowTag" -> "report")) .schema(rawData.schema.get) .load("partitionedDataObjectPath") .show

List files for given partition values

get partition values formatted by partition layout

Method for subclasses to override the base path for this DataObject. This is for instance needed if pathPrefix is defined in a connection.

Returns the user-defined schema for reading from the data source. By default, this should return schema but it may be customized by data objects that have a source schema and ignore the user-defined schema on read operations.

If a user-defined schema is returned, it overrides any schema inference. If no user-defined schema is set, the schema may be inferred depending on the configuration and type of data frame reader.

prepare paths to be searched

Configure a housekeeping mode to e.g cleanup, archive and compact partitions. Default is None.

A unique identifier for this instance.

Called during init phase for checks and initialization. If possible dont change the system until execution phase.

Return the InstanceRegistry parsed from the SDL configuration used for this run.

List partitions on data object's root path

Additional metadata for the DataObject

Move given partitions. This is used to archive partitions by housekeeping. Note: this is optional to implement.

Returns the configured options for the Spark DataFrameReader/DataFrameWriter.

Return a String specifying the partition layout. For Hadoop the default partition layout is colname1=<value1>/colname2=<value2>/.../

Definition of partition columns

The root path of the files that are handled by this DataObject.

Runs operations after writing to DataObject

Runs operations before writing to DataObject Note: As the transformed SubFeed doesnt yet exist in Action.preWrite, no partition values can be passed as parameters as in preRead

Prepare & test DataObject's prerequisits

This runs during the "prepare" operation of the DAG.

Make a given path relative to this DataObjects base path

Overwrite or Append new data. When writing partitioned data, this applies only to partitions concerned.

An optional DataObject user-defined schema definition.

Some DataObjects support optional schema inference. Specifying this attribute disables automatic schema inference. When the wrapped data source contains a source schema, this schema attribute is ignored.

Note: This is only used by the functionality defined in CanCreateDataFrame, that is, when reading Spark data frames from the underlying data container. io.smartdatalake.workflow.action.Actions that bypass Spark data frames ignore the schema attribute if it is defined.

An optional, minimal schema that a DataObject schema must have to pass schema validation.

The schema validation semantics are: - Schema A is valid in respect to a minimal schema B when B is a subset of A. This means: the whole column set of B is contained in the column set of A.

• A column of B is contained in A when A contains a column with equal name and data type.
• Column order is ignored.
• Column nullability is ignored.
• Duplicate columns in terms of name and data type are eliminated (set semantics).

Note: This is mainly used by the functionality defined in CanCreateDataFrame and CanWriteDataFrame, that is, when reading or writing Spark data frames from/to the underlying data container. io.smartdatalake.workflow.action.Actions that work with files ignore the schemaMin attribute if it is defined. Additionally schemaMin can be used to define the schema used if there is no data or table doesn't yet exist.

default separator for paths

Definition of repartition operation before writing DataFrame with Spark to Hadoop.

This is called before any output stream is created to initialize writing. It is used to apply SaveMode, e.g. deleting existing partitions.

Given some FileRef for another DataObject, translate the paths to the root path of this DataObject

Validate the schema of a given Spark Data Frame df against a given expected schema.

Validate the schema of a given Spark Data Frame df that it contains the specified partition columns

Validate the schema of a given Spark Data Frame df that it contains the specified primary key columns

Validate the schema of a given Spark Data Frame df against schemaMin.

Writes the provided DataFrame to the filesystem.

The partitionValues attribute is used to partition the output by the given columns on the file system.

Write DataFrame to specific Path with properties of this DataObject. This is needed for compacting partitions by housekeeping. Note: this is optional to implement.

Write Spark structured streaming DataFrame The default implementation uses foreachBatch and this traits writeDataFrame method to write the DataFrame. Some DataObjects will override this with specific implementations (Kafka).