case class CustomDfsTransformerConfig(className: Option[String] = None, scalaFile: Option[String] = None, scalaCode: Option[String] = None, sqlCode: Option[Map[DataObjectId, String]] = None, options: Option[Map[String, String]] = None, runtimeOptions: Option[Map[String, String]] = None) extends Product with Serializable

Configuration of a custom Spark-DataFrame transformation between many inputs and many outputs (n:m). Define a transform function which receives a map of input DataObjectIds with DataFrames and a map of options and has to return a map of output DataObjectIds with DataFrames, see also trait CustomDfsTransformer.

className

Optional class name implementing trait CustomDfsTransformer

scalaFile

Optional file where scala code for transformation is loaded from. The scala code in the file needs to be a function of type fnTransformType.

scalaCode

Optional scala code for transformation. The scala code needs to be a function of type fnTransformType.

sqlCode

Optional map of DataObjectId and corresponding SQL Code. Use tokens %{<key>} to replace with runtimeOptions in SQL code. Example: "select * from test where run = %{runId}"

options

Options to pass to the transformation

runtimeOptions

optional tuples of [key, spark sql expression] to be added as additional options when executing transformation. The spark sql expressions are evaluated against an instance of DefaultExpressionData.

Annotations
@Scaladoc()
Linear Supertypes
Serializable, Serializable, Product, Equals, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. CustomDfsTransformerConfig
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new CustomDfsTransformerConfig(className: Option[String] = None, scalaFile: Option[String] = None, scalaCode: Option[String] = None, sqlCode: Option[Map[DataObjectId, String]] = None, options: Option[Map[String, String]] = None, runtimeOptions: Option[Map[String, String]] = None)

    className

    Optional class name implementing trait CustomDfsTransformer

    scalaFile

    Optional file where scala code for transformation is loaded from. The scala code in the file needs to be a function of type fnTransformType.

    scalaCode

    Optional scala code for transformation. The scala code needs to be a function of type fnTransformType.

    sqlCode

    Optional map of DataObjectId and corresponding SQL Code. Use tokens %{<key>} to replace with runtimeOptions in SQL code. Example: "select * from test where run = %{runId}"

    options

    Options to pass to the transformation

    runtimeOptions

    optional tuples of [key, spark sql expression] to be added as additional options when executing transformation. The spark sql expressions are evaluated against an instance of DefaultExpressionData.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. val className: Option[String]
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  9. val impl: DfsTransformer
  10. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  11. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  13. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  14. val options: Option[Map[String, String]]
  15. val runtimeOptions: Option[Map[String, String]]
  16. val scalaCode: Option[String]
  17. val scalaFile: Option[String]
  18. val sqlCode: Option[Map[DataObjectId, String]]
  19. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  20. def toString(): String
    Definition Classes
    CustomDfsTransformerConfig → AnyRef → Any
  21. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped