io.smartdatalake.workflow.action.customlogic
CustomDfCreatorConfig
Companion object CustomDfCreatorConfig
case class CustomDfCreatorConfig(className: Option[String] = None, scalaFile: Option[String] = None, scalaCode: Option[String] = None, options: Option[Map[String, String]] = None) extends Product with Serializable
Configuration of a custom Spark-DataFrame creator as part of CustomDfDataObject Define a exec function which receives a map of options and returns a DataFrame to be used as input. Optionally define a schema function to return a StructType used as schema in init-phase. See also trait CustomDfCreator.
Note that for now implementing CustomDfCreator.schema method is only possible with className configuration attribute.
- className
Optional class name implementing trait CustomDfCreator
- scalaFile
Optional file where scala code for creator is loaded from. The scala code in the file needs to be a function of type fnExecType.
- scalaCode
Optional scala code for creator. The scala code needs to be a function of type fnExecType.
- options
Options to pass to the creator
- Annotations
- @Scaladoc()
- Alphabetic
- By Inheritance
- CustomDfCreatorConfig
- Serializable
- Serializable
- Product
- Equals
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
-
new
CustomDfCreatorConfig(className: Option[String] = None, scalaFile: Option[String] = None, scalaCode: Option[String] = None, options: Option[Map[String, String]] = None)
- className
Optional class name implementing trait CustomDfCreator
- scalaFile
Optional file where scala code for creator is loaded from. The scala code in the file needs to be a function of type fnExecType.
- scalaCode
Optional scala code for creator. The scala code needs to be a function of type fnExecType.
- options
Options to pass to the creator
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
- val className: Option[String]
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native() @HotSpotIntrinsicCandidate()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def exec(implicit context: ActionPipelineContext): DataFrame
- val fnEmptySchema: fnSchemaType
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
- val impl: CustomDfCreator
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
- val options: Option[Map[String, String]]
- val scalaCode: Option[String]
- val scalaFile: Option[String]
- def schema(implicit context: ActionPipelineContext): Option[StructType]
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- CustomDfCreatorConfig → AnyRef → Any
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
Deprecated Value Members
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] ) @Deprecated
- Deprecated