class DefaultSmartDataLakeBuilder extends SmartDataLakeBuilder

Default Smart Data Lake Command Line Application.

Implementation Note: This must be a class and not an object in order to be found by reflection in DatabricksSmartDataLakeBuilder

Annotations
@Scaladoc()
Linear Supertypes
SmartDataLakeBuilder, SmartDataLakeLogger, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DefaultSmartDataLakeBuilder
  2. SmartDataLakeBuilder
  3. SmartDataLakeLogger
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DefaultSmartDataLakeBuilder()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val appType: String
    Definition Classes
    SmartDataLakeBuilder
  5. val appVersion: String
    Definition Classes
    SmartDataLakeBuilder
  6. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. final def execActionDAG(actionDAGRun: ActionDAGRun, actionsSelected: Seq[Action], context: ActionPipelineContext, lastStartTime: Option[LocalDateTime] = None): Seq[SubFeed]

    Execute one action DAG iteration and call recursion if streaming mode Must be implemented with tail recursion to avoid stack overflow error for long running streaming jobs.

    Execute one action DAG iteration and call recursion if streaming mode Must be implemented with tail recursion to avoid stack overflow error for long running streaming jobs.

    Definition Classes
    SmartDataLakeBuilder
    Annotations
    @Scaladoc() @tailrec()
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  13. def initConfigFromEnvironment: SmartDataLakeBuilderConfig

    Create a new SDL configuration.

    Create a new SDL configuration.

    Could be used in the future to set default values.

    returns

    a new, initialized SmartDataLakeBuilderConfig.

    Definition Classes
    SmartDataLakeBuilder
    Annotations
    @Scaladoc()
  14. val instanceRegistry: InstanceRegistry

    InstanceRegistry instance

    InstanceRegistry instance

    Definition Classes
    SmartDataLakeBuilder
  15. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  16. lazy val logger: Logger
    Attributes
    protected
    Definition Classes
    SmartDataLakeLogger
    Annotations
    @transient()
  17. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  19. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  20. def parseAndRun(args: Array[String], ignoreOverrideJars: Boolean = false): Unit
  21. def parseCommandLineArguments(args: Array[String], config: SmartDataLakeBuilderConfig): Option[SmartDataLakeBuilderConfig]

    Parses the supplied (command line) arguments.

    Parses the supplied (command line) arguments.

    This method parses command line arguments and creates the corresponding SmartDataLakeBuilderConfig

    args

    an Array of command line arguments.

    config

    a configuration initialized with default values.

    returns

    a new configuration with default values overwritten from the supplied command line arguments.

    Definition Classes
    SmartDataLakeBuilder
    Annotations
    @Scaladoc()
  22. val parser: OptionParser[SmartDataLakeBuilderConfig]

    The Parser defines how to extract the options from the command line args.

    The Parser defines how to extract the options from the command line args. Subclasses SmartDataLakeBuilder can define additional options to be extracted.

    Attributes
    protected
    Definition Classes
    SmartDataLakeBuilder
  23. def run(appConfig: SmartDataLakeBuilderConfig): Map[RuntimeEventState, Int]

    Run the application with the provided configuration.

    Run the application with the provided configuration.

    appConfig

    Application configuration (parsed from command line).

    Definition Classes
    SmartDataLakeBuilder
    Annotations
    @Scaladoc()
  24. def startSimulation(appConfig: SmartDataLakeBuilderConfig, initialSubFeeds: Seq[SparkSubFeed], dataObjectsState: Seq[DataObjectState] = Seq())(implicit instanceRegistry: InstanceRegistry, session: SparkSession): (Seq[SparkSubFeed], Map[RuntimeEventState, Int])

    Start a simulation run.

    Start a simulation run. This executes the DAG and returns all subfeeds including the transformed DataFrames. Only prepare and init are executed. All initial subfeeds must be provided as input.

    Note: this only works with SparkActions for now

    appConfig

    application configuration

    initialSubFeeds

    initial subfeeds for DataObjects at the beginning of the DAG

    dataObjectsState

    state for incremental DataObjects

    returns

    tuple of list of final subfeeds and statistics (action count per RuntimeEventState)

    Definition Classes
    SmartDataLakeBuilder
    Annotations
    @Scaladoc()
  25. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  26. def toString(): String
    Definition Classes
    AnyRef → Any
  27. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  28. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  29. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from SmartDataLakeBuilder

Inherited from SmartDataLakeLogger

Inherited from AnyRef

Inherited from Any

Ungrouped