Packages

implicit class DatasetConfOperations extends AnyRef

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DatasetConfOperations
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DatasetConfOperations(ds: DatasetConf)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  9. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  10. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  11. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  13. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  14. def read(implicit config: Configuration, spark: SparkSession): DataFrame

    Using an instance of Spark and the current configuration, reads the dataset from either the tableName or from the location.

    Using an instance of Spark and the current configuration, reads the dataset from either the tableName or from the location.

    config

    configuration currently loaded

    spark

    instance of SparkSession

  15. def readCurrent(implicit config: Configuration, spark: SparkSession): DataFrame

    Read current version of a given DatasetConf based on its loadType

    Read current version of a given DatasetConf based on its loadType

    config

    configuration currently loaded

    spark

    instance of SparkSession

  16. def replacePath(pattern: String, replacement: String): DatasetConf

    Replace the substrings matching the pattern with the replacement value in the path.

    Replace the substrings matching the pattern with the replacement value in the path.

    pattern

    The sequence to be replaced

    replacement

    The replacement sequence

    returns

    A new DatasetConf with its path updated

  17. def replacePlaceholders(placeholder: String, replacement: String): DatasetConf

    Replace the placeholders in the DatasetConf with the replacement value.

    Replace the placeholders in the DatasetConf with the replacement value. The replacement value will be cleaned before being used in the table name. Valid table names only contain alphabet characters, numbers and _. All invalid characters will be replaced by an _.

    placeholder

    The placeholder value to be replaced

    replacement

    The replacement sequence

    returns

    A new DatasetConf with its placeholders filled

  18. def replaceTableName(pattern: String, replacement: String, clean: Boolean): DatasetConf

    Replace the substrings matching the pattern with the replacement value in the table name.

    Replace the substrings matching the pattern with the replacement value in the table name. Valid table names only contain alphabet characters, numbers and _. If clean is set to true, all invalid characters will be replaced by an _. The method will not fail when using a replacement value with invalid characters, but Spark might fail later when trying to create the table.

    pattern

    The sequence to be replaced

    replacement

    The replacement sequence

    clean

    If true, cleans the replacement sequence

    returns

    A new DatasetConf with its table name updated

  19. def resetTo(dateTime: LocalDateTime)(implicit config: Configuration, spark: SparkSession): Unit
  20. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  21. def tableExist(implicit spark: SparkSession): Boolean
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  25. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from AnyRef

Inherited from Any

Ungrouped