object Environment

Environment dependent configurations. They can be set - by Java system properties (prefixed with "sdl.", e.g. "sdl.hadoopAuthoritiesWithAclsRequired") - by Environment variables (prefixed with "SDL_" and camelCase converted to uppercase, e.g. "SDL_HADOOP_AUTHORITIES_WITH_ACLS_REQUIRED") - by a custom io.smartdatalake.app.SmartDataLakeBuilder implementation for your environment, which sets these variables directly.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Environment
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def classLoader: ClassLoader
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  7. val configPathsForLocalSubstitution: Seq[String]
  8. var enableAutomaticDataFrameCaching: Boolean

    Set to true if you want to enable automatic caching of DataFrames that are used multiple times (default=true).

  9. var enableCheckConfigDuplicates: Boolean

    Set to true to enable check for duplicate first class object definitions when loading configuration (default=true).

    Set to true to enable check for duplicate first class object definitions when loading configuration (default=true). The check fails if Connections, DataObjects or Actions are defined in multiple locations.

  10. var enableOverwriteUnpartitionedSparkFileDataObjectAdls: Boolean

    Set to true if you want to enable workaround to overwrite unpartitioned SparkFileDataObject on Azure ADLSv2 (default=false).

  11. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  13. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  14. def globalConfig: GlobalConfig
  15. var hadoopAuthoritiesWithAclsRequired: Seq[String]

    List of hadoop authorities for which acls must be configured The environment parameter can contain multiple authorities separated by comma.

    List of hadoop authorities for which acls must be configured The environment parameter can contain multiple authorities separated by comma. An authority is compared against the filesystem URI with contains(...)

  16. var hadoopDefaultSchemeAuthority: Option[URI]

    Set default hadoop schema and authority for path

  17. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  18. var hdfsAclsLimitToBasedir: Boolean

    Limit setting ACL's to Basedir (default=true) See hdfsAclsUserHomeLevel or hdfsBasedir on how the basedir is determined

  19. var hdfsAclsMinLevelPermissionModify: Int

    Modifying ACL's is only allowed below and including the following level (default=2) See also io.smartdatalake.util.misc.AclUtil

  20. var hdfsAclsMinLevelPermissionOverwrite: Int

    Overwriting ACL's is only allowed below and including the following level (default=5) See also io.smartdatalake.util.misc.AclUtil

  21. var hdfsAclsUserHomeLevel: Int

    Set path level of user home to determine basedir automatically (Default=2 -> /user/myUserHome)

  22. var hdfsBasedir: Option[URI]

    Set basedir explicitly.

    Set basedir explicitly. This overrides automatically detected user home for acl constraints by hdfsAclsUserHomeLevel.

  23. def instanceRegistry: InstanceRegistry
  24. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  25. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  26. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  27. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  28. val runIdPartitionColumnName: String
  29. var runtimeDataNumberOfExecutionsToKeep: Int

    Number of Executions to keep runtime data for in streaming mode (default = 10).

    Number of Executions to keep runtime data for in streaming mode (default = 10). Must be bigger than 1.

  30. var schemaEvolutionNewColumnsLast: Boolean

    ordering of columns in SchemaEvolution result - true: result schema is ordered according to existing schema, new columns are appended - false: result schema is ordered according to new schema, deleted columns are appended

  31. var schemaValidationDeepComarison: Boolean

    If true, schema validation inspects the whole hierarchy of structured data types.

    If true, schema validation inspects the whole hierarchy of structured data types. This allows partial matches for schemaMin validation. If false, structural data types must match exactly to validate.

    Example:

    1. Using io.smartdatalake.workflow.dataobject.SchemaValidation.validateSchemaMin: val schema = StructType.fromDDL("c1 STRING, c2 STRUCT(c2_1 INT, c2_2 STRING)") validates against StructType.fromDDL("c1 STRING, c2 STRUCT(c2_1 INT)") only if schemaValidationDeepComarison == true.

  32. var schemaValidationIgnoresNullability: Boolean

    If true, schema validation does not consider nullability of columns/fields when checking for equality.

    If true, schema validation does not consider nullability of columns/fields when checking for equality. If false, schema validation considers two columns/fields different when their nullability property is not equal.

  33. var simplifyFinalExceptionLog: Boolean

    Simplify final exception for better usability of log - truncate stacktrace starting from "monix.*" entries - limit logical plan in AnalysisException to 5 lines

  34. def sparkSession: SparkSession
  35. var stopStreamingGracefully: Boolean
  36. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  37. var taskSkippedExceptionLogLevel: Level

    Set log level for exceptions about skipped Actions, e.g.

    Set log level for exceptions about skipped Actions, e.g. NoDataToProcessWarning (default=info).

  38. def toString(): String
    Definition Classes
    AnyRef → Any
  39. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  40. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  41. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from AnyRef

Inherited from Any

Ungrouped