trait Writer extends AnyRef
- Alphabetic
- By Inheritance
- Writer
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Abstract Value Members
-
abstract
def
insert(location: String, databaseName: String, tableName: String, updates: DataFrame, partitioning: List[String], format: String)(implicit spark: SparkSession): DataFrame
Insert or append data into a table Does not resolve duplicates
Insert or append data into a table Does not resolve duplicates
- location
full path of where the data will be located
- tableName
the name of the updated/created table
- updates
new data to be merged with existing data
- spark
a valid spark session
- returns
the data as a dataframe
-
abstract
def
scd1(location: String, databaseName: String, tableName: String, updates: DataFrame, primaryKeys: Seq[String], oidName: String, createdOnName: String, updatedOnName: String, partitioning: List[String], format: String)(implicit spark: SparkSession): DataFrame
Update the data only if the data has changed Insert new data maintains updatedOn and createdOn timestamps for each record usually used for dimension table for which keeping the full historic is not required.
Update the data only if the data has changed Insert new data maintains updatedOn and createdOn timestamps for each record usually used for dimension table for which keeping the full historic is not required.
- location
full path of where the data will be located
- tableName
the name of the updated/created table
- updates
new data to be merged with existing data
- primaryKeys
name of the columns holding the unique id
- oidName
name of the column holding the hash of the column that can change over time (or version number)
- createdOnName
name of the column holding the creation timestamp
- updatedOnName
name of the column holding the last update timestamp
- spark
a valid spark session
- returns
the data as a dataframe
-
abstract
def
scd2(location: String, databaseName: String, tableName: String, updates: DataFrame, primaryKeys: Seq[String], buidName: String, oidName: String, isCurrentName: String, partitioning: List[String], format: String, validFromName: String, validToName: String, minValidFromDate: LocalDate, maxValidToDate: LocalDate)(implicit spark: SparkSession): DataFrame
Update the data only if the data has changed Insert new data maintains updatedOn and createdOn timestamps for each record usually used for dimension table for which keeping the full historic is required.
Update the data only if the data has changed Insert new data maintains updatedOn and createdOn timestamps for each record usually used for dimension table for which keeping the full historic is required.
- location
full path of where the data will be located
- tableName
the name of the updated/created table
- updates
new data to be merged with existing data
- primaryKeys
name of the columns holding the unique id
- oidName
name of the column holding the hash of the column that can change over time (or version number)
- spark
a valid spark session
- returns
the data as a dataframe
-
abstract
def
upsert(location: String, databaseName: String, tableName: String, updates: DataFrame, primaryKeys: Seq[String], partitioning: List[String], format: String)(implicit spark: SparkSession): DataFrame
Update or insert data into a table Resolves duplicates by using the list of primary key passed as argument
Update or insert data into a table Resolves duplicates by using the list of primary key passed as argument
- location
full path of where the data will be located
- tableName
the name of the updated/created table
- updates
new data to be merged with existing data
- primaryKeys
name of the columns holding the unique id
- spark
a valid spark session
- returns
the data as a dataframe
-
abstract
def
writeOnce(location: String, databaseName: String, tableName: String, df: DataFrame, partitioning: List[String], format: String, options: Map[String, String])(implicit spark: SparkSession): DataFrame
Overwrites the data located in output/tableName usually used for small/test tables.
Overwrites the data located in output/tableName usually used for small/test tables.
- location
full path of where the data will be located
- databaseName
the name of the database
- tableName
the name of the table
- df
the data to write
- partitioning
list of column used as partition
- format
format of the data to be written
- spark
a valid spark session
- returns
the data as a dataframe
Concrete Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native() @HotSpotIntrinsicCandidate()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
Deprecated Value Members
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] ) @Deprecated
- Deprecated