class OrphanetGeneSet extends ETLP
- Alphabetic
- By Inheritance
- OrphanetGeneSet
- ETLP
- ETL
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new OrphanetGeneSet()(implicit conf: Configuration)
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- val HGNC_gene_id: (Node) ⇒ Option[String]
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
- val association_status: (Node) ⇒ Option[String]
- val association_type: (Node) ⇒ Option[String]
- val association_type_id: (Node) ⇒ Option[Long]
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native() @HotSpotIntrinsicCandidate()
-
implicit
val
conf: Configuration
- Definition Classes
- ETL
-
def
defaultSampling: PartialFunction[String, (DataFrame) ⇒ DataFrame]
- Definition Classes
- ETL
-
val
destination: DatasetConf
- Definition Classes
- OrphanetGeneSet → ETL
- val ensembl_gene_id: (Node) ⇒ Option[String]
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
extract(lastRunDateTime: LocalDateTime, currentRunDateTime: LocalDateTime)(implicit spark: SparkSession): Map[String, DataFrame]
Reads data from a file system and produce a Map[DatasetConf, DataFrame].
Reads data from a file system and produce a Map[DatasetConf, DataFrame]. This method should avoid transformation and joins but can implement filters in order to make the ETL more efficient.
- spark
an instance of SparkSession
- returns
all the data needed to pass to the transform method and produce the desired output.
- Definition Classes
- OrphanetGeneSet → ETL
- val genatlas_gene_id: (Node) ⇒ Option[String]
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
getLastRunDateFor(ds: DatasetConf)(implicit spark: SparkSession): LocalDateTime
If possible, fetch the last run date time from the dataset passed in argument
If possible, fetch the last run date time from the dataset passed in argument
- ds
dataset
- spark
a spark session
- returns
the last run date or the minDateTime
- Definition Classes
- ETL
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
lastReleaseId(implicit spark: SparkSession): String
- Definition Classes
- ETLP
-
def
load(data: DataFrame, lastRunDateTime: LocalDateTime, currentRunDateTime: LocalDateTime)(implicit spark: SparkSession): DataFrame
Loads the output data into a persistent storage.
Loads the output data into a persistent storage. The output destination can be any of: object store, database or flat files...
- data
output data produced by the transform method.
- spark
an instance of SparkSession
- Definition Classes
- OrphanetGeneSet → ETL
-
val
log: Logger
- Definition Classes
- ETL
-
val
maxDateTime: LocalDateTime
- Definition Classes
- ETL
-
val
minDateTime: LocalDateTime
- Definition Classes
- ETL
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
- val omim_gene_id: (Node) ⇒ Option[String]
- val orphanet_disease_history: DatasetConf
- val orphanet_gene_association: DatasetConf
- def parseProduct6XML(doc: Elem): Seq[OrphanetGeneAssociation]
- def parseProduct9XML(doc: Elem): Seq[OrphanetDiseaseHistory]
-
def
publish()(implicit spark: SparkSession): Unit
OPTIONAL - Contains all actions needed to be done in order to make the data available to users like creating a view with the data.
- val reactome_gene_id: (Node) ⇒ Option[String]
-
def
reset()(implicit spark: SparkSession): Unit
Reset the ETL by removing the destination dataset.
Reset the ETL by removing the destination dataset.
- Definition Classes
- ETL
-
def
run(runSteps: Seq[RunStep] = RunStep.default_load, lastRunDateTime: Option[LocalDateTime] = None, currentRunDateTime: Option[LocalDateTime] = None)(implicit spark: SparkSession): DataFrame
Entry point of the etl - execute this method in order to run the whole ETL
Entry point of the etl - execute this method in order to run the whole ETL
- spark
an instance of SparkSession
- Definition Classes
- ETL
-
def
sampling: PartialFunction[String, (DataFrame) ⇒ DataFrame]
Logic used when the ETL is run as a SAMPLE_LOAD
Logic used when the ETL is run as a SAMPLE_LOAD
- Definition Classes
- ETL
- val swiss_prot_gene_id: (Node) ⇒ Option[String]
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
def
transform(data: Map[String, DataFrame], lastRunDateTime: LocalDateTime, currentRunDateTime: LocalDateTime)(implicit spark: SparkSession): DataFrame
Takes a Map[DatasetConf, DataFrame] as input and apply a set of transformation to it to produce the ETL output.
Takes a Map[DatasetConf, DataFrame] as input and apply a set of transformation to it to produce the ETL output. It is recommended to not read any additional data but to use the extract() method instead to inject input data.
- data
input data
- spark
an instance of SparkSession
- Definition Classes
- OrphanetGeneSet → ETL
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
Deprecated Value Members
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] ) @Deprecated
- Deprecated