Packages

case class DAG[N <: DAGNode] extends SmartDataLakeLogger with Product with Serializable

A generic directed acyclic graph (DAG) consisting of DAGNodes interconnected with directed DAGEdges.

This DAG can have multiple start nodes and multiple end nodes as well as disconnected parts.

Annotations
@Scaladoc()
Linear Supertypes
Serializable, Serializable, Product, Equals, SmartDataLakeLogger, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DAG
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. SmartDataLakeLogger
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def buildTaskGraph[A <: DAGResult](eventListener: DAGEventListener[N])(operation: (DAGNode, Seq[A]) ⇒ Seq[A])(implicit scheduler: Scheduler): Task[Seq[Try[A]]]

    Build a single task that is a combination of node computations (node tasks) executed in the topological order defined by the DAG.

    Build a single task that is a combination of node computations (node tasks) executed in the topological order defined by the DAG.

    Monix tasks is a library for lazy cancelable futures

    eventListener

    A instance of DAGEventListener to be notified about progress of DAG execution

    operation

    A function that computes the result (DAGResult) for the current node, given the result of its predecessors given.

    scheduler

    The Scheduler to use for Tasks.

    Annotations
    @Scaladoc()
    Note

    This method does not trigger any execution but builds a complex collection of tasks and synchronization boundaries that specify a correct order of execution as defined by the DAG. The computation only runs when the returning task is scheduled for execution.

    See also

    https://medium.com/@sderosiaux/are-scala-futures-the-past-69bd62b9c001

  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  7. val endNodes: Seq[DAGNode]
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  10. def getNodes: Seq[N]

    Get the nodes of this DAG Note: we must filter nodes which are not of the preliminary node type of this DAG, e.g.

    Get the nodes of this DAG Note: we must filter nodes which are not of the preliminary node type of this DAG, e.g. InitDAGNode

    returns

    list of nodes of the preliminary node type of this DAG

    Annotations
    @Scaladoc()
  11. def getResultTask[A <: DAGResult](tasks: Map[NodeId, Task[Try[Seq[A]]]], nodeId: NodeId, resultId: String): Task[Try[A]]

    Create a task that fetches a specific DAGResult produced by the node with id nodeId.

    Create a task that fetches a specific DAGResult produced by the node with id nodeId.

    A

    The result type - supertype DAGResult ensures it has a resultId defined.

    tasks

    A map of tasks that compute (future) results indexed by node.

    nodeId

    The id of the producing node.

    resultId

    The id of the result to search among all nodes results.

    returns

    The task that computes the result specified by nodeId and resultId.

    Annotations
    @Scaladoc()
  12. val incomingEdgesMap: Map[NodeId, Seq[DAGEdge]]
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. lazy val logger: Logger
    Attributes
    protected
    Definition Classes
    SmartDataLakeLogger
    Annotations
    @transient()
  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  18. def render(nodeToString: (DAGNode) ⇒ String): String

    Create text representation of the graph by using an ASCII graph layout library

    Create text representation of the graph by using an ASCII graph layout library

    Annotations
    @Scaladoc()
  19. val sortedNodes: Seq[DAGNode]
  20. val startNodes: Seq[DAGNode]
  21. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  22. def trySeqToSeqTry[A](trySeq: Try[Seq[A]]): Seq[Try[A]]

    Convert a Try of a Result-List to a List of Result-Try's

    Convert a Try of a Result-List to a List of Result-Try's

    Annotations
    @Scaladoc()
  23. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  25. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from SmartDataLakeLogger

Inherited from AnyRef

Inherited from Any

Ungrouped