Packages

abstract class WaimakApp[E <: Env with WaimakEnv] extends SparkApp[E]

This is a SparkApp specifically for applications using Waimak

E

the type of the WaimakEnv implementation (must be a case class)

Linear Supertypes
SparkApp[E], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. WaimakApp
  2. SparkApp
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new WaimakApp()(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[E])

Abstract Value Members

  1. abstract def confDefaults(env: E): Map[String, String]

    Default Spark configuration values to use for the application

    Default Spark configuration values to use for the application

    env

    the environment

    returns

    a map containing default Spark configuration

    Definition Classes
    SparkApp
  2. abstract def flow(emptyFlow: SparkDataFlow, env: E): SparkDataFlow

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def cleanupEnv(sparkSession: SparkSession, envPrefix: String): Unit

    Cleans up the environment associated with this application

    Cleans up the environment associated with this application

    sparkSession

    the SparkSession

    envPrefix

    the prefix for keys in the SparkConf needed by the Env implementation

    Definition Classes
    SparkApp
  6. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )
  7. def createEnv(sparkSession: SparkSession, envPrefix: String): Unit

    Create the environment associated with this application

    Create the environment associated with this application

    sparkSession

    the SparkSession

    envPrefix

    the prefix for keys in the SparkConf needed by the Env implementation

    Definition Classes
    SparkApp
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. def parseEnv(sparkSession: SparkSession, envPrefix: String): E

    Parses configuration in the SparkSession into the environment case class (type E)

    Parses configuration in the SparkSession into the environment case class (type E)

    sparkSession

    the SparkSession

    envPrefix

    the prefix for keys in the SparkConf needed by the Env implementation

    returns

    a parsed case class of type E

    Definition Classes
    SparkApp
  18. def run(sparkSession: SparkSession, env: E): Unit

    Run the application for given environment and configuration case classes

    Run the application for given environment and configuration case classes

    sparkSession

    the SparkSession

    env

    the environment

    Attributes
    protected
    Definition Classes
    WaimakApp → SparkApp
  19. def runSparkApp(sparkSession: SparkSession, envPrefix: String): Unit

    Runs the application

    Runs the application

    N.B does not create the environment - use createEnv

    sparkSession

    the SparkSession

    envPrefix

    the prefix for keys in the SparkConf needed by the Env implementation

    Definition Classes
    SparkApp
  20. def runWithEnv(env: E, sparkSession: SparkSession): Unit
    Attributes
    protected
    Definition Classes
    SparkApp
  21. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )

Inherited from SparkApp[E]

Inherited from AnyRef

Inherited from Any

Ungrouped