case class WebserviceFileDataObject(id: DataObjectId, url: String, additionalHeaders: Map[String, String] = Map(), timeouts: Option[HttpTimeoutConfig] = None, readTimeoutMs: Option[Int] = None, authMode: Option[AuthMode] = None, mimeType: Option[String] = None, writeMethod: WebserviceMethod = WebserviceMethod.Post, proxy: Option[HttpProxyConfig] = None, followRedirects: Boolean = false, partitionDefs: Seq[WebservicePartitionDefinition] = Seq(), partitionLayout: Option[String] = None, metadata: Option[DataObjectMetadata] = None)(implicit instanceRegistry: InstanceRegistry) extends FileRefDataObject with CanCreateInputStream with CanCreateOutputStream with SmartDataLakeLogger with Product with Serializable

DataObject to call webservice and return response as InputStream This is implemented as FileRefDataObject because the response is treated as some file content. FileRefDataObjects support partitioned data. For a WebserviceFileDataObject partitions are mapped as query parameters to create query string. All possible query parameter values must be given in configuration.

partitionDefs

list of partitions with list of possible values for every entry

partitionLayout

definition of partitions in query string. Use %<partitionColName>% as placeholder for partition column value in layout.

Annotations
@Scaladoc()
Linear Supertypes
Serializable, Serializable, Product, Equals, CanCreateOutputStream, CanCreateInputStream, FileRefDataObject, FileDataObject, CanHandlePartitions, DataObject, AtlasExportable, SmartDataLakeLogger, ParsableFromConfig[DataObject], SdlConfigObject, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. WebserviceFileDataObject
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. CanCreateOutputStream
  7. CanCreateInputStream
  8. FileRefDataObject
  9. FileDataObject
  10. CanHandlePartitions
  11. DataObject
  12. AtlasExportable
  13. SmartDataLakeLogger
  14. ParsableFromConfig
  15. SdlConfigObject
  16. AnyRef
  17. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new WebserviceFileDataObject(id: DataObjectId, url: String, additionalHeaders: Map[String, String] = Map(), timeouts: Option[HttpTimeoutConfig] = None, readTimeoutMs: Option[Int] = None, authMode: Option[AuthMode] = None, mimeType: Option[String] = None, writeMethod: WebserviceMethod = WebserviceMethod.Post, proxy: Option[HttpProxyConfig] = None, followRedirects: Boolean = false, partitionDefs: Seq[WebservicePartitionDefinition] = Seq(), partitionLayout: Option[String] = None, metadata: Option[DataObjectMetadata] = None)(implicit instanceRegistry: InstanceRegistry)

    partitionDefs

    list of partitions with list of possible values for every entry

    partitionLayout

    definition of partitions in query string. Use %<partitionColName>% as placeholder for partition column value in layout.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val additionalHeaders: Map[String, String]
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def atlasName: String
    Definition Classes
    DataObject → AtlasExportable
  7. def atlasQualifiedName(prefix: String): String
    Definition Classes
    AtlasExportable
  8. val authMode: Option[AuthMode]
  9. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  10. def createInputStream(query: String)(implicit context: ActionPipelineContext): InputStream

    Same as getResponse, but returns response as InputStream

    Same as getResponse, but returns response as InputStream

    query

    it should be possible to define the partition to read as query string, but this is not yet implemented

    Definition Classes
    WebserviceFileDataObject → CanCreateInputStream
    Annotations
    @Scaladoc()
  11. def createOutputStream(path: String, overwrite: Boolean)(implicit context: ActionPipelineContext): OutputStream

    path

    is ignored for webservices

    overwrite

    is ignored for webservices

    returns

    outputstream that writes to WebService once it's closed

    Definition Classes
    WebserviceFileDataObject → CanCreateOutputStream
    Annotations
    @Scaladoc()
  12. def deleteAll(implicit context: ActionPipelineContext): Unit

    Delete all data.

    Delete all data. This is used to implement SaveMode.Overwrite.

    Definition Classes
    FileRefDataObject
    Annotations
    @Scaladoc()
  13. def deleteFileRefs(fileRefs: Seq[FileRef])(implicit context: ActionPipelineContext): Unit

    Delete given files.

    Delete given files. This is used to cleanup files after they are processed.

    Definition Classes
    FileRefDataObject
    Annotations
    @Scaladoc()
  14. def endWritingOutputStreams(partitionValues: Seq[PartitionValues])(implicit context: ActionPipelineContext): Unit

    This is called after all output streams have been written.

    This is called after all output streams have been written. It is used for e.g. making sure empty partitions are created as well.

    Definition Classes
    WebserviceFileDataObject → CanCreateOutputStream
  15. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. def expectedPartitionsCondition: Option[String]

    Definition of partitions that are expected to exists.

    Definition of partitions that are expected to exists. This is used to validate that partitions being read exists and don't return no data. Define a Spark SQL expression that is evaluated against a PartitionValues instance and returns true or false example: "elements['yourColName'] > 2017"

    returns

    true if partition is expected to exist.

    Definition Classes
    WebserviceFileDataObject → CanHandlePartitions
  17. def extractPartitionValuesFromPath(filePath: String)(implicit context: ActionPipelineContext): PartitionValues

    Extract partition values from a given file path

    Extract partition values from a given file path

    Attributes
    protected
    Definition Classes
    FileRefDataObject
    Annotations
    @Scaladoc()
  18. def factory: FromConfigFactory[DataObject]

    Returns the factory that can parse this type (that is, type CO).

    Returns the factory that can parse this type (that is, type CO).

    Typically, implementations of this method should return the companion object of the implementing class. The companion object in turn should implement FromConfigFactory.

    returns

    the factory (object) for this class.

    Definition Classes
    WebserviceFileDataObject → ParsableFromConfig
  19. val fileName: String

    Definition of fileName.

    Definition of fileName. Default is an asterix to match everything. This is concatenated with the partition layout to search for files.

    Definition Classes
    FileRefDataObject
    Annotations
    @Scaladoc()
  20. val followRedirects: Boolean
  21. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  22. def getConnection[T <: Connection](connectionId: ConnectionId)(implicit registry: InstanceRegistry, ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

    Handle class cast exception when getting objects from instance registry

    Handle class cast exception when getting objects from instance registry

    Attributes
    protected
    Definition Classes
    DataObject
    Annotations
    @Scaladoc()
  23. def getConnectionReg[T <: Connection](connectionId: ConnectionId, registry: InstanceRegistry)(implicit ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T
    Attributes
    protected
    Definition Classes
    DataObject
  24. def getFileRefs(partitionValues: Seq[PartitionValues])(implicit context: ActionPipelineContext): Seq[FileRef]

    For WebserviceFileDataObject, every partition is mapped to one FileRef

    For WebserviceFileDataObject, every partition is mapped to one FileRef

    partitionValues

    List of partition values to be filtered. If empty all files in root path of DataObject will be listed.

    returns

    List of FileRefs

    Definition Classes
    WebserviceFileDataObject → FileRefDataObject
    Annotations
    @Scaladoc()
  25. def getPartitionString(partitionValues: PartitionValues)(implicit context: ActionPipelineContext): Option[String]

    get partition values formatted by partition layout

    get partition values formatted by partition layout

    Definition Classes
    FileRefDataObject
    Annotations
    @Scaladoc()
  26. def getPath(implicit context: ActionPipelineContext): String

    Method for subclasses to override the base path for this DataObject.

    Method for subclasses to override the base path for this DataObject. This is for instance needed if pathPrefix is defined in a connection.

    Definition Classes
    FileRefDataObject
    Annotations
    @Scaladoc()
  27. def getResponse(query: Option[String] = None): Array[Byte]

    Calls webservice and returns response

    Calls webservice and returns response

    query

    optional URL with replaced placeholders to call

    returns

    Response as Array[Byte]

    Annotations
    @Scaladoc()
  28. def getSearchPaths(partitionValues: Seq[PartitionValues])(implicit context: ActionPipelineContext): Seq[(PartitionValues, String)]

    prepare paths to be searched

    prepare paths to be searched

    Attributes
    protected
    Definition Classes
    FileRefDataObject
    Annotations
    @Scaladoc()
  29. def housekeepingMode: Option[HousekeepingMode]

    Configure a housekeeping mode to e.g cleanup, archive and compact partitions.

    Configure a housekeeping mode to e.g cleanup, archive and compact partitions. Default is None.

    Definition Classes
    DataObject
    Annotations
    @Scaladoc()
  30. val id: DataObjectId

    A unique identifier for this instance.

    A unique identifier for this instance.

    Definition Classes
    WebserviceFileDataObject → DataObject → SdlConfigObject
  31. implicit val instanceRegistry: InstanceRegistry
  32. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  33. def listPartitions(implicit context: ActionPipelineContext): Seq[PartitionValues]

    List partition values defined for this web service.

    List partition values defined for this web service. Note that this is a fixed list.

    Definition Classes
    WebserviceFileDataObject → CanHandlePartitions
    Annotations
    @Scaladoc()
  34. lazy val logger: Logger
    Attributes
    protected
    Definition Classes
    SmartDataLakeLogger
    Annotations
    @transient()
  35. val metadata: Option[DataObjectMetadata]

    Additional metadata for the DataObject

    Additional metadata for the DataObject

    Definition Classes
    WebserviceFileDataObject → DataObject
  36. val mimeType: Option[String]
  37. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  38. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  39. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  40. val partitionDefs: Seq[WebservicePartitionDefinition]
  41. val partitionLayout: Option[String]

    Definition of partition layout use %<partitionColName>% as placeholder and * for globs in layout Note: if you have globs in partition layout, it's not possible to write files to this DataObject Note: if this is a directory, you must add a final backslash to the partition layout

    Definition of partition layout use %<partitionColName>% as placeholder and * for globs in layout Note: if you have globs in partition layout, it's not possible to write files to this DataObject Note: if this is a directory, you must add a final backslash to the partition layout

    Definition Classes
    WebserviceFileDataObject → FileRefDataObject
  42. def partitions: Seq[String]

    Definition of partition columns

    Definition of partition columns

    Definition Classes
    WebserviceFileDataObject → CanHandlePartitions
  43. def path: String

    No root path needed for Webservice.

    No root path needed for Webservice. It can be included in webserviceOptions.url.

    Definition Classes
    WebserviceFileDataObject → FileDataObject
    Annotations
    @Scaladoc()
  44. def postResponse(body: Array[Byte], query: Option[String] = None): Array[Byte]

    Calls webservice POST method with binary data as body

    Calls webservice POST method with binary data as body

    body

    post body as Byte Array, type will be determined by Tika

    query

    optional URL with replaced placeholders to call

    returns

    Response as Array[Byte]

    Annotations
    @Scaladoc()
  45. def postWrite(partitionValues: Seq[PartitionValues])(implicit context: ActionPipelineContext): Unit

    Runs operations after writing to DataObject

    Runs operations after writing to DataObject

    Definition Classes
    WebserviceFileDataObject → DataObject
  46. def prepare(implicit context: ActionPipelineContext): Unit

    Prepare & test DataObject's prerequisits

    Prepare & test DataObject's prerequisits

    This runs during the "prepare" operation of the DAG.

    Definition Classes
    WebserviceFileDataObject → FileDataObject → DataObject
  47. val proxy: Option[HttpProxyConfig]
  48. val readTimeoutMs: Option[Int]
  49. def relativizePath(filePath: String)(implicit context: ActionPipelineContext): String

    Make a given path relative to this DataObjects base path

    Make a given path relative to this DataObjects base path

    Definition Classes
    WebserviceFileDataObject → FileDataObject
  50. val saveMode: SDLSaveMode

    Overwrite or Append new data.

    Overwrite or Append new data. When writing partitioned data, this applies only to partitions concerned.

    Definition Classes
    WebserviceFileDataObject → FileRefDataObject
  51. val separator: Char

    default separator for paths

    default separator for paths

    Attributes
    protected
    Definition Classes
    FileDataObject
    Annotations
    @Scaladoc()
  52. def startWritingOutputStreams(partitionValues: Seq[PartitionValues])(implicit context: ActionPipelineContext): Unit

    This is called before any output stream is created to initialize writing.

    This is called before any output stream is created to initialize writing. It is used to apply SaveMode, e.g. deleting existing partitions.

    Definition Classes
    WebserviceFileDataObject → CanCreateOutputStream
  53. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  54. val tika: Tika
  55. val timeouts: Option[HttpTimeoutConfig]
  56. def toStringShort: String
    Definition Classes
    DataObject
  57. def translateFileRefs(fileRefs: Seq[FileRef])(implicit context: ActionPipelineContext): Seq[FileRefMapping]

    Given some FileRef for another DataObject, translate the paths to the root path of this DataObject

    Given some FileRef for another DataObject, translate the paths to the root path of this DataObject

    Definition Classes
    FileRefDataObject
    Annotations
    @Scaladoc()
  58. val url: String
  59. def validateSchemaHasPartitionCols(df: DataFrame, role: String): Unit

    Validate the schema of a given Spark Data Frame df that it contains the specified partition columns

    Validate the schema of a given Spark Data Frame df that it contains the specified partition columns

    df

    The data frame to validate.

    role

    role used in exception message. Set to read or write.

    Definition Classes
    CanHandlePartitions
    Annotations
    @Scaladoc()
    Exceptions thrown

    SchemaViolationException if the partitions columns are not included.

  60. def validateSchemaHasPrimaryKeyCols(df: DataFrame, primaryKeyCols: Seq[String], role: String): Unit

    Validate the schema of a given Spark Data Frame df that it contains the specified primary key columns

    Validate the schema of a given Spark Data Frame df that it contains the specified primary key columns

    df

    The data frame to validate.

    role

    role used in exception message. Set to read or write.

    Definition Classes
    CanHandlePartitions
    Annotations
    @Scaladoc()
    Exceptions thrown

    SchemaViolationException if the partitions columns are not included.

  61. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  62. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  63. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  64. val writeMethod: WebserviceMethod

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from CanCreateOutputStream

Inherited from CanCreateInputStream

Inherited from FileRefDataObject

Inherited from FileDataObject

Inherited from CanHandlePartitions

Inherited from DataObject

Inherited from AtlasExportable

Inherited from SmartDataLakeLogger

Inherited from ParsableFromConfig[DataObject]

Inherited from SdlConfigObject

Inherited from AnyRef

Inherited from Any

Ungrouped