Trait

org.sunbird.cloud.storage

IStorageService

Related Doc: package storage

Permalink

trait IStorageService extends AnyRef

Linear Supertypes
AnyRef, Any
Known Subclasses
AzureStorageService, BaseStorageService, GcloudStorageService, S3StorageService
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. IStorageService
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def copyObjects(fromContainer: String, fromKey: String, toContainer: String, toKey: String, isDirectory: Option[Boolean] = Option(false)): Unit

    Permalink

    Copy objects from one container to another container or between different folders within the same container.

    Copy objects from one container to another container or between different folders within the same container. This assumes that the credentials provided have access to both containers.

    fromContainer

    String - The container to copy the object from.

    fromKey

    String - The object prefix to copy from.

    toContainer

    String - The container to copy the object to.

    toKey

    String - The object prefix to copy to.

    isDirectory

    Option[Boolean] - Whether the copy is a file or folder? Defaults to false i.e copy one file.

  2. abstract def deleteObject(container: String, objectKey: String, isDirectory: Option[Boolean] = Option(false)): Unit

    Permalink

    Delete an object from the cloud store

    Delete an object from the cloud store

    container

    String - The container/bucket of the object.

    objectKey

    String - The object key to delete.

    isDirectory

    Option[Boolean] - Whether the object is a directory and need to be deleted recursively? Optional and defaults to false.

  3. abstract def deleteObjects(container: String, objectKeys: List[(String, Boolean)]): Unit

    Permalink

    Delete objects from the cloud store

    Delete objects from the cloud store

    container

    String - The container/bucket of the objects

    objectKeys

    List[(String, Boolean)] - The objects to delete. The tuple contains the object key to be deleted and whether the object is a folder so that recursive delete is triggered.

  4. abstract def download(container: String, objectKey: String, localPath: String, isDirectory: Option[Boolean] = Option(false)): Unit

    Permalink

    Download file/folder from cloud storage

    Download file/folder from cloud storage

    container

    String - The container/bucket of the file

    objectKey

    String - The key/path of the file to download from

    localPath

    String - The local destination path to download to

    isDirectory

    Option[Boolean] - Whether the file is a directory and need to be downloaded recursively? Optional and defaults to false.

  5. abstract def extractArchive(container: String, objectKey: String, toKey: String): Unit

    Permalink

    Remote extract a archived file on cloud storage to a given folder within the same container.

    Remote extract a archived file on cloud storage to a given folder within the same container.

    container

    String - The container/bucket of the archive file.

    objectKey

    String - The blob object archive file

    toKey

    String - The destination folder on the bucket/container to extract to.

  6. abstract def getObject(container: String, objectKey: String, withPayload: Option[Boolean] = Option(false)): Blob

    Permalink

    Get the blob object details

    Get the blob object details

    container

    String - The container/bucket of the file

    objectKey

    String - The key/path of the blob object

    withPayload

    Option[Boolean] - Get payload as well while fetching details. Defaults to false.

    returns

    Blob - The blob object.

  7. abstract def getObjectData(container: String, objectKey: String): Array[String]

    Permalink

    Get the blob object data.

    Get the blob object data.

    container

    String - The container/bucket of the file

    objectKey

    String - The key/path of the blob object

    returns

    Array[String] - object data

  8. abstract def getPaths(container: String, objects: List[Blob]): List[String]

    Permalink

    Get HDFS compatible file paths to be used in tech stack like Spark.

    Get HDFS compatible file paths to be used in tech stack like Spark. For ex: for S3 the file path is prefixed with s3n://<bucket>/<key> and for Azure blob storage it would be wasbs://<container-name>@<storage-account-name>.blob.core.windows.net/<key>/

    container

    String - The container/bucket of the objects

    objects

    List[Blob] - The Blob objects in the given container

    returns

    List[String] - HDFS compatible file paths.

  9. abstract def getSignedURL(container: String, objectKey: String, ttl: Option[Int] = None, permission: Option[String] = Option("r")): String

    Permalink

    Get pre-signed URL to access an object in the cloud store.

    Get pre-signed URL to access an object in the cloud store.

    container

    String - The container/bucket of the file

    objectKey

    String - The key/path of the file to pre-sign

    ttl

    Option[Int] - The ttl/expiry for the pre-signed URL. Defaults to "max.signedurl.ttl" defined in global configuration*

    permission

    String - The permission of pre-signed url the values are w (write), r (read). Defaults to "read".

    returns

    String - The pre-signed url

  10. abstract def getSignedURLV2(container: String, objectKey: String, ttl: Option[Int] = None, permission: Option[String] = Option("r"), contentType: Option[String] = Option("text/plain"), additionalParams: Option[Map[String, String]] = None): String

    Permalink

    Get pre-signed URL to access an object in the cloud store.

    Get pre-signed URL to access an object in the cloud store.

    container

    String - The container/bucket of the file

    objectKey

    String - The key/path of the file to pre-sign

    ttl

    Option[Int] - The ttl/expiry for the pre-signed URL. Defaults to "max.signedurl.ttl" defined in global configuration*

    permission

    String - The permission of pre-signed url the values are w (write), r (read). Defaults to "read".

    contentType

    String - The object type of the content.

    returns

    String - The pre-signed url

  11. abstract def getUri(container: String, _prefix: String, isDirectory: Option[Boolean] = Option(false)): String

    Permalink

    Get the URI of the given prefix

    Get the URI of the given prefix

    returns

    String - URI of the given prefix

  12. abstract def listObjectKeys(container: String, _prefix: String): List[String]

    Permalink

    List object keys from cloud storage for a given prefix.

    List object keys from cloud storage for a given prefix. Similar to listObjects()

    container

    String - The container/bucket

    _prefix

    String - The object prefix to list objects. The prefix can be folder or pattern.

    returns

    List[Blob] - The blob objects for the given prefix.

  13. abstract def listObjects(container: String, prefix: String, withPayload: Option[Boolean] = Option(false)): List[Blob]

    Permalink

    List objects from cloud storage for a given prefix.

    List objects from cloud storage for a given prefix.

    container

    String - The container/bucket

    prefix

    String - The object prefix to list objects. The prefix can be folder or pattern.

    withPayload

    Option[Boolean] - Does the listing of objects include payload as well? Defaults to false

    returns

    List[Blob] - The blob objects for the given prefix.

  14. abstract def put(container: String, content: Array[Byte], objectKey: String, isPublic: Option[Boolean] = Option(false), isDirectory: Option[Boolean] = Option(false), ttl: Option[Int] = None, retryCount: Option[Int] = None): String

    Permalink

    Put a blob in the cloud with the given content data.

    Put a blob in the cloud with the given content data. The difference between this and upload() method is that this method takes in byte array and sets is payload for the object.

    container

    String - The container/bucket to upload the file to.

    content

    String - The byte array of an object.

    objectKey

    String - The destination key/path to upload the file to. If the path exists it will be overwritten.

    isPublic

    Option[Boolean] - Whether the file should have public read access? Optional and defaults to false.

    isDirectory

    Option[Boolean] - Whether the file is a directory and need to upload folder recursively? Optional and defaults to false.

    ttl

    Option[Int] - The ttl/expiry for the file. Optional and default is never expires

    retryCount

    Option[Int] - Number of times the upload will be retried before failing. Defaults to "max.retries" defined in global configuration

    returns

    String - The url of the file/folder uploaded

  15. abstract def searchObjectkeys(container: String, prefix: String, fromDate: Option[String] = None, toDate: Option[String] = None, delta: Option[Int] = None, pattern: String = "yyyy-MM-dd"): List[String]

    Permalink

    Similar to searchObjects().

    Similar to searchObjects(). The only difference is that this method only returns the object keys instead of full blob objects.

    container

    String - The container/bucket.

    prefix

    String - The prefix to search on.

    fromDate

    Option[String] - The date to search from. Optional

    toDate

    Option[String] - The date to search to. Optional

    delta

    Option[Int] - The delta to search from given a from date or to date. Optional. If delta is provided and both fromDate and toDate are empty, the toDate will be defaulted to current date

    pattern

    String - The date pattern of from and to date. Defaulst to "yyyy-MM-dd"

    returns

    List[String] - The object keys

  16. abstract def searchObjects(container: String, prefix: String, fromDate: Option[String] = None, toDate: Option[String] = None, delta: Option[Int] = None, pattern: String = "yyyy-MM-dd"): List[Blob]

    Permalink

    Search for objects for a given prefix and return only keys.

    Search for objects for a given prefix and return only keys. Specifically used for telemetry files as the files are prefixed by sync date. By design the payload is not included in the search. If payload is required use getObject() method after the search is complete.

    // TODO: Provide examples of all combinations of fromDate, toDate and delta

    container

    String - The container/bucket.

    prefix

    String - The prefix to search on.

    fromDate

    Option[String] - The date to search from. Optional

    toDate

    Option[String] - The date to search to. Optional

    delta

    Option[Int] - The delta to search from given a from date or to date. Optional. If delta is provided and both fromDate and toDate are empty, the toDate will be defaulted to current date

    pattern

    String - The date pattern of from and to date. Defaulst to "yyyy-MM-dd"

    returns

    List[Blob] - The object keys

  17. abstract def upload(container: String, file: String, objectKey: String, isDirectory: Option[Boolean] = Option(false), attempt: Option[Int] = Option(1), retryCount: Option[Int] = None, ttl: Option[Int] = None): String

    Permalink

    Upload a file/folder to cloud

    Upload a file/folder to cloud

    container

    String - The container/bucket to upload the file to.

    file

    String - The file path.

    objectKey

    String - The destination key/path to upload the file to. If the path exists it will be overwritten.

    isDirectory

    Option[Boolean] - Whether the file is a directory and need to upload folder recursively?

    retryCount

    Option[Int] - Number of times the upload will be retried before failing. Defaults to global configuration "max.retries"

    ttl

    Option[Int] - The ttl/expiry for the file. Optional and default is never expires

    returns

    String - The url of the file/folder uploaded

  18. abstract def uploadFolder(container: String, file: String, objectKey: String, isPublic: Option[Boolean] = Option(false), ttl: Option[Int] = None, retryCount: Option[Int] = None, attempt: Int = 1)(implicit execution: ExecutionContext): Future[List[String]]

    Permalink

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  7. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  8. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  9. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  10. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  11. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  12. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  13. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  14. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  15. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  16. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  17. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  18. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  19. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped