class AuditTableFile extends AuditTable with Logging

Implementation of the AuditTable which is backed up by append only block storage like HDFS.

Created by Alexei Perelighin on 2018/03/03

Linear Supertypes
Logging, AuditTable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. AuditTableFile
  2. Logging
  3. AuditTable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new AuditTableFile(tableInfo: AuditTableInfo, regions: Seq[AuditTableRegionInfo], storageOps: FileStorageOps, baseFolder: Path, newRegionID: (AuditTableFile) ⇒ String)

    tableInfo

    static details about the table, with custom metadata

    regions

    list of region details

    storageOps

    object that actually interacts with the physical storage

    baseFolder

    parent folder which contains folders with table names

    newRegionID

    function that generates region ids

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def activeRegionIDs(): Option[Seq[String]]

    Returns regions ids for all active regions.

  5. def allBetween(from: Option[Timestamp], to: Option[Timestamp]): Option[Dataset[_]]

    Include all records between the given timestamps.

    Include all records between the given timestamps.

    returns

    if no data in storage layer, return None

    Definition Classes
    AuditTableFile → AuditTable
  6. def append(ds: Dataset[_], lastUpdated: Column, appendTS: Timestamp): Try[(AuditTable, Long)]

    Appends a new set of records to the audit table.

    Appends a new set of records to the audit table.

    Fails when is called second time on same instance.

    ds

    records to append

    lastUpdated

    column that returns java.sql.Timestamp that will be used for de-duplication on the primary keys

    appendTS

    timestamp of when the append has happened. It will not be used for de-duplications

    returns

    (new state of the AuditTable, count of appended records) or error

    Definition Classes
    AuditTableFile → AuditTable
  7. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  8. val baseFolder: Path
  9. def calcRegionStats(ds: Dataset[_]): (Long, Timestamp)
    Attributes
    protected
  10. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  11. val coldPath: Path
    Attributes
    protected
  12. def compact(compactTS: Timestamp, trashMaxAge: Duration, smallRegionRowThreshold: Long, compactionPartitioner: CompactionPartitioner, recompactAll: Boolean): Try[AuditTable]

    Request optimisation of the storage layer.

    Request optimisation of the storage layer.

    Fails when is called second time on same instance.

    compactTS

    timestamp of when the compaction is requested, will not be used for any filtering of the data

    trashMaxAge

    Maximum age of old region files kept in the .Trash folder after a compaction has happened.

    smallRegionRowThreshold

    the row number threshold to use for determining small regions to be compacted. Default is 50000000

    compactionPartitioner

    a partitioner object that dictates how many partitions should be generated for a given region

    recompactAll

    Whether to recompact all regions regardless of size (i.e. ignore smallRegionRowThreshold)

    returns

    new state of the AuditTable

    Definition Classes
    AuditTableFile → AuditTable
  13. def compactRegions(toCompact: Seq[AuditTableRegionInfo], compactTS: Timestamp, compactionPartitioner: CompactionPartitioner): Try[AuditTableFile]
    Attributes
    protected
  14. def deduplicate(ds: Dataset[_]): Dataset[_]
  15. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  17. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  18. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  19. def getLatestTimestamp(): Option[Timestamp]

    Returns latest timestamp of records stored in the audit table.

    Returns latest timestamp of records stored in the audit table.

    Definition Classes
    AuditTableFile → AuditTable
  20. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  21. val hotPath: Path
    Attributes
    protected
  22. def initNewTable(): Try[AuditTableFile.this.type]

    Initializes audit table in the storage layer.

    Initializes audit table in the storage layer. It will also persist all of the metadata (name, primary keys, custom meta) to the storage layer.

    returns

    new state of the table or error

    Definition Classes
    AuditTableFile → AuditTable
  23. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  24. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  25. def logAndReturn[A](a: A, msg: String, level: Level): A
    Definition Classes
    Logging
  26. def logAndReturn[A](a: A, message: (A) ⇒ String, level: Level): A
    Definition Classes
    Logging
  27. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  34. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def markToUpdate(): Unit

    Each function that modifies the state of the storage layer must call this function in the first line.

    Each function that modifies the state of the storage layer must call this function in the first line. As audit's table state can be modified only once.

    Attributes
    protected
  39. def meta: Map[String, String]

    Custom attributes assigned by the client application during table creation.

    Custom attributes assigned by the client application during table creation.

    Definition Classes
    AuditTableFile → AuditTable
  40. val metaBasePath: Path
    Attributes
    protected
  41. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  42. val newRegionID: (AuditTableFile) ⇒ String
  43. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  44. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  45. val regionInfoBasePath: Path
    Attributes
    protected[storage]
  46. val regions: Seq[AuditTableRegionInfo]
    Definition Classes
    AuditTableFile → AuditTable
  47. def regionsToCompact(smallRegionRowThreshold: Long, recompactAll: Boolean): Seq[AuditTableRegionInfo]
    Attributes
    protected
  48. def snapshot(ts: Timestamp): Option[Dataset[_]]

    Generates snapshot that contains only the latest records for the given timestamp.

    Generates snapshot that contains only the latest records for the given timestamp. De-duplication happens on the primary keys.

    ts

    use records that are closest to this timestamp

    returns

    if no data in storage layer, return None

    Definition Classes
    AuditTableFile → AuditTable
  49. val storageOps: FileStorageOps
  50. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  51. val tableInfo: AuditTableInfo
  52. def tableName: String

    Name of the table.

    Name of the table.

    Definition Classes
    AuditTableFile → AuditTable
  53. val tablePath: Path
    Attributes
    protected[storage]
  54. def toString(): String
    Definition Classes
    AnyRef → Any
  55. def updateTableInfo(tableInfo: AuditTableInfo): Try[AuditTable]

    Update the metadata for this table

    Update the metadata for this table

    tableInfo

    the new metadata

    returns

    new state of the AuditTable

    Definition Classes
    AuditTableFile → AuditTable
  56. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  57. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  58. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  59. var wasModified: Boolean

    Not thread safe.

    Not thread safe. Protection against using mutator functions more than one time.

    Attributes
    protected

Inherited from Logging

Inherited from AuditTable

Inherited from AnyRef

Inherited from Any

Ungrouped