Packages

trait SnapshotManagement extends AnyRef

Manages the creation, computation, and access of Snapshot's for Delta tables. Responsibilities include:

  • Figuring out the set of files that are required to compute a specific version of a table
  • Updating and exposing the latest snapshot of the Delta table in a thread-safe manner
Self Type
DeltaLog
Linear Supertypes
AnyRef, Any
Known Subclasses
DeltaLog
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SnapshotManagement
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. def createSnapshot(initSegment: LogSegment, minFileRetentionTimestamp: Long, checkpointMetadataOptHint: Option[CheckpointMetaData]): Snapshot
    Attributes
    protected
  7. def createSnapshotFromGivenOrEquivalentLogSegment(initSegment: LogSegment)(snapshotCreator: (LogSegment) => Snapshot): Snapshot

    Create a Snapshot from the given LogSegment.

    Create a Snapshot from the given LogSegment. If failing to create the snapshot, we will search an equivalent LogSegment using a different checkpoint and retry up to DeltaSQLConf.DELTA_SNAPSHOT_LOADING_MAX_RETRIES times.

    Attributes
    protected
  8. val currentSnapshot: CapturedSnapshot
    Attributes
    protected
    Annotations
    @volatile()
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  12. def getCheckpointMetadataForSegment(segment: LogSegment, checkpointMetadataOptHint: Option[CheckpointMetaData]): Option[CheckpointMetaData]

    Returns the CheckpointMetaData for the given LogSegment.

    Returns the CheckpointMetaData for the given LogSegment. If the passed checkpointMetadataOptHint matches the segment, then it is returned directly.

    Attributes
    protected
  13. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def getLogSegmentForVersion(startCheckpoint: Option[Long], versionToLoad: Option[Long], files: Option[Array[FileStatus]]): Option[LogSegment]

    Helper function for the getLogSegmentForVersion above.

    Helper function for the getLogSegmentForVersion above. Called with a provided files list, and will then try to construct a new LogSegment using that.

    Attributes
    protected
  15. def getLogSegmentForVersion(startCheckpoint: Option[Long], versionToLoad: Option[Long] = None): Option[LogSegment]

    Get a list of files that can be used to compute a Snapshot at version versionToLoad, If versionToLoad is not provided, will generate the list of files that are needed to load the latest version of the Delta table.

    Get a list of files that can be used to compute a Snapshot at version versionToLoad, If versionToLoad is not provided, will generate the list of files that are needed to load the latest version of the Delta table. This method also performs checks to ensure that the delta files are contiguous.

    startCheckpoint

    A potential start version to perform the listing of the DeltaLog, typically that of a known checkpoint. If this version's not provided, we will start listing from version 0.

    versionToLoad

    A specific version to load. Typically used with time travel and the Delta streaming source. If not provided, we will try to load the latest version of the table.

    returns

    Some LogSegment to build a Snapshot if files do exist after the given startCheckpoint. None, if the directory was missing or empty.

    Attributes
    protected
  16. def getLogSegmentFrom(startingCheckpoint: Option[CheckpointMetaData]): Option[LogSegment]

    Get the LogSegment that will help in computing the Snapshot of the table at DeltaLog initialization, or None if the directory was empty/missing.

    Get the LogSegment that will help in computing the Snapshot of the table at DeltaLog initialization, or None if the directory was empty/missing.

    startingCheckpoint

    A checkpoint that we can start our listing from

    Attributes
    protected
  17. def getSnapshotAt(version: Long, commitTimestamp: Option[Long] = None, lastCheckpointHint: Option[CheckpointInstance] = None): Snapshot

    Get the snapshot at version.

  18. def getSnapshotAtInit(lastCheckpointOpt: Option[CheckpointMetaData]): CapturedSnapshot

    Load the Snapshot for this Delta table at initialization.

    Load the Snapshot for this Delta table at initialization. This method uses the lastCheckpoint file as a hint on where to start listing the transaction log directory. If the _delta_log directory doesn't exist, this method will return an InitialSnapshot.

    Attributes
    protected
  19. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  20. def installSnapshotInternal(previousSnapshot: Snapshot, segmentOpt: Option[LogSegment], updateTimestamp: Long, isAsync: Boolean): Snapshot

    Install the provided segmentOpt as the currentSnapshot on the cluster

  21. def isDeltaCommitOrCheckpointFile(path: Path): Boolean

    Returns true if the path is delta log files.

    Returns true if the path is delta log files. Delta log files can be delta commit file (e.g., 000000000.json), or checkpoint file. (e.g., 000000001.checkpoint.00001.00003.parquet)

    path

    Path of a file

    returns

    Boolean Whether the file is delta log files

    Attributes
    protected
  22. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  23. def listFrom(startVersion: Long): Iterator[FileStatus]

    Get an iterator of files in the _delta_log directory starting with the startVersion.

    Get an iterator of files in the _delta_log directory starting with the startVersion.

    Attributes
    protected
  24. def listFromOrNone(startVersion: Long): Option[Iterator[FileStatus]]

    Returns an iterator containing a list of files found from the provided path

    Returns an iterator containing a list of files found from the provided path

    Attributes
    protected
  25. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  26. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  27. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  28. def replaceSnapshot(newSnapshot: Snapshot, updateTimestamp: Long): Unit

    Replace the given snapshot with the provided one.

    Replace the given snapshot with the provided one.

    Attributes
    protected
  29. def snapshot: Snapshot

    Returns the current snapshot.

    Returns the current snapshot. Note this does not automatically update().

  30. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  31. def toString(): String
    Definition Classes
    AnyRef → Any
  32. def update(stalenessAcceptable: Boolean = false, checkIfUpdatedSinceTs: Option[Long] = None): Snapshot

    Update ActionLog by applying the new delta files if any.

    Update ActionLog by applying the new delta files if any.

    stalenessAcceptable

    Whether we can accept working with a stale version of the table. If the table has surpassed our staleness tolerance, we will update to the latest state of the table synchronously. If staleness is acceptable, and the table hasn't passed the staleness tolerance, we will kick off a job in the background to update the table state, and can return a stale snapshot in the meantime.

    checkIfUpdatedSinceTs

    Skip the update if we've already updated the snapshot since the specified timestamp.

  33. def updateInternal(isAsync: Boolean): Snapshot

    Queries the store for new delta files and applies them to the current state.

    Queries the store for new delta files and applies them to the current state. Note: the caller should hold deltaLogLock before calling this method.

    Attributes
    protected
  34. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  35. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  36. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped