Packages

case class DeltaSourceOffset(sourceVersion: Long, reservoirId: String, reservoirVersion: Long, index: Long, isStartingVersion: Boolean) extends Offset with Product with Serializable

Tracks how far we processed in when reading changes from the DeltaLog.

Note this class retains the naming of Reservoir to maintain compatibility with serialized offsets from the beta period.

sourceVersion

The version of serialization that this offset is encoded with.

reservoirId

The id of the table we are reading from. Used to detect misconfiguration when restarting a query.

reservoirVersion

The version of the table that we are current processing.

index

The index in the sequence of AddFiles in this version. Used to break large commits into multiple batches. This index is created by sorting on modificationTimestamp and path.

isStartingVersion

Whether this offset denotes a query that is starting rather than processing changes. When starting a new query, we first process all data present in the table at the start and then move on to processing new data that has arrived.

Linear Supertypes
Serializable, Product, Equals, Offset, Offset, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaSourceOffset
  2. Serializable
  3. Product
  4. Equals
  5. Offset
  6. Offset
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new DeltaSourceOffset(sourceVersion: Long, reservoirId: String, reservoirVersion: Long, index: Long, isStartingVersion: Boolean)

    sourceVersion

    The version of serialization that this offset is encoded with.

    reservoirId

    The id of the table we are reading from. Used to detect misconfiguration when restarting a query.

    reservoirVersion

    The version of the table that we are current processing.

    index

    The index in the sequence of AddFiles in this version. Used to break large commits into multiple batches. This index is created by sorting on modificationTimestamp and path.

    isStartingVersion

    Whether this offset denotes a query that is starting rather than processing changes. When starting a new query, we first process all data present in the table at the start and then move on to processing new data that has arrived.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. def compare(otherOffset: DeltaSourceOffset): Int

    Compare two DeltaSourceOffsets which are on the same table and source version.

    Compare two DeltaSourceOffsets which are on the same table and source version.

    returns

    0 for equivalent offsets. negative if this offset is less than otherOffset. Positive if this offset is greater than otherOffset

  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: AnyRef): Boolean
    Definition Classes
    Offset → AnyRef → Any
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  10. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def hashCode(): Int
    Definition Classes
    Offset → AnyRef → Any
  12. val index: Long
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. val isStartingVersion: Boolean
  15. def json(): String
    Definition Classes
    DeltaSourceOffset → Offset
  16. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  17. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. def productElementNames: Iterator[String]
    Definition Classes
    Product
  20. val reservoirId: String
  21. val reservoirVersion: Long
  22. val sourceVersion: Long
  23. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  24. def toString(): String
    Definition Classes
    Offset → AnyRef → Any
  25. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  26. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  27. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from Offset

Inherited from Offset

Inherited from AnyRef

Inherited from Any

Ungrouped