Packages

c

com.nvidia.spark.rapids.RapidsDiskStore

RapidsDiskColumnarBatch

class RapidsDiskColumnarBatch extends RapidsDiskBuffer with RapidsHostBatchBuffer

A RapidsDiskBuffer that should remain in the host, producing host-backed ColumnarBatch if the caller invokes getHostColumnarBatch, but not producing anything on the device.

Linear Supertypes
RapidsHostBatchBuffer, RapidsDiskBuffer, RapidsDiskStore.RapidsBufferBase, RapidsBuffer, AutoCloseable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. RapidsDiskColumnarBatch
  2. RapidsHostBatchBuffer
  3. RapidsDiskBuffer
  4. RapidsBufferBase
  5. RapidsBuffer
  6. AutoCloseable
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new RapidsDiskColumnarBatch(id: RapidsBufferId, fileOffset: Long, size: Long, uncompressedSize: Long, meta: TableMeta, spillPriority: Long)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def addReference(): Boolean

    Try to add a reference to this buffer to acquire it.

    Try to add a reference to this buffer to acquire it.

    returns

    true if the reference was added or false if this buffer is no longer valid

    Definition Classes
    RapidsBufferBase → RapidsBuffer
    Note

    The close method must be called for every successfully obtained reference.

  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  7. def close(): Unit

    close() is called by client code to decrease the ref count of this RapidsBufferBase.

    close() is called by client code to decrease the ref count of this RapidsBufferBase. In the off chance that by the time close is invoked, the buffer was freed (not valid) then this close call winds up freeing the resources of the rapids buffer.

    Definition Classes
    RapidsDiskBuffer → RapidsBufferBase → AutoCloseable
  8. def columnarBatchFromDeviceBuffer(devBuffer: DeviceMemoryBuffer, sparkTypes: Array[DataType]): ColumnarBatch
    Attributes
    protected
    Definition Classes
    RapidsBufferBase
  9. def copyToMemoryBuffer(srcOffset: Long, dst: MemoryBuffer, dstOffset: Long, length: Long, stream: Stream): Unit

    Copy the content of this buffer into the specified memory buffer, starting from the given offset.

    Copy the content of this buffer into the specified memory buffer, starting from the given offset.

    srcOffset

    offset to start copying from.

    dst

    the memory buffer to copy into.

    dstOffset

    offset to copy into.

    length

    number of bytes to copy.

    stream

    CUDA stream to use

    Definition Classes
    RapidsBufferBase → RapidsBuffer
  10. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  12. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. def free(): Unit

    Mark the buffer as freed and no longer valid.

    Mark the buffer as freed and no longer valid. This is called by the store when removing a buffer (it is no longer tracked).

    Definition Classes
    RapidsBufferBase → RapidsBuffer
    Note

    The resources may not be immediately released if the buffer has outstanding references. In that case the resources will be released when the reference count reaches zero.

  14. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. def getColumnarBatch(sparkTypes: Array[DataType]): ColumnarBatch

    Get the columnar batch within this buffer.

    Get the columnar batch within this buffer. The caller must have successfully acquired the buffer beforehand.

    sparkTypes

    the spark data types the batch should have

    Definition Classes
    RapidsDiskColumnarBatch → RapidsBufferBase → RapidsBuffer
    Note

    It is the responsibility of the caller to close the batch.

    ,

    If the buffer is compressed data then the resulting batch will be built using GpuCompressedColumnVector, and it is the responsibility of the caller to deal with decompressing the data if necessary.

    See also

    addReference

  16. def getCopyIterator: RapidsBufferCopyIterator

    At spill time, obtain an iterator used to copy this buffer to a different tier.

    At spill time, obtain an iterator used to copy this buffer to a different tier.

    Definition Classes
    RapidsBuffer
  17. def getDeviceMemoryBuffer: DeviceMemoryBuffer

    TODO: we want to remove this method from the buffer, instead we want the catalog to be responsible for producing the DeviceMemoryBuffer by asking the buffer.

    TODO: we want to remove this method from the buffer, instead we want the catalog to be responsible for producing the DeviceMemoryBuffer by asking the buffer. This hides the RapidsBuffer from clients and simplifies locking.

    Definition Classes
    RapidsBufferBase → RapidsBuffer
  18. def getHostColumnarBatch(sparkTypes: Array[DataType]): ColumnarBatch

    Get the host-backed columnar batch from this buffer.

    Get the host-backed columnar batch from this buffer. The caller must have successfully acquired the buffer beforehand.

    If this RapidsBuffer was added originally to the device tier, or if this is a just a buffer (not a batch), this function will throw.

    sparkTypes

    the spark data types the batch should have

    Definition Classes
    RapidsDiskColumnarBatch → RapidsHostBatchBuffer → RapidsBuffer
    Note

    It is the responsibility of the caller to close the batch.

    See also

    addReference

  19. def getHostMemoryBuffer: HostMemoryBuffer

    Get the host memory buffer from the underlying storage.

    Get the host memory buffer from the underlying storage. If the buffer currently resides outside of host memory, a new HostMemoryBuffer is created with the data copied over. The caller must have successfully acquired the buffer beforehand.

    Definition Classes
    RapidsBufferBase → RapidsBuffer
    Note

    It is the responsibility of the caller to close the buffer.

    See also

    addReference

  20. def getMemoryBuffer: MemoryBuffer

    Get the underlying memory buffer.

    Get the underlying memory buffer. This may be either a HostMemoryBuffer or a DeviceMemoryBuffer depending on where the buffer currently resides. The caller must have successfully acquired the buffer beforehand.

    Definition Classes
    RapidsDiskColumnarBatch → RapidsDiskBuffer → RapidsBuffer
    Note

    It is the responsibility of the caller to close the buffer.

    See also

    addReference

  21. def getPackedSizeBytes: Long

    The size of this buffer if it has already gone through contiguous_split.

    The size of this buffer if it has already gone through contiguous_split.

    Definition Classes
    RapidsBuffer
    Note

    Use this function when allocating a target buffer for spill or shuffle purposes.

  22. def getSpillPriority: Long

    Get the spill priority value for this buffer.

    Get the spill priority value for this buffer. Lower values are higher priority for spilling, meaning buffers with lower values will be preferred for spilling over buffers with a higher value.

    Definition Classes
    RapidsBufferBase → RapidsBuffer
  23. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  24. val id: RapidsBufferId

    The buffer identifier for this buffer.

    The buffer identifier for this buffer.

    Definition Classes
    RapidsBufferBase → RapidsBuffer
  25. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  26. var isValid: Boolean
    Attributes
    protected[this]
    Definition Classes
    RapidsBufferBase
  27. def makeChunkedPacker: ChunkedPacker

    Makes a new chunked packer.

    Makes a new chunked packer. It is the responsibility of the caller to close this.

    Definition Classes
    RapidsBuffer
  28. def materializeMemoryBuffer: MemoryBuffer

    Materialize the memory buffer from the underlying storage.

    Materialize the memory buffer from the underlying storage.

    If the buffer resides in device or host memory, only reference count is incremented. If the buffer resides in secondary storage, a new host or device memory buffer is created, with the data copied to the new buffer. The caller must have successfully acquired the buffer beforehand.

    Attributes
    protected
    Definition Classes
    RapidsBufferBase
    Note

    It is the responsibility of the caller to close the buffer.

    ,

    This is an internal API only used by Rapids buffer stores.

    See also

    addReference

  29. val memoryUsedBytes: Long

    The size of this buffer in bytes in its _current_ store.

    The size of this buffer in bytes in its _current_ store. As the buffer goes through contiguous split (either added as a contiguous table already, or spilled to host), its size changes because contiguous_split adds its own alignment padding.

    Definition Classes
    RapidsDiskBuffer → RapidsBuffer
    Note

    Do not use this size to allocate a target buffer to copy, always use getPackedSize.

  30. def meta: TableMeta

    Descriptor for how the memory buffer is formatted

    Descriptor for how the memory buffer is formatted

    Definition Classes
    RapidsBufferBase → RapidsBuffer
  31. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  32. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  33. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  34. var refcount: Int
    Attributes
    protected[this]
    Definition Classes
    RapidsBufferBase
  35. def releaseResources(): Unit

    Release the underlying resources for this buffer.

    Release the underlying resources for this buffer.

    Attributes
    protected
    Definition Classes
    RapidsDiskBuffer → RapidsBufferBase
  36. def setSpillPriority(priority: Long): Unit

    Set the spill priority for this buffer.

    Set the spill priority for this buffer. Lower values are higher priority for spilling, meaning buffers with lower values will be preferred for spilling over buffers with a higher value.

    priority

    new priority value for this buffer

    Definition Classes
    RapidsBufferBase → RapidsBuffer
    Note

    should only be called from the buffer catalog

  37. val storageTier: StorageTier

    The storage tier for this buffer

    The storage tier for this buffer

    Definition Classes
    RapidsDiskBuffer → RapidsBuffer
  38. val supportsChunkedPacker: Boolean
    Definition Classes
    RapidsBuffer
  39. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  40. def toString(): String
    Definition Classes
    RapidsBufferBase → AnyRef → Any
  41. def updateSpillability(): Unit

    Function invoked by the RapidsBufferStore.addBuffer method that prompts the specific RapidsBuffer to check its reference counting to make itself spillable or not.

    Function invoked by the RapidsBufferStore.addBuffer method that prompts the specific RapidsBuffer to check its reference counting to make itself spillable or not. Only RapidsTable and RapidsHostMemoryBuffer implement this method.

    Definition Classes
    RapidsBuffer
  42. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  43. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  44. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  45. def withMemoryBufferReadLock[K](body: (MemoryBuffer) ⇒ K): K

    Obtains a read lock on this instance of RapidsBuffer and calls the function in body while holding the lock.

    Obtains a read lock on this instance of RapidsBuffer and calls the function in body while holding the lock.

    K

    any return type specified by body

    body

    function that takes a MemoryBuffer and produces K

    returns

    the result of body(memoryBuffer)

    Definition Classes
    RapidsBufferBase → RapidsBuffer
  46. def withMemoryBufferWriteLock[K](body: (MemoryBuffer) ⇒ K): K

    Obtains a write lock on this instance of RapidsBuffer and calls the function in body while holding the lock.

    Obtains a write lock on this instance of RapidsBuffer and calls the function in body while holding the lock.

    K

    any return type specified by body

    body

    function that takes a MemoryBuffer and produces K

    returns

    the result of body(memoryBuffer)

    Definition Classes
    RapidsBufferBase → RapidsBuffer

Inherited from RapidsHostBatchBuffer

Inherited from RapidsDiskBuffer

Inherited from RapidsDiskStore.RapidsBufferBase

Inherited from RapidsBuffer

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped