RAPIDS Accelerator for Apache Spark SQL Plugin 24.10.1 API - com.nvidia.spark.rapids.RapidsDiskStore.RapidsDiskBuffer

final def !=(arg0: Any): Boolean

Definition Classes: AnyRef → Any

final def ##(): Int

Definition Classes: AnyRef → Any

final def ==(arg0: Any): Boolean

Definition Classes: AnyRef → Any

def addReference(): Boolean

Try to add a reference to this buffer to acquire it.

returns: true if the reference was added or false if this buffer is no longer valid

Definition Classes: RapidsBufferBase → RapidsBuffer
Note: The close method must be called for every successfully obtained reference.

final def asInstanceOf[T0]: T0

Definition Classes: Any

def clone(): AnyRef

Attributes: protected[lang]
Definition Classes: AnyRef
Annotations: @throws( ... ) @native()

def close(): Unit

close() is called by client code to decrease the ref count of this RapidsBufferBase.

close() is called by client code to decrease the ref count of this RapidsBufferBase. In the off chance that by the time close is invoked, the buffer was freed (not valid) then this close call winds up freeing the resources of the rapids buffer.

Definition Classes: RapidsDiskBuffer → RapidsBufferBase → AutoCloseable

def columnarBatchFromDeviceBuffer(devBuffer: DeviceMemoryBuffer, sparkTypes: Array[DataType]): ColumnarBatch

Attributes: protected
Definition Classes: RapidsBufferBase

def copyToMemoryBuffer(srcOffset: Long, dst: MemoryBuffer, dstOffset: Long, length: Long, stream: Stream): Unit

Copy the content of this buffer into the specified memory buffer, starting from the given offset.

srcOffset: offset to start copying from.
dst: the memory buffer to copy into.
dstOffset: offset to copy into.
length: number of bytes to copy.
stream: CUDA stream to use

Definition Classes: RapidsBufferBase → RapidsBuffer

final def eq(arg0: AnyRef): Boolean

Definition Classes: AnyRef

def equals(arg0: Any): Boolean

Definition Classes: AnyRef → Any

def finalize(): Unit

Attributes: protected[lang]
Definition Classes: AnyRef
Annotations: @throws( classOf[java.lang.Throwable] )

def free(): Unit

Mark the buffer as freed and no longer valid.

Mark the buffer as freed and no longer valid. This is called by the store when removing a buffer (it is no longer tracked).

Definition Classes: RapidsBufferBase → RapidsBuffer
Note: The resources may not be immediately released if the buffer has outstanding references. In that case the resources will be released when the reference count reaches zero.

final def getClass(): Class[_]

Definition Classes: AnyRef → Any
Annotations: @native()

def getColumnarBatch(sparkTypes: Array[DataType]): ColumnarBatch

Get the columnar batch within this buffer.

Get the columnar batch within this buffer. The caller must have successfully acquired the buffer beforehand.

sparkTypes: the spark data types the batch should have

Definition Classes: RapidsBufferBase → RapidsBuffer
Note: It is the responsibility of the caller to close the batch.
,
If the buffer is compressed data then the resulting batch will be built using GpuCompressedColumnVector, and it is the responsibility of the caller to deal with decompressing the data if necessary.
See also: addReference

def getCopyIterator: RapidsBufferCopyIterator

At spill time, obtain an iterator used to copy this buffer to a different tier.

Definition Classes: RapidsBuffer

def getDeviceMemoryBuffer: DeviceMemoryBuffer

TODO: we want to remove this method from the buffer, instead we want the catalog to be responsible for producing the DeviceMemoryBuffer by asking the buffer.

TODO: we want to remove this method from the buffer, instead we want the catalog to be responsible for producing the DeviceMemoryBuffer by asking the buffer. This hides the RapidsBuffer from clients and simplifies locking.

Definition Classes: RapidsBufferBase → RapidsBuffer

def getHostColumnarBatch(sparkTypes: Array[DataType]): ColumnarBatch

Get the host-backed columnar batch from this buffer.

Get the host-backed columnar batch from this buffer. The caller must have successfully acquired the buffer beforehand.

If this RapidsBuffer was added originally to the device tier, or if this is a just a buffer (not a batch), this function will throw.

sparkTypes: the spark data types the batch should have

Definition Classes: RapidsBuffer
Note: It is the responsibility of the caller to close the batch.
See also: addReference

def getHostMemoryBuffer: HostMemoryBuffer

Get the host memory buffer from the underlying storage.

Get the host memory buffer from the underlying storage. If the buffer currently resides outside of host memory, a new HostMemoryBuffer is created with the data copied over. The caller must have successfully acquired the buffer beforehand.

Definition Classes: RapidsBufferBase → RapidsBuffer
Note: It is the responsibility of the caller to close the buffer.
See also: addReference

def getMemoryBuffer: MemoryBuffer

Get the underlying memory buffer.

Get the underlying memory buffer. This may be either a HostMemoryBuffer or a DeviceMemoryBuffer depending on where the buffer currently resides. The caller must have successfully acquired the buffer beforehand.

Definition Classes: RapidsDiskBuffer → RapidsBuffer
Note: It is the responsibility of the caller to close the buffer.
See also: addReference

def getPackedSizeBytes: Long

The size of this buffer if it has already gone through contiguous_split.

Definition Classes: RapidsBuffer
Note: Use this function when allocating a target buffer for spill or shuffle purposes.

def getSpillPriority: Long

Get the spill priority value for this buffer.

Get the spill priority value for this buffer. Lower values are higher priority for spilling, meaning buffers with lower values will be preferred for spilling over buffers with a higher value.

Definition Classes: RapidsBufferBase → RapidsBuffer

def hashCode(): Int

Definition Classes: AnyRef → Any
Annotations: @native()

val id: RapidsBufferId

The buffer identifier for this buffer.

Definition Classes: RapidsBufferBase → RapidsBuffer

final def isInstanceOf[T0]: Boolean

Definition Classes: Any

var isValid: Boolean

Attributes: protected[this]
Definition Classes: RapidsBufferBase

def makeChunkedPacker: ChunkedPacker

Makes a new chunked packer.

Makes a new chunked packer. It is the responsibility of the caller to close this.

Definition Classes: RapidsBuffer

def materializeMemoryBuffer: MemoryBuffer

Materialize the memory buffer from the underlying storage.

If the buffer resides in device or host memory, only reference count is incremented. If the buffer resides in secondary storage, a new host or device memory buffer is created, with the data copied to the new buffer. The caller must have successfully acquired the buffer beforehand.

Attributes: protected
Definition Classes: RapidsBufferBase
Note: It is the responsibility of the caller to close the buffer.
,
This is an internal API only used by Rapids buffer stores.
See also: addReference

val memoryUsedBytes: Long

The size of this buffer in bytes in its _current_ store.

The size of this buffer in bytes in its _current_ store. As the buffer goes through contiguous split (either added as a contiguous table already, or spilled to host), its size changes because contiguous_split adds its own alignment padding.

Definition Classes: RapidsDiskBuffer → RapidsBuffer
Note: Do not use this size to allocate a target buffer to copy, always use getPackedSize.

def meta: TableMeta

Descriptor for how the memory buffer is formatted

Definition Classes: RapidsBufferBase → RapidsBuffer

final def ne(arg0: AnyRef): Boolean

Definition Classes: AnyRef

final def notify(): Unit

Definition Classes: AnyRef
Annotations: @native()

final def notifyAll(): Unit

Definition Classes: AnyRef
Annotations: @native()

var refcount: Int

Attributes: protected[this]
Definition Classes: RapidsBufferBase

def releaseResources(): Unit

Release the underlying resources for this buffer.

Attributes: protected
Definition Classes: RapidsDiskBuffer → RapidsBufferBase

def setSpillPriority(priority: Long): Unit

Set the spill priority for this buffer.

Set the spill priority for this buffer. Lower values are higher priority for spilling, meaning buffers with lower values will be preferred for spilling over buffers with a higher value.

priority: new priority value for this buffer

Definition Classes: RapidsBufferBase → RapidsBuffer
Note: should only be called from the buffer catalog

val storageTier: StorageTier

The storage tier for this buffer

Definition Classes: RapidsDiskBuffer → RapidsBuffer

val supportsChunkedPacker: Boolean

Definition Classes: RapidsBuffer

final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes: AnyRef

def toString(): String

Definition Classes: RapidsBufferBase → AnyRef → Any

def updateSpillability(): Unit

Function invoked by the RapidsBufferStore.addBuffer method that prompts the specific RapidsBuffer to check its reference counting to make itself spillable or not.

Function invoked by the RapidsBufferStore.addBuffer method that prompts the specific RapidsBuffer to check its reference counting to make itself spillable or not. Only RapidsTable and RapidsHostMemoryBuffer implement this method.

Definition Classes: RapidsBuffer

final def wait(): Unit

Definition Classes: AnyRef
Annotations: @throws( ... )

final def wait(arg0: Long, arg1: Int): Unit

Definition Classes: AnyRef
Annotations: @throws( ... )

final def wait(arg0: Long): Unit

Definition Classes: AnyRef
Annotations: @throws( ... ) @native()

def withMemoryBufferReadLock[K](body: (MemoryBuffer) ⇒ K): K

Obtains a read lock on this instance of RapidsBuffer and calls the function in body while holding the lock.

K: any return type specified by body
body: function that takes a MemoryBuffer and produces K
returns: the result of body(memoryBuffer)

Definition Classes: RapidsBufferBase → RapidsBuffer

def withMemoryBufferWriteLock[K](body: (MemoryBuffer) ⇒ K): K

Obtains a write lock on this instance of RapidsBuffer and calls the function in body while holding the lock.

K: any return type specified by body
body: function that takes a MemoryBuffer and produces K
returns: the result of body(memoryBuffer)

Definition Classes: RapidsBufferBase → RapidsBuffer

Packages

RapidsDiskBuffer

class RapidsDiskBuffer extends RapidsBufferBase

Instance Constructors

Value Members

Inherited from RapidsDiskStore.RapidsBufferBase

Inherited from RapidsBuffer

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped