c

org.apache.spark.sql.rapids.execution.python

GpuArrowPythonWriter

abstract class GpuArrowPythonWriter extends GpuArrowWriter

Linear Supertypes
GpuArrowWriter, AutoCloseable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. GpuArrowPythonWriter
  2. GpuArrowWriter
  3. AutoCloseable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new GpuArrowPythonWriter(inputSchema: StructType, maxBatchSize: Long)

Abstract Value Members

  1. abstract def writeUDFs(dataOut: DataOutputStream): Unit
    Attributes
    protected

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. def close(): Unit
    Definition Classes
    GpuArrowWriter → AutoCloseable
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. val inputSchema: StructType
    Definition Classes
    GpuArrowPythonWriter → GpuArrowWriter
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. val maxBatchSize: Long
    Definition Classes
    GpuArrowPythonWriter → GpuArrowWriter
  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. final def reset(): Unit

    This is design to reuse the writer options

    This is design to reuse the writer options

    Definition Classes
    GpuArrowWriter
  19. final def start(dataOut: DataOutputStream): Unit

    Make the writer ready to write data, should be called before writing any batch

    Make the writer ready to write data, should be called before writing any batch

    Definition Classes
    GpuArrowWriter
  20. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  21. def toString(): String
    Definition Classes
    AnyRef → Any
  22. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  25. final def write(batch: ColumnarBatch): Unit
    Definition Classes
    GpuArrowWriter
  26. final def writeAndClose(batch: ColumnarBatch): Unit
    Definition Classes
    GpuArrowWriter
  27. def writeCommand(dataOut: DataOutputStream, confs: Map[String, String]): Unit
  28. final def writeEmptyIteratorOnCpu(dataOut: DataOutputStream, arrowSchema: Schema): Unit

    This is for writing the empty partition.

    This is for writing the empty partition. In this case CPU will still send the schema to Python workers by calling the "start" API of the Java Arrow writer, but GPU will send out nothing, leading to the IPC error. And it is not easy to do as what Spark does on GPU, because the C++ Arrow writer used by GPU will only send out the schema iff there is some data. Besides, it does not expose a "start" API to do this. So here we leverage the Java Arrow writer to do similar things as Spark. It is OK because sending out schema has nothing to do with GPU. (Most code is copied from Spark)

Inherited from GpuArrowWriter

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped