Type Members

1.  final class Variant extends AnyRef

   This class is structurally equivalent to org.apache.spark.unsafe.types.VariantVal.

   This class is structurally equivalent to org.apache.spark.unsafe.types.VariantVal. We define a new class to avoid depending on or modifying Spark.

2.  class VariantBuilder extends AnyRef

   Build variant value and metadata by parsing JSON values.

3.  class VariantSizeLimitException extends RuntimeException

   An exception indicating that we are attempting to build a variant with it value or metadata exceeding the 16MiB size limit.

4.  class VariantUtil extends AnyRef

   This class defines constants related to the variant format and provides functions for manipulating variant binaries.

   This class defines constants related to the variant format and provides functions for manipulating variant binaries.

   A variant is made up of 2 binaries: value and metadata. A variant value consists of a one-byte header and a number of content bytes (can be zero). The header byte is divided into upper 6 bits (called "type info") and lower 2 bits (called "basic type"). The content format is explained in the below constants for all possible basic type and type info values.

   The variant metadata includes a version id and a dictionary of distinct strings (case-sensitive). Its binary format is: - Version: 1-byte unsigned integer. The only acceptable value is 1 currently. - Dictionary size: 4-byte little-endian unsigned integer. The number of keys in the dictionary. - Offsets: (size + 1) * 4-byte little-endian unsigned integers. offsets[i] represents the starting position of string i, counting starting from the address of offsets[0]. Strings must be stored contiguously, so we don’t need to store the string size, instead, we compute it with offset[i + 1] - offset[i]. - UTF-8 string data.