parquet.column.values.rle

Class RunLengthBitPackingHybridEncoder

java.lang.Object

parquet.column.values.rle.RunLengthBitPackingHybridEncoder

public class RunLengthBitPackingHybridEncoder
extends Object

Encodes values using a combination of run length encoding and bit packing, according to the following grammar:

 rle-bit-packed-hybrid: <length> <encoded-data>
 length := length of the <encoded-data> in bytes stored as 4 bytes little endian
 encoded-data := <run>*
 run := <bit-packed-run> | <rle-run>
 bit-packed-run := <bit-packed-header> <bit-packed-values>
 bit-packed-header := varint-encode(<bit-pack-count> << 1 | 1)
 // we always bit-pack a multiple of 8 values at a time, so we only store the number of values / 8
 bit-pack-count := (number of values in this run) / 8
 bit-packed-values :=  bit packed back to back, from LSB to MSB
 rle-run := <rle-header> <repeated-value>
 rle-header := varint-encode( (number of times repeated) << 1)
 repeated-value := value that is repeated, using a fixed-width of round-up-to-next-byte(bit-width)
 
 

NOTE: this class is only responsible for creating and returning the <encoded-data> portion of the above grammar. The <length> portion is done by RunLengthBitPackingHybridValuesWriter

Only supports values >= 0 // TODO: is that ok? Should we make a signed version?

Author:

Alex Levenson

Constructor Summary

Constructors

Constructor and Description
RunLengthBitPackingHybridEncoder(int bitWidth, int initialCapacity)

Method Summary

Methods

Modifier and Type	Method and Description
long	getAllocatedSize()
long	getBufferedSize()
void	reset() Reset this encoder for re-use
BytesInput	toBytes()
void	writeInt(int value)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RunLengthBitPackingHybridEncoder

public RunLengthBitPackingHybridEncoder(int bitWidth,
                                int initialCapacity)

Method Detail

writeInt

public void writeInt(int value)
              throws IOException

Throws:

IOException

toBytes

public BytesInput toBytes()
                   throws IOException

Throws:

IOException

reset

public void reset()

Reset this encoder for re-use

getBufferedSize

public long getBufferedSize()

getAllocatedSize

public long getAllocatedSize()