org.apache.hadoop.fs.azure

Class BlockBlobAppendStream

java.lang.Object

java.io.OutputStream

org.apache.hadoop.fs.azure.BlockBlobAppendStream

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable, org.apache.hadoop.fs.StreamCapabilities, org.apache.hadoop.fs.Syncable

public class BlockBlobAppendStream
extends OutputStream
implements org.apache.hadoop.fs.Syncable, org.apache.hadoop.fs.StreamCapabilities

Stream object that implements append for Block Blobs in WASB. The stream object implements hflush/hsync and block compaction. Block compaction is the process of replacing a sequence of small blocks with one big block. Azure Block blobs supports up to 50000 blocks and every hflush/hsync generates one block. When the number of blocks is above 32000, the process of compaction decreases the total number of blocks, if possible. If compaction is disabled, hflush/hsync are empty functions. The stream object uses background threads for uploading the blocks and the block blob list. Blocks can be uploaded concurrently. However, when the block list is uploaded, block uploading should stop. If a block is uploaded before the block list and the block id is not in the list, the block will be lost. If the block is uploaded after the block list and the block id is in the list, the block list upload will fail. The exclusive access for the block list upload is managed by uploadingSemaphore.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Field Summary

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

DROPBEHIND, HFLUSH, HSYNC, READAHEAD, UNBUFFER

Constructor Summary

Constructors

Constructor and Description
BlockBlobAppendStream(StorageInterface.CloudBlockBlobWrapper blob, String aKey, int bufferSize, boolean compactionEnabled, com.microsoft.azure.storage.OperationContext opContext) BlockBlobAppendStream constructor.

Method Summary

Modifier and Type	Method and Description
void	close() Force all data in the output stream to be written to Azure storage.
void	flush() Flushes this output stream and forces any buffered output bytes to be written out.
boolean	hasCapability(String capability) The Synchronization capabilities of this stream depend upon the compaction policy.
void	hflush() Force all data in the output stream to be written to Azure storage.
void	hsync() Force all data in the output stream to be written to Azure storage.
void	write(byte[] data, int offset, int length) Writes length bytes from the specified byte array starting at offset to this output stream.
void	write(int byteVal) Writes the specified byte to this output stream.

Methods inherited from class java.io.OutputStream

write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BlockBlobAppendStream

public BlockBlobAppendStream(StorageInterface.CloudBlockBlobWrapper blob,
                             String aKey,
                             int bufferSize,
                             boolean compactionEnabled,
                             com.microsoft.azure.storage.OperationContext opContext)
                      throws IOException

BlockBlobAppendStream constructor.

Parameters:

blob - Azure Block Blob

aKey - blob's name

bufferSize - the maximum size of a blob block.

compactionEnabled - is the compaction process enabled for this blob

opContext - Azure Store operation context for the blob

Throws:

IOException - if an I/O error occurs. In particular, an IOException may be thrown if the output stream cannot be used for append operations

Method Detail

write

public void write(int byteVal)
           throws IOException

Writes the specified byte to this output stream. The general contract for write is that one byte is written to the output stream. The byte to be written is the eight low-order bits of the argument b. The 24 high-order bits of b are ignored.

Specified by:

write in class OutputStream

Parameters:

byteVal - the byteValue to write.

Throws:

IOException - if an I/O error occurs. In particular, an IOException may be thrown if the output stream has been closed.

write

public void write(byte[] data,
                  int offset,
                  int length)
           throws IOException

Writes length bytes from the specified byte array starting at offset to this output stream.

Overrides:

write in class OutputStream

Parameters:

data - the byte array to write.

offset - the start offset in the data.

length - the number of bytes to write.

Throws:

IOException - if an I/O error occurs. In particular, an IOException may be thrown if the output stream has been closed.

flush

public void flush()
           throws IOException

Flushes this output stream and forces any buffered output bytes to be written out. If any data remains in the payload it is committed to the service. Data is queued for writing and forced out to the service before the call returns.

Specified by:

flush in interface Flushable

Overrides:

flush in class OutputStream

Throws:

IOException

hsync

public void hsync()
           throws IOException

Force all data in the output stream to be written to Azure storage. Wait to return until this is complete.

Specified by:

hsync in interface org.apache.hadoop.fs.Syncable

Throws:

IOException

hflush

public void hflush()
            throws IOException

Force all data in the output stream to be written to Azure storage. Wait to return until this is complete.

Specified by:

hflush in interface org.apache.hadoop.fs.Syncable

Throws:

IOException

hasCapability

public boolean hasCapability(String capability)

The Synchronization capabilities of this stream depend upon the compaction policy.

Specified by:

hasCapability in interface org.apache.hadoop.fs.StreamCapabilities

Parameters:

capability - string to query the stream support for.

Returns:

true for hsync and hflush when compaction is enabled.

close

public void close()
           throws IOException

Force all data in the output stream to be written to Azure storage. Wait to return until this is complete. Close the access to the stream and shutdown the upload thread pool. If the blob was created, its lease will be released. Any error encountered caught in threads and stored will be rethrown here after cleanup.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class OutputStream

Throws:

IOException