org.apache.hadoop.fs.azure

Class AzureNativeFileSystemStore

java.lang.Object

org.apache.hadoop.fs.azure.AzureNativeFileSystemStore

@InterfaceAudience.Private
public class AzureNativeFileSystemStore
extends Object

Core implementation of Windows Azure Filesystem for Hadoop. Provides the bridging logic between Hadoop's abstract filesystem and Azure Storage

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_DOWNLOAD_BLOCK_SIZE
static boolean	DEFAULT_ENABLE_FLAT_LISTING Enable flat listing of blobs as default option.
static long	DEFAULT_HADOOP_BLOCK_SIZE
static int	DEFAULT_UPLOAD_BLOCK_SIZE
static boolean	DEFAULT_USE_SECURE_MODE Default values to control SAS Key mode.
static String	KEY_ATOMIC_RENAME_DIRECTORIES Configuration key to indicate the set of directories in WASB where we should do atomic folder rename synchronized with createNonRecursive.
static String	KEY_BLOCK_BLOB_WITH_COMPACTION_DIRECTORIES Configuration key to indicate the set of directories in WASB where we should store files as block blobs with block compaction enabled.
static String	KEY_ENABLE_FLAT_LISTING Configuration key to enable flat listing of blobs.
static String	KEY_PAGE_BLOB_DIRECTORIES Configuration key to indicate the set of directories in WASB where we should store files as page blobs instead of block blobs.
static String	KEY_USE_LOCAL_SAS_KEY_MODE By default the SAS Key mode is expected to run in Romote key mode.
static String	KEY_USE_SECURE_MODE Configuration keys to identify if WASB needs to run in Secure mode.
static org.slf4j.Logger	LOG
static String	NO_ACCESS_TO_CONTAINER_MSG The error message template when container is not accessible.

Constructor Summary

Constructors

Constructor and Description
AzureNativeFileSystemStore()

Method Summary

Modifier and Type	Method and Description
SelfRenewingLease	acquireLease(String key) Get a lease on the blob identified by key.
void	changePermissionStatus(String key, org.apache.hadoop.fs.permission.PermissionStatus newPermission) Changes the permission status on the given key.
void	close()
boolean	delete(String key) API implementation to delete a blob in the back end azure storage.
boolean	delete(String key, SelfRenewingLease lease) API implementation to delete a blob in the back end azure storage.
void	dump() Diagnostic method to dump state to the console.
boolean	explicitFileExists(String key) Checks whether an explicit file/folder exists.
protected void	finalize()
static String	getAccountKeyFromConfiguration(String accountName, org.apache.hadoop.conf.Configuration conf)
BandwidthGaugeUpdater	getBandwidthGaugeUpdater()
long	getHadoopBlockSize() Returns the file block size.
String	getLinkInFileMetadata(String key) If the blob with the given key exists and has a link in its metadata to a temporary file (see storeEmptyLinkFile), this method returns the key to that temporary file.
void	initialize(URI uri, org.apache.hadoop.conf.Configuration conf, AzureFileSystemInstrumentation instrumentation) Method for the URI and configuration object necessary to create a storage session with an Azure session.
boolean	isAtomicRenameKey(String key) Checks if the given key in Azure storage should have synchronized atomic folder rename createNonRecursive implemented.
boolean	isBlockBlobWithCompactionKey(String key) Checks if the given key in Azure Storage should be stored as a block blobs with compaction enabled instead of normal block blob.
boolean	isKeyForDirectorySet(String key, Set<String> dirSet)
boolean	isPageBlobKey(String key) Checks if the given key in Azure Storage should be stored as a page blob instead of block blob.
org.apache.hadoop.fs.azure.FileMetadata[]	list(String prefix, int maxListingCount, int maxListingDepth)
void	purge(String prefix) Delete all keys with the given prefix.
void	rename(String srcKey, String dstKey)
void	rename(String srcKey, String dstKey, boolean acquireLease, SelfRenewingLease existingLease)
void	rename(String srcKey, String dstKey, boolean acquireLease, SelfRenewingLease existingLease, boolean overwriteDestination)
InputStream	retrieve(String key)
InputStream	retrieve(String key, long startByteOffset)
DataOutputStream	retrieveAppendStream(String key, int bufferSize)
org.apache.hadoop.fs.azure.FileMetadata	retrieveMetadata(String key)
void	storeEmptyFolder(String key, org.apache.hadoop.fs.permission.PermissionStatus permissionStatus)
void	storeEmptyLinkFile(String key, String tempBlobKey, org.apache.hadoop.fs.permission.PermissionStatus permissionStatus) Stores an empty blob that's linking to the temporary file where're we're uploading the initial data.
DataOutputStream	storefile(String keyEncoded, org.apache.hadoop.fs.permission.PermissionStatus permissionStatus, String key)
void	updateFolderLastModifiedTime(String key, Date lastModified, SelfRenewingLease folderLease)
void	updateFolderLastModifiedTime(String key, SelfRenewingLease folderLease)

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

KEY_USE_SECURE_MODE

public static final String KEY_USE_SECURE_MODE

Configuration keys to identify if WASB needs to run in Secure mode. In Secure mode all interactions with Azure storage is performed using SAS uris. There are two sub modes within the Secure mode , one is remote SAS key mode where the SAS keys are generated from a remote process and local mode where SAS keys are generated within WASB.

See Also:

Constant Field Values

KEY_USE_LOCAL_SAS_KEY_MODE

public static final String KEY_USE_LOCAL_SAS_KEY_MODE

By default the SAS Key mode is expected to run in Romote key mode. This flags sets it to run on the local mode.

See Also:

Constant Field Values

KEY_PAGE_BLOB_DIRECTORIES

public static final String KEY_PAGE_BLOB_DIRECTORIES

Configuration key to indicate the set of directories in WASB where we should store files as page blobs instead of block blobs. Entries should be plain directory names (i.e. not URIs) with no leading or trailing slashes. Delimit the entries with commas.

See Also:

Constant Field Values

KEY_BLOCK_BLOB_WITH_COMPACTION_DIRECTORIES

public static final String KEY_BLOCK_BLOB_WITH_COMPACTION_DIRECTORIES

Configuration key to indicate the set of directories in WASB where we should store files as block blobs with block compaction enabled. Entries can be directory paths relative to the container (e.g. "/path") or fully qualified wasb:// URIs (e.g. wasb://container@example.blob.core.windows.net/path)

See Also:

Constant Field Values

KEY_ATOMIC_RENAME_DIRECTORIES

public static final String KEY_ATOMIC_RENAME_DIRECTORIES

Configuration key to indicate the set of directories in WASB where we should do atomic folder rename synchronized with createNonRecursive.

See Also:

Constant Field Values

KEY_ENABLE_FLAT_LISTING

public static final String KEY_ENABLE_FLAT_LISTING

Configuration key to enable flat listing of blobs. This config is useful only if listing depth is AZURE_UNBOUNDED_DEPTH.

See Also:

Constant Field Values

DEFAULT_DOWNLOAD_BLOCK_SIZE

public static final int DEFAULT_DOWNLOAD_BLOCK_SIZE

See Also:

Constant Field Values

DEFAULT_UPLOAD_BLOCK_SIZE

public static final int DEFAULT_UPLOAD_BLOCK_SIZE

See Also:

Constant Field Values

DEFAULT_HADOOP_BLOCK_SIZE

public static final long DEFAULT_HADOOP_BLOCK_SIZE

See Also:

Constant Field Values

DEFAULT_USE_SECURE_MODE

public static final boolean DEFAULT_USE_SECURE_MODE

Default values to control SAS Key mode. By default we set the values to false.

See Also:

Constant Field Values

DEFAULT_ENABLE_FLAT_LISTING

public static final boolean DEFAULT_ENABLE_FLAT_LISTING

Enable flat listing of blobs as default option. This is useful only if listing depth is AZURE_UNBOUNDED_DEPTH.

See Also:

Constant Field Values

NO_ACCESS_TO_CONTAINER_MSG

public static final String NO_ACCESS_TO_CONTAINER_MSG

The error message template when container is not accessible.

See Also:

Constant Field Values

Constructor Detail

AzureNativeFileSystemStore

public AzureNativeFileSystemStore()

Method Detail

getBandwidthGaugeUpdater

public BandwidthGaugeUpdater getBandwidthGaugeUpdater()

initialize

public void initialize(URI uri,
                       org.apache.hadoop.conf.Configuration conf,
                       AzureFileSystemInstrumentation instrumentation)
                throws IllegalArgumentException,
                       AzureException,
                       IOException

Method for the URI and configuration object necessary to create a storage session with an Azure session. It parses the scheme to ensure it matches the storage protocol supported by this file system.

Parameters:

uri - - URI for target storage blob.

conf - - reference to configuration object.

instrumentation - - the metrics source that will keep track of operations here.

Throws:

IllegalArgumentException - if URI or job object is null, or invalid scheme.

AzureException

IOException

getAccountKeyFromConfiguration

public static String getAccountKeyFromConfiguration(String accountName,
                                                    org.apache.hadoop.conf.Configuration conf)
                                             throws KeyProviderException

Throws:

KeyProviderException

isPageBlobKey

public boolean isPageBlobKey(String key)

Checks if the given key in Azure Storage should be stored as a page blob instead of block blob.

isBlockBlobWithCompactionKey

public boolean isBlockBlobWithCompactionKey(String key)

Checks if the given key in Azure Storage should be stored as a block blobs with compaction enabled instead of normal block blob.

Parameters:

key - blob name

Returns:

true, if the file is in directory with block compaction enabled.

isAtomicRenameKey

public boolean isAtomicRenameKey(String key)

Checks if the given key in Azure storage should have synchronized atomic folder rename createNonRecursive implemented.

isKeyForDirectorySet

public boolean isKeyForDirectorySet(String key,
                                    Set<String> dirSet)

getHadoopBlockSize

public long getHadoopBlockSize()

Returns the file block size. This is a fake value used for integration of the Azure store with Hadoop.

Returns:

The file block size.

storefile

public DataOutputStream storefile(String keyEncoded,
                                  org.apache.hadoop.fs.permission.PermissionStatus permissionStatus,
                                  String key)
                           throws AzureException

Throws:

AzureException

storeEmptyFolder

public void storeEmptyFolder(String key,
                             org.apache.hadoop.fs.permission.PermissionStatus permissionStatus)
                      throws AzureException

Throws:

AzureException

storeEmptyLinkFile

public void storeEmptyLinkFile(String key,
                               String tempBlobKey,
                               org.apache.hadoop.fs.permission.PermissionStatus permissionStatus)
                        throws AzureException

Stores an empty blob that's linking to the temporary file where're we're uploading the initial data.

Throws:

AzureException

getLinkInFileMetadata

public String getLinkInFileMetadata(String key)
                             throws AzureException

If the blob with the given key exists and has a link in its metadata to a temporary file (see storeEmptyLinkFile), this method returns the key to that temporary file. Otherwise, returns null.

Throws:

AzureException

retrieveMetadata

public org.apache.hadoop.fs.azure.FileMetadata retrieveMetadata(String key)
                                                         throws IOException

Throws:

IOException

retrieve

public InputStream retrieve(String key)
                     throws AzureException,
                            IOException

Throws:

AzureException

IOException

retrieve

public InputStream retrieve(String key,
                            long startByteOffset)
                     throws AzureException,
                            IOException

Throws:

AzureException

IOException

list

public org.apache.hadoop.fs.azure.FileMetadata[] list(String prefix,
                                                      int maxListingCount,
                                                      int maxListingDepth)
                                               throws IOException

Throws:

IOException

delete

public boolean delete(String key,
                      SelfRenewingLease lease)
               throws IOException

API implementation to delete a blob in the back end azure storage.

Parameters:

key - - key to the blob being deleted.

lease - - Active lease on the blob.

Returns:

return true when delete is successful, false if blob cannot be found or delete is not possible without exception.

Throws:

IOException - Exception encountered while deleting in azure storage.

delete

public boolean delete(String key)
               throws IOException

API implementation to delete a blob in the back end azure storage.

Parameters:

key - - key to the blob being deleted.

Returns:

return true when delete is successful, false if blob cannot be found or delete is not possible without exception.

Throws:

IOException - Exception encountered while deleting in azure storage.

rename

public void rename(String srcKey,
                   String dstKey)
            throws IOException

Throws:

IOException

rename

public void rename(String srcKey,
                   String dstKey,
                   boolean acquireLease,
                   SelfRenewingLease existingLease)
            throws IOException

Throws:

IOException

rename

public void rename(String srcKey,
                   String dstKey,
                   boolean acquireLease,
                   SelfRenewingLease existingLease,
                   boolean overwriteDestination)
            throws IOException

Throws:

IOException

explicitFileExists

public boolean explicitFileExists(String key)
                           throws AzureException

Checks whether an explicit file/folder exists. This is used by redo of atomic rename. There was a bug(apache jira HADOOP-12780) during atomic rename if process crashes after an inner directory has been renamed but still there are file under that directory to be renamed then after the process comes again it tries to redo the renames. It checks whether the directory exists or not by calling filesystem.exist. But filesystem.Exists will treat that directory as implicit directory and return true as file exists under that directory. So It will try try to rename that directory and will fail as the corresponding blob does not exist. So this method explicitly checks for the blob.

Throws:

AzureException

changePermissionStatus

public void changePermissionStatus(String key,
                                   org.apache.hadoop.fs.permission.PermissionStatus newPermission)
                            throws AzureException

Changes the permission status on the given key.

Throws:

AzureException

purge

public void purge(String prefix)
           throws IOException

Delete all keys with the given prefix. Used for testing.

Parameters:

prefix - prefix of objects to be deleted.

Throws:

IOException - Exception encountered while deleting keys.

acquireLease

public SelfRenewingLease acquireLease(String key)
                               throws AzureException

Get a lease on the blob identified by key. This lease will be renewed indefinitely by a background thread.

Throws:

AzureException

updateFolderLastModifiedTime

public void updateFolderLastModifiedTime(String key,
                                         Date lastModified,
                                         SelfRenewingLease folderLease)
                                  throws AzureException

Throws:

AzureException

updateFolderLastModifiedTime

public void updateFolderLastModifiedTime(String key,
                                         SelfRenewingLease folderLease)
                                  throws AzureException

Throws:

AzureException

dump

public void dump()
          throws IOException

Diagnostic method to dump state to the console.

Throws:

IOException - Exception encountered while dumping to console.

close

public void close()

finalize

protected void finalize()
                 throws Throwable

Overrides:

finalize in class Object

Throws:

Throwable

retrieveAppendStream

public DataOutputStream retrieveAppendStream(String key,
                                             int bufferSize)
                                      throws IOException

Throws:

IOException