org.apache.hadoop.fs.azurebfs

Class AzureBlobFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.security.token.DelegationTokenIssuer

Direct Known Subclasses:

SecureAzureBlobFileSystem

@InterfaceStability.Evolving
public class AzureBlobFileSystem
extends org.apache.hadoop.fs.FileSystem

A FileSystem for reading and writing files stored on Windows Azure

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.DirListingIterator<T extends org.apache.hadoop.fs.FileStatus>, org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Constructor Summary

Constructors

Constructor and Description
AzureBlobFileSystem()

Method Summary

Modifier and Type	Method and Description
void	access(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsAction mode) Checks if the user can access a path.
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path f, int bufferSize, org.apache.hadoop.util.Progressable progress)
void	close()
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path f, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flags, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
boolean	delete(org.apache.hadoop.fs.Path f, boolean recursive)
protected void	finalize()
org.apache.hadoop.fs.permission.AclStatus	getAclStatus(org.apache.hadoop.fs.Path path) Gets the ACL of a file or directory.
org.apache.hadoop.security.token.Token<?>	getDelegationToken(String renewer) Get a delegation token from remote service endpoint if 'fs.azure.enable.kerberos.support' is set to 'true', and 'fs.azure.enable.delegation.token' is set to 'true'.
org.apache.hadoop.fs.BlockLocation[]	getFileBlockLocations(org.apache.hadoop.fs.FileStatus file, long start, long len) Return an array containing hostnames, offset and size of portions of the given file.
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path f)
org.apache.hadoop.fs.Path	getHomeDirectory()
String	getOwnerUser() Get the username of the FS.
String	getOwnerUserPrimaryGroup() Get the group name of the owner of the FS.
String	getScheme()
URI	getUri()
org.apache.hadoop.fs.Path	getWorkingDirectory()
void	initialize(URI uri, org.apache.hadoop.conf.Configuration configuration)
boolean	isSecureScheme()
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path f)
org.apache.hadoop.fs.Path	makeQualified(org.apache.hadoop.fs.Path path) Qualify a path to one which uses this FileSystem and, if relative, made absolute.
boolean	mkdirs(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission)
void	modifyAclEntries(org.apache.hadoop.fs.Path path, List<org.apache.hadoop.fs.permission.AclEntry> aclSpec) Modifies ACL entries of files and directories.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path path, int bufferSize)
void	removeAcl(org.apache.hadoop.fs.Path path) Removes all but the base ACL entries of files and directories.
void	removeAclEntries(org.apache.hadoop.fs.Path path, List<org.apache.hadoop.fs.permission.AclEntry> aclSpec) Removes ACL entries from files and directories.
void	removeDefaultAcl(org.apache.hadoop.fs.Path path) Removes all default ACL entries from files and directories.
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
void	setAcl(org.apache.hadoop.fs.Path path, List<org.apache.hadoop.fs.permission.AclEntry> aclSpec) Fully replaces ACL of files and directories, discarding all existing entries.
void	setOwner(org.apache.hadoop.fs.Path path, String owner, String group) Set owner of a path (i.e.
void	setPermission(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission) Set permission of a path.
void	setWorkingDirectory(org.apache.hadoop.fs.Path newDir)
String	toString()

Methods inherited from class org.apache.hadoop.fs.FileSystem

append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createFile, createNewFile, createPathHandle, createSnapshot, createSnapshot, createSymlink, delete, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAdditionalTokenIssuers, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalServiceName, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultBlockSize, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, listXAttrs, mkdirs, mkdirs, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setDefaultUri, setDefaultUri, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

addDelegationTokens, collectDelegationTokens

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

AzureBlobFileSystem

public AzureBlobFileSystem()

Method Detail

initialize

public void initialize(URI uri,
                       org.apache.hadoop.conf.Configuration configuration)
                throws IOException

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object

isSecureScheme

public boolean isSecureScheme()

getUri

public URI getUri()

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path path,
                                                   int bufferSize)
                                            throws IOException

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path f,
                                                      org.apache.hadoop.fs.permission.FsPermission permission,
                                                      boolean overwrite,
                                                      int bufferSize,
                                                      short replication,
                                                      long blockSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

createNonRecursive

public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path f,
                                                                  org.apache.hadoop.fs.permission.FsPermission permission,
                                                                  boolean overwrite,
                                                                  int bufferSize,
                                                                  short replication,
                                                                  long blockSize,
                                                                  org.apache.hadoop.util.Progressable progress)
                                                           throws IOException

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

createNonRecursive

public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path f,
                                                                  org.apache.hadoop.fs.permission.FsPermission permission,
                                                                  EnumSet<org.apache.hadoop.fs.CreateFlag> flags,
                                                                  int bufferSize,
                                                                  short replication,
                                                                  long blockSize,
                                                                  org.apache.hadoop.util.Progressable progress)
                                                           throws IOException

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

createNonRecursive

public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path f,
                                                                  boolean overwrite,
                                                                  int bufferSize,
                                                                  short replication,
                                                                  long blockSize,
                                                                  org.apache.hadoop.util.Progressable progress)
                                                           throws IOException

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path f,
                                                      int bufferSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

rename

public boolean rename(org.apache.hadoop.fs.Path src,
                      org.apache.hadoop.fs.Path dst)
               throws IOException

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

delete

public boolean delete(org.apache.hadoop.fs.Path f,
                      boolean recursive)
               throws IOException

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f)
                                             throws IOException

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path f,
                      org.apache.hadoop.fs.permission.FsPermission permission)
               throws IOException

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f)
                                              throws IOException

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

makeQualified

public org.apache.hadoop.fs.Path makeQualified(org.apache.hadoop.fs.Path path)

Qualify a path to one which uses this FileSystem and, if relative, made absolute.

Overrides:

makeQualified in class org.apache.hadoop.fs.FileSystem

Parameters:

path - to qualify.

Returns:

this path if it contains a scheme and authority and is absolute, or a new path that includes a path and authority and is fully qualified

Throws:

IllegalArgumentException - if the path has a schema/URI different from this FileSystem.

See Also:

Path.makeQualified(URI, Path)

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path newDir)

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

getScheme

public String getScheme()

Overrides:

getScheme in class org.apache.hadoop.fs.FileSystem

getHomeDirectory

public org.apache.hadoop.fs.Path getHomeDirectory()

Overrides:

getHomeDirectory in class org.apache.hadoop.fs.FileSystem

getFileBlockLocations

public org.apache.hadoop.fs.BlockLocation[] getFileBlockLocations(org.apache.hadoop.fs.FileStatus file,
                                                                  long start,
                                                                  long len)

Return an array containing hostnames, offset and size of portions of the given file. For ABFS we'll just lie and give fake hosts to make sure we get many splits in MR jobs.

Overrides:

getFileBlockLocations in class org.apache.hadoop.fs.FileSystem

finalize

protected void finalize()
                 throws Throwable

Overrides:

finalize in class Object

Throws:

Throwable

getOwnerUser

public String getOwnerUser()

Get the username of the FS.

Returns:

the short name of the user who instantiated the FS

getOwnerUserPrimaryGroup

public String getOwnerUserPrimaryGroup()

Get the group name of the owner of the FS.

Returns:

primary group name

setOwner

public void setOwner(org.apache.hadoop.fs.Path path,
                     String owner,
                     String group)
              throws IOException

Set owner of a path (i.e. a file or a directory). The parameters owner and group cannot both be null.

Overrides:

setOwner in class org.apache.hadoop.fs.FileSystem

Parameters:

path - The path

owner - If it is null, the original username remains unchanged.

group - If it is null, the original groupname remains unchanged.

Throws:

IOException

setPermission

public void setPermission(org.apache.hadoop.fs.Path path,
                          org.apache.hadoop.fs.permission.FsPermission permission)
                   throws IOException

Set permission of a path.

Overrides:

setPermission in class org.apache.hadoop.fs.FileSystem

Parameters:

path - The path

permission - Access permission

Throws:

IOException

modifyAclEntries

public void modifyAclEntries(org.apache.hadoop.fs.Path path,
                             List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)
                      throws IOException

Modifies ACL entries of files and directories. This method can add new ACL entries or modify the permissions on existing ACL entries. All existing ACL entries that are not specified in this call are retained without changes. (Modifications are merged into the current ACL.)

Overrides:

modifyAclEntries in class org.apache.hadoop.fs.FileSystem

Parameters:

path - Path to modify

aclSpec - List of AbfsAclEntry describing modifications

Throws:

IOException - if an ACL could not be modified

removeAclEntries

public void removeAclEntries(org.apache.hadoop.fs.Path path,
                             List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)
                      throws IOException

Removes ACL entries from files and directories. Other ACL entries are retained.

Overrides:

removeAclEntries in class org.apache.hadoop.fs.FileSystem

Parameters:

path - Path to modify

aclSpec - List of AclEntry describing entries to remove

Throws:

IOException - if an ACL could not be modified

removeDefaultAcl

public void removeDefaultAcl(org.apache.hadoop.fs.Path path)
                      throws IOException

Removes all default ACL entries from files and directories.

Overrides:

removeDefaultAcl in class org.apache.hadoop.fs.FileSystem

Parameters:

path - Path to modify

Throws:

IOException - if an ACL could not be modified

removeAcl

public void removeAcl(org.apache.hadoop.fs.Path path)
               throws IOException

Removes all but the base ACL entries of files and directories. The entries for user, group, and others are retained for compatibility with permission bits.

Overrides:

removeAcl in class org.apache.hadoop.fs.FileSystem

Parameters:

path - Path to modify

Throws:

IOException - if an ACL could not be removed

setAcl

public void setAcl(org.apache.hadoop.fs.Path path,
                   List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)
            throws IOException

Fully replaces ACL of files and directories, discarding all existing entries.

Overrides:

setAcl in class org.apache.hadoop.fs.FileSystem

Parameters:

path - Path to modify

aclSpec - List of AclEntry describing modifications, must include entries for user, group, and others for compatibility with permission bits.

Throws:

IOException - if an ACL could not be modified

getAclStatus

public org.apache.hadoop.fs.permission.AclStatus getAclStatus(org.apache.hadoop.fs.Path path)
                                                       throws IOException

Gets the ACL of a file or directory.

Overrides:

getAclStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

path - Path to get

Returns:

AbfsAclStatus describing the ACL of the file or directory

Throws:

IOException - if an ACL could not be read

access

public void access(org.apache.hadoop.fs.Path path,
                   org.apache.hadoop.fs.permission.FsAction mode)
            throws IOException

Checks if the user can access a path. The mode specifies which access checks to perform. If the requested permissions are granted, then the method returns normally. If access is denied, then the method throws an AccessControlException.

Overrides:

access in class org.apache.hadoop.fs.FileSystem

Parameters:

path - Path to check

mode - type of access to check

Throws:

org.apache.hadoop.security.AccessControlException - if access is denied

FileNotFoundException - if the path does not exist

IOException - see specific implementation

getDelegationToken

public org.apache.hadoop.security.token.Token<?> getDelegationToken(String renewer)
                                                             throws IOException

Get a delegation token from remote service endpoint if 'fs.azure.enable.kerberos.support' is set to 'true', and 'fs.azure.enable.delegation.token' is set to 'true'.

Specified by:

getDelegationToken in interface org.apache.hadoop.security.token.DelegationTokenIssuer

Overrides:

getDelegationToken in class org.apache.hadoop.fs.FileSystem

Parameters:

renewer - the account name that is allowed to renew the token.

Returns:

delegation token

Throws:

IOException - thrown when getting the current user.