org.finra.herd.dao

Interface S3Dao

All Known Implementing Classes:

S3DaoImpl

public interface S3Dao

A DAO for Amazon AWS S3.

Field Summary

Fields

Modifier and Type	Field and Description
static String	SIGNER_OVERRIDE_V4 The AWS signer type value to use when signing with Signature Version 4.

Method Summary

Modifier and Type	Method and Description
int	abortMultipartUploads(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto, Date thresholdDate) Aborts any multipart uploads that were initiated in the specified S3 storage older than threshold date.
S3FileTransferResultsDto	copyFile(S3FileCopyRequestParamsDto s3FileCopyRequestParamsDto) Copies an S3 object from the source S3 bucket to the same path in target bucket.
void	createDirectory(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto) Creates an S3 object of 0 byte size that represents a directory.
void	deleteDirectory(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto) Deletes keys/key versions from specified bucket with matching prefix.
void	deleteFileList(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto) Deletes a list of keys from specified bucket.
S3FileTransferResultsDto	downloadDirectory(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto) Downloads a directory from S3 to the local file system.
S3FileTransferResultsDto	downloadFile(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto) Downloads a file from S3 to the local file system.
String	generateGetObjectPresignedUrl(String bucketName, String key, Date expiration, S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto) Generates a GET pre-signed URL for the given object in S3 identified by its bucket name and key.
com.amazonaws.services.s3.model.ObjectMetadata	getObjectMetadata(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto) Gets the metadata for the specified Amazon S3 object without actually fetching the object itself.
Properties	getProperties(String bucketName, String key, S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto) Gets an object from S3 and parses it as a Properties.
List<com.amazonaws.services.s3.model.S3ObjectSummary>	listDirectory(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto) Lists all S3 objects matching the S3 key prefix in the given bucket (S3 bucket name).
List<com.amazonaws.services.s3.model.S3ObjectSummary>	listDirectory(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto, boolean ignoreZeroByteDirectoryMarkers) Lists all S3 objects matching the S3 key prefix in the given bucket (S3 bucket name).
List<com.amazonaws.services.s3.model.S3VersionSummary>	listVersions(S3FileTransferRequestParamsDto params) Lists all S3 versions matching the S3 key prefix in the given bucket (S3 bucket name).
void	restoreObjects(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto, int expirationInDays, String archiveRetrievalOption) Requests to restore a list of keys in the specified bucket.
boolean	s3FileExists(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto) Validates that file exists in S3 using S3Client.getObject() method.
void	tagObjects(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto, S3FileTransferRequestParamsDto s3ObjectTaggerParamsDto, List<com.amazonaws.services.s3.model.S3ObjectSummary> s3ObjectSummaries, com.amazonaws.services.s3.model.Tag tag) Tags S3 objects with the specified S3 object tag.
void	tagVersions(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto, S3FileTransferRequestParamsDto s3ObjectTaggerParamsDto, List<com.amazonaws.services.s3.model.S3VersionSummary> s3VersionSummaries, com.amazonaws.services.s3.model.Tag tag) Tags S3 versions with the specified S3 object tag.
S3FileTransferResultsDto	uploadDirectory(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto) Uploads a local directory of files into S3.
S3FileTransferResultsDto	uploadFile(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto) Uploads a local file into S3.
S3FileTransferResultsDto	uploadFileList(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto) Uploads a list of local files into S3.
void	validateGlacierS3FilesRestored(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto) Validates that all specified Glacier storage class files are restored.
void	validateS3File(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto, Long fileSizeInBytes) Validates that file exists in S3 and has size that equals to a specified value.

Field Detail

SIGNER_OVERRIDE_V4

static final String SIGNER_OVERRIDE_V4

The AWS signer type value to use when signing with Signature Version 4. This constant needs to be the same as what is defined in AmazonS3Client.S3_V4_SIGNER. Unfortunately, that one is private so we have to create our own duplicate constant here.

See Also:

Constant Field Values

Method Detail

abortMultipartUploads

int abortMultipartUploads(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto,
                          Date thresholdDate)

Aborts any multipart uploads that were initiated in the specified S3 storage older than threshold date.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name specifies the name of the bucket containing the multipart uploads to abort.

thresholdDate - the date indicating which multipart uploads should be aborted

Returns:

the total number of aborted multipart uploads

copyFile

S3FileTransferResultsDto copyFile(S3FileCopyRequestParamsDto s3FileCopyRequestParamsDto)
                           throws InterruptedException

Copies an S3 object from the source S3 bucket to the same path in target bucket. This method does not delete the source S3 object.

Parameters:

s3FileCopyRequestParamsDto - the S3 file copy request parameters.

Returns:

the results.

Throws:

InterruptedException - if any problems were encountered.

createDirectory

void createDirectory(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto)

Creates an S3 object of 0 byte size that represents a directory.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name and S3 key prefix identify the S3 object to be created.

deleteDirectory

void deleteDirectory(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto)

Deletes keys/key versions from specified bucket with matching prefix.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name and S3 key prefix identify the S3 objects to be deleted.

deleteFileList

void deleteFileList(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto)

Deletes a list of keys from specified bucket.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name and the file list identify the S3 objects to be deleted.

downloadDirectory

S3FileTransferResultsDto downloadDirectory(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto)
                                    throws InterruptedException

Downloads a directory from S3 to the local file system.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name and S3 key prefix are for the target of the copy. The local path is the local file to be copied.

Returns:

the results.

Throws:

InterruptedException - if any problems were encountered.

downloadFile

S3FileTransferResultsDto downloadFile(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto)
                               throws InterruptedException

Downloads a file from S3 to the local file system.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name and S3 key prefix are for the target of the copy. The local path is the local file to be copied.

Returns:

the results.

Throws:

InterruptedException - if any problems were encountered.

generateGetObjectPresignedUrl

String generateGetObjectPresignedUrl(String bucketName,
                                     String key,
                                     Date expiration,
                                     S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto)

Generates a GET pre-signed URL for the given object in S3 identified by its bucket name and key. Uses the proxy information and signer override specified in the given S3FileTransferRequestParamsDto.

Parameters:

bucketName - The bucket name of the object

key - The S3 key to the object

expiration - The expiration date at which point the new pre-signed URL will no longer be accepted by Amazon S3.

s3FileTransferRequestParamsDto - The parameters which contain information on how to use S3

Returns:

a pre-signed URL

getObjectMetadata

com.amazonaws.services.s3.model.ObjectMetadata getObjectMetadata(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto)

Gets the metadata for the specified Amazon S3 object without actually fetching the object itself.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name and S3 key prefix identify the S3 object's whose metadata is being retrieved.

Returns:

null if object key is not found, otherwise all Amazon S3 object metadata for the specified object

getProperties

Properties getProperties(String bucketName,
                         String key,
                         S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto)

Gets an object from S3 and parses it as a Properties.

Parameters:

bucketName - The S3 bucket name

key - The S3 object key

s3FileTransferRequestParamsDto - S3FileTransferRequestParamsDto with proxy information

Returns:

Properties

listDirectory

List<com.amazonaws.services.s3.model.S3ObjectSummary> listDirectory(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto)

Lists all S3 objects matching the S3 key prefix in the given bucket (S3 bucket name).

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name and S3 key prefix identify the S3 objects to get listed.

Returns:

the list of all S3 objects represented by S3 object summaries that match the prefix in the given bucket.

listDirectory

List<com.amazonaws.services.s3.model.S3ObjectSummary> listDirectory(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto,
                                                                    boolean ignoreZeroByteDirectoryMarkers)

Lists all S3 objects matching the S3 key prefix in the given bucket (S3 bucket name).

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name and S3 key prefix identify the S3 objects to get listed.

ignoreZeroByteDirectoryMarkers - specifies whether to ignore 0 byte objects that represent S3 directories.

Returns:

the list of all S3 objects represented by S3 object summaries that match the prefix in the given bucket.

listVersions

List<com.amazonaws.services.s3.model.S3VersionSummary> listVersions(S3FileTransferRequestParamsDto params)

Lists all S3 versions matching the S3 key prefix in the given bucket (S3 bucket name). The S3 bucket name and S3 key prefix that identify the S3 versions to get listed are taken from the S3 file transfer request parameters DTO.

Parameters:

params - the S3 file transfer request parameters

Returns:

the list of all S3 versions that match the prefix in the given bucket

restoreObjects

void restoreObjects(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto,
                    int expirationInDays,
                    String archiveRetrievalOption)

Requests to restore a list of keys in the specified bucket.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name and the file list identify the S3 objects to be restored

expirationInDays - the time, in days, between when an object is restored to the bucket and when it expires

archiveRetrievalOption - the archive retrieval option when restoring an archived object

s3FileExists

boolean s3FileExists(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto)
              throws RuntimeException

Validates that file exists in S3 using S3Client.getObject() method.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name and S3 key prefix identify the S3 file

Returns:

true if the S3 file exists, false otherwise

Throws:

RuntimeException - if file existence check fails

tagObjects

void tagObjects(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto,
                S3FileTransferRequestParamsDto s3ObjectTaggerParamsDto,
                List<com.amazonaws.services.s3.model.S3ObjectSummary> s3ObjectSummaries,
                com.amazonaws.services.s3.model.Tag tag)

Tags S3 objects with the specified S3 object tag.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. This set of parameters contains the S3 bucket name

s3ObjectTaggerParamsDto - the S3 file transfer request parameters to be used for tagging S3 objects

s3ObjectSummaries - the list of S3 objects to be tagged

tag - the S3 object tag

tagVersions

void tagVersions(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto,
                 S3FileTransferRequestParamsDto s3ObjectTaggerParamsDto,
                 List<com.amazonaws.services.s3.model.S3VersionSummary> s3VersionSummaries,
                 com.amazonaws.services.s3.model.Tag tag)

Tags S3 versions with the specified S3 object tag.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. This set of parameters contains the S3 bucket name

s3ObjectTaggerParamsDto - the S3 file transfer request parameters to be used for tagging S3 objects

s3VersionSummaries - the list of S3 versions to be tagged

tag - the S3 object tag

uploadDirectory

S3FileTransferResultsDto uploadDirectory(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto)
                                  throws InterruptedException

Uploads a local directory of files into S3.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name and S3 key prefix are for the target of the copy. The local path is the local file to be copied.

Returns:

the results.

Throws:

InterruptedException - if any problems were encountered.

uploadFile

S3FileTransferResultsDto uploadFile(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto)
                             throws InterruptedException

Uploads a local file into S3.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name and S3 key prefix are for the target of the copy. The local path is the local file to be copied.

Returns:

the file transfer results.

Throws:

InterruptedException - if any problems were encountered.

uploadFileList

S3FileTransferResultsDto uploadFileList(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto)
                                 throws InterruptedException

Uploads a list of local files into S3.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name, S3 key prefix, and the file list (files) are for the target of the copy. The local path and the file list (files) are the local files to be copied. The keys of the files are calculated relative to the common parent directory (local path) and the S3 key prefix.

Returns:

the file transfer results.

Throws:

InterruptedException - if any problems were encountered.

validateGlacierS3FilesRestored

void validateGlacierS3FilesRestored(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto)
                             throws RuntimeException

Validates that all specified Glacier storage class files are restored.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name and the file list identify the S3 objects to be validated

Throws:

RuntimeException - if file validation fails

validateS3File

void validateS3File(S3FileTransferRequestParamsDto s3FileTransferRequestParamsDto,
                    Long fileSizeInBytes)
             throws RuntimeException

Validates that file exists in S3 and has size that equals to a specified value.

Parameters:

s3FileTransferRequestParamsDto - the S3 file transfer request parameters. The S3 bucket name and S3 key prefix identify the S3 file.

fileSizeInBytes - the expected file size in bytes

Throws:

RuntimeException - if file validation fails