org.finra.herd.service.helper

Class BusinessObjectDataDdlPartitionsHelper

java.lang.Object

org.finra.herd.service.helper.BusinessObjectDataDdlPartitionsHelper

@Component
public class BusinessObjectDataDdlPartitionsHelper
extends Object

Helper methods used for both DDL generator and Partitions generator

Field Summary

Fields

Modifier and Type	Field and Description
static String	NO_PARTITIONING_PARTITION_KEY The partition key value for business object data without partitioning.
static String	NO_PARTITIONING_PARTITION_VALUE The partition value for business object data without partitioning.
static String	REGEX_S3_EMPTY_PARTITION The regular expression that represents an empty partition in S3, this is because hadoop file system implements directory support in S3 by creating empty files with the "directoryname_$folder$" suffix.

Constructor Summary

Constructors

Constructor and Description
BusinessObjectDataDdlPartitionsHelper()

Method Summary

Modifier and Type	Method and Description
BusinessObjectDataDdlRequest	buildBusinessObjectDataDdlRequest(BusinessObjectDataPartitionsRequest request) Builds business object data DDL request object instance and initialise it with the business object data partitions request field values.
protected org.finra.herd.service.helper.BusinessObjectDataDdlPartitionsHelper.GenerateDdlRequestWrapper	buildGenerateDdlPartitionsWrapper(BusinessObjectDataDdlRequest request, BusinessObjectFormatEntity businessObjectFormatEntity, CustomDdlEntity customDdlEntity, BusinessObjectDataStatusEntity businessObjectDataStatusEntity, List<String> storageNames, List<StorageEntity> requestedStorageEntities, Map<String,StorageEntity> cachedStorageEntities, Map<String,String> cachedS3BucketNames) Generates the create table Hive 13 DDL as per specified business object data DDL request.
protected List<StorageUnitAvailabilityDto>	excludeDuplicateBusinessObjectData(List<StorageUnitAvailabilityDto> storageUnitAvailabilityDtos, List<String> storageNames) Eliminate storage units that belong to the same business object data by picking storage unit registered in a storage listed earlier in the list of storage names specified in the request.
List<HivePartitionDto>	getHivePartitions(BusinessObjectDataKey businessObjectDataKey, List<SchemaColumn> autoDiscoverableSubPartitionColumns, String s3KeyPrefix, Collection<String> storageFiles, String storageName) Gets a list of Hive partitions.
protected void	notAllowNonAvailableRegisteredSubPartitions(BusinessObjectDefinitionEntity businessObjectDefinitionEntity, String businessObjectFormatUsage, FileTypeEntity fileTypeEntity, Integer businessObjectFormatVersion, List<List<String>> matchedAvailablePartitionFilters, List<List<String>> availablePartitions, List<String> storageNames, List<StorageEntity> storageEntities, StoragePlatformEntity s3StoragePlatformEntity) Searches for and fails on any of "non-available" registered sub-partitions as per list of "matched" partition filters.
List<StorageUnitAvailabilityDto>	processPartitionFiltersForGenerateDdlPartitions(org.finra.herd.service.helper.BusinessObjectDataDdlPartitionsHelper.GenerateDdlRequestWrapper generateDdlRequest) Process partitionFilters to retrieve storage unit availability dtos
void	processStorageUnitsForGenerateDdlPartitions(org.finra.herd.service.helper.BusinessObjectDataDdlPartitionsHelper.GenerateDdlRequestWrapper generateDdlRequest, StringBuilder sb, List<Partition> partitions, HashMap<String,String> replacements, BusinessObjectFormat businessObjectFormatForSchema, String ifNotExistsOption, List<StorageUnitAvailabilityDto> storageUnitAvailabilityDtos) Adds the relative "alter table add partition" statements for each storage unit entity.
BusinessObjectFormat	validatePartitionFiltersAndFormat(org.finra.herd.service.helper.BusinessObjectDataDdlPartitionsHelper.GenerateDdlRequestWrapper generateDdlRequest) Validate partionFilters and business format

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NO_PARTITIONING_PARTITION_KEY

public static final String NO_PARTITIONING_PARTITION_KEY

The partition key value for business object data without partitioning.

See Also:

Constant Field Values

NO_PARTITIONING_PARTITION_VALUE

public static final String NO_PARTITIONING_PARTITION_VALUE

The partition value for business object data without partitioning.

See Also:

Constant Field Values

REGEX_S3_EMPTY_PARTITION

public static final String REGEX_S3_EMPTY_PARTITION

The regular expression that represents an empty partition in S3, this is because hadoop file system implements directory support in S3 by creating empty files with the "directoryname_$folder$" suffix.

See Also:

Constant Field Values

Constructor Detail

BusinessObjectDataDdlPartitionsHelper

public BusinessObjectDataDdlPartitionsHelper()

Method Detail

buildBusinessObjectDataDdlRequest

public BusinessObjectDataDdlRequest buildBusinessObjectDataDdlRequest(BusinessObjectDataPartitionsRequest request)

Builds business object data DDL request object instance and initialise it with the business object data partitions request field values.

Parameters:

request - the business object data partitions request

Returns:

the newly created BusinessObjectDataDdlRequest object instance

buildGenerateDdlPartitionsWrapper

protected org.finra.herd.service.helper.BusinessObjectDataDdlPartitionsHelper.GenerateDdlRequestWrapper buildGenerateDdlPartitionsWrapper(BusinessObjectDataDdlRequest request,
                                                                                                                                          BusinessObjectFormatEntity businessObjectFormatEntity,
                                                                                                                                          CustomDdlEntity customDdlEntity,
                                                                                                                                          BusinessObjectDataStatusEntity businessObjectDataStatusEntity,
                                                                                                                                          List<String> storageNames,
                                                                                                                                          List<StorageEntity> requestedStorageEntities,
                                                                                                                                          Map<String,StorageEntity> cachedStorageEntities,
                                                                                                                                          Map<String,String> cachedS3BucketNames)

Generates the create table Hive 13 DDL as per specified business object data DDL request.

Parameters:

request - the business object data DDL request

businessObjectFormatEntity - the business object format entity

customDdlEntity - the optional custom DDL entity

businessObjectDataStatusEntity - the business object data status for available business object data

storageNames - the list of storage names

requestedStorageEntities - the list of storage entities per storage names specified in the request

cachedStorageEntities - the map of storage names in upper case to the relative storage entities

cachedS3BucketNames - the map of storage names in upper case to the relative S3 bucket names

Returns:

the generateDDL and generatePartitions Wrapper class object

validatePartitionFiltersAndFormat

public BusinessObjectFormat validatePartitionFiltersAndFormat(org.finra.herd.service.helper.BusinessObjectDataDdlPartitionsHelper.GenerateDdlRequestWrapper generateDdlRequest)

Validate partionFilters and business format

Parameters:

generateDdlRequest - the generateDdlRequestWrapper object

Returns:

BusinessObjectFormat object

processPartitionFiltersForGenerateDdlPartitions

public List<StorageUnitAvailabilityDto> processPartitionFiltersForGenerateDdlPartitions(org.finra.herd.service.helper.BusinessObjectDataDdlPartitionsHelper.GenerateDdlRequestWrapper generateDdlRequest)

Process partitionFilters to retrieve storage unit availability dtos

Parameters:

generateDdlRequest - the generateDdlRequestWrapper object

Returns:

the list of storage unit availability dtos

processStorageUnitsForGenerateDdlPartitions

public void processStorageUnitsForGenerateDdlPartitions(org.finra.herd.service.helper.BusinessObjectDataDdlPartitionsHelper.GenerateDdlRequestWrapper generateDdlRequest,
                                                        StringBuilder sb,
                                                        List<Partition> partitions,
                                                        HashMap<String,String> replacements,
                                                        BusinessObjectFormat businessObjectFormatForSchema,
                                                        String ifNotExistsOption,
                                                        List<StorageUnitAvailabilityDto> storageUnitAvailabilityDtos)

Adds the relative "alter table add partition" statements for each storage unit entity. Please note that each request partition value might result in multiple available storage unit entities (subpartitions).

Parameters:

generateDdlRequest - the generate ddl request wrapper object

sb - the string builder to be updated with the "alter table add partition" statements

partitions - the list to store business object data partitions

replacements - the hash map of string values to be used to substitute the custom DDL tokens with their actual values

businessObjectFormatForSchema - the business object format to be used for schema

ifNotExistsOption - specifies if generated DDL contains "if not exists" option

storageUnitAvailabilityDtos - the list of storage unit availability DTOs

excludeDuplicateBusinessObjectData

protected List<StorageUnitAvailabilityDto> excludeDuplicateBusinessObjectData(List<StorageUnitAvailabilityDto> storageUnitAvailabilityDtos,
                                                                              List<String> storageNames)
                                                                       throws IllegalArgumentException

Eliminate storage units that belong to the same business object data by picking storage unit registered in a storage listed earlier in the list of storage names specified in the request. If storage names are not specified, simply fail on business object data instances registered with multiple storage.

Parameters:

storageUnitAvailabilityDtos - the list of storage unit availability DTOs

storageNames - the list of storage names

Returns:

the updated list of storage unit availability DTOs

Throws:

IllegalArgumentException - on business object data being registered in multiple storage and storage names are not specified to resolve this

notAllowNonAvailableRegisteredSubPartitions

protected void notAllowNonAvailableRegisteredSubPartitions(BusinessObjectDefinitionEntity businessObjectDefinitionEntity,
                                                           String businessObjectFormatUsage,
                                                           FileTypeEntity fileTypeEntity,
                                                           Integer businessObjectFormatVersion,
                                                           List<List<String>> matchedAvailablePartitionFilters,
                                                           List<List<String>> availablePartitions,
                                                           List<String> storageNames,
                                                           List<StorageEntity> storageEntities,
                                                           StoragePlatformEntity s3StoragePlatformEntity)

Searches for and fails on any of "non-available" registered sub-partitions as per list of "matched" partition filters.

Parameters:

businessObjectDefinitionEntity - the business object definition entity

businessObjectFormatUsage - the business object format usage (case-insensitive)

fileTypeEntity - the file type entity

businessObjectFormatVersion - the optional business object format version. If a business object format version isn't specified, the latest available format version for each partition value will be used

matchedAvailablePartitionFilters - the list of "matched" partition filters

availablePartitions - the list of already discovered "available" partitions, where each partition consists of primary and optional sub-partition values

storageNames - the list of storage names

storageEntities - the list of storage entities

s3StoragePlatformEntity - the S3 storage platform entity

getHivePartitions

public List<HivePartitionDto> getHivePartitions(BusinessObjectDataKey businessObjectDataKey,
                                                List<SchemaColumn> autoDiscoverableSubPartitionColumns,
                                                String s3KeyPrefix,
                                                Collection<String> storageFiles,
                                                String storageName)

Gets a list of Hive partitions. For single level partitioning, no auto-discovery of sub-partitions (sub-directories) is needed - the business object data will be represented by a single Hive partition instance. For multiple level partitioning, this method performs an auto-discovery of all sub-partitions (sub-directories) and creates a Hive partition object instance for each partition.

Parameters:

businessObjectDataKey - the business object data key

autoDiscoverableSubPartitionColumns - the auto-discoverable sub-partition columns

s3KeyPrefix - the S3 key prefix

storageFiles - the storage files

storageName - the storage name

Returns:

the list of Hive partitions