org.finra.herd.service.helper

Class Hive13DdlGenerator

java.lang.Object

org.finra.herd.service.helper.DdlGenerator

org.finra.herd.service.helper.Hive13DdlGenerator

@Component
public class Hive13DdlGenerator
extends DdlGenerator

The DDL generator for Hive 13.

Field Summary

Fields

Modifier and Type	Field and Description
static String	NO_PARTITIONING_PARTITION_KEY The partition key value for business object data without partitioning.
static String	NO_PARTITIONING_PARTITION_VALUE The partition value for business object data without partitioning.
static String	ORC_HIVE_FILE_FORMAT Hive file format for ORC files.
static String	PARQUET_HIVE_FILE_FORMAT Hive file format for PARQUET files.
static String	TEXT_HIVE_FILE_FORMAT Hive file format for text files.

Fields inherited from class org.finra.herd.service.helper.DdlGenerator

NON_PARTITIONED_TABLE_LOCATION_CUSTOM_DDL_TOKEN, TABLE_NAME_CUSTOM_DDL_TOKEN

Constructor Summary

Constructors

Constructor and Description
Hive13DdlGenerator()

Method Summary

Modifier and Type	Method and Description
String	escapeSingleQuotes(String string) Escapes single quote characters, if not already escaped, with an extra backslash.
protected List<StorageUnitAvailabilityDto>	excludeDuplicateBusinessObjectData(List<StorageUnitAvailabilityDto> storageUnitAvailabilityDtos, List<String> storageNames) Eliminate storage units that belong to the same business object data by picking storage unit registered in a storage listed earlier in the list of storage names specified in the request.
String	generateCreateTableDdl(BusinessObjectDataDdlRequest request, BusinessObjectFormatEntity businessObjectFormatEntity, CustomDdlEntity customDdlEntity, List<String> storageNames, Map<String,StorageEntity> storageEntities, Map<String,String> s3BucketNames) Generates the create table Hive 13 DDL as per specified business object data DDL request.
String	generateCreateTableDdl(BusinessObjectFormatDdlRequest request, BusinessObjectFormatEntity businessObjectFormatEntity, CustomDdlEntity customDdlEntity) Generates the create table Hive 13 DDL as per specified business object format DDL request.
String	generateReplaceColumnsStatement(BusinessObjectFormatDdlRequest request, BusinessObjectFormatEntity businessObjectFormatEntity)
String	getDdlCharacterValue(String string) Gets the DDL character value based on the specified configured character value.
String	getDdlCharacterValue(String string, boolean escapeSingleBackslash) Gets the DDL character value based on the specified configured character value.
BusinessObjectDataDdlOutputFormatEnum	getDdlOutputFormat() Gets the supported DDL output format that this generator can support.
List<HivePartitionDto>	getHivePartitions(BusinessObjectDataKey businessObjectDataKey, List<SchemaColumn> autoDiscoverableSubPartitionColumns, String s3KeyPrefix, Collection<String> storageFiles, String storageName) Gets a list of Hive partitions.
Pattern	getHivePathPattern(List<SchemaColumn> partitionColumns) Gets a pattern to match Hive partition sub-directories.
protected void	notAllowNonAvailableRegisteredSubPartitions(BusinessObjectFormatKey businessObjectFormatKey, List<List<String>> matchedAvailablePartitionFilters, List<List<String>> availablePartitions, List<String> storageNames) Searches for and fails on any of "non-available" registered sub-partitions as per list of "matched" partition filters.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NO_PARTITIONING_PARTITION_KEY

public static final String NO_PARTITIONING_PARTITION_KEY

The partition key value for business object data without partitioning.

See Also:

Constant Field Values

NO_PARTITIONING_PARTITION_VALUE

public static final String NO_PARTITIONING_PARTITION_VALUE

The partition value for business object data without partitioning.

See Also:

Constant Field Values

ORC_HIVE_FILE_FORMAT

public static final String ORC_HIVE_FILE_FORMAT

Hive file format for ORC files.

See Also:

Constant Field Values

PARQUET_HIVE_FILE_FORMAT

public static final String PARQUET_HIVE_FILE_FORMAT

Hive file format for PARQUET files.

See Also:

Constant Field Values

TEXT_HIVE_FILE_FORMAT

public static final String TEXT_HIVE_FILE_FORMAT

Hive file format for text files.

See Also:

Constant Field Values

Constructor Detail

Hive13DdlGenerator

public Hive13DdlGenerator()

Method Detail

escapeSingleQuotes

public String escapeSingleQuotes(String string)

Escapes single quote characters, if not already escaped, with an extra backslash.

Parameters:

string - the input text

Returns:

the output text with all single quote characters escaped by an extra backslash

generateCreateTableDdl

public String generateCreateTableDdl(BusinessObjectDataDdlRequest request,
                                     BusinessObjectFormatEntity businessObjectFormatEntity,
                                     CustomDdlEntity customDdlEntity,
                                     List<String> storageNames,
                                     Map<String,StorageEntity> storageEntities,
                                     Map<String,String> s3BucketNames)

Generates the create table Hive 13 DDL as per specified business object data DDL request.

Specified by:

generateCreateTableDdl in class DdlGenerator

Parameters:

request - the business object data DDL request

businessObjectFormatEntity - the business object format entity

customDdlEntity - the optional custom DDL entity

storageNames - the list of storage names

storageEntities - the map of storage names in upper case to the relative storage entities

s3BucketNames - the map of storage names in upper case to the relative S3 bucket names

Returns:

the create table Hive DDL

generateCreateTableDdl

public String generateCreateTableDdl(BusinessObjectFormatDdlRequest request,
                                     BusinessObjectFormatEntity businessObjectFormatEntity,
                                     CustomDdlEntity customDdlEntity)

Generates the create table Hive 13 DDL as per specified business object format DDL request.

Specified by:

generateCreateTableDdl in class DdlGenerator

Parameters:

request - the business object format DDL request

businessObjectFormatEntity - the business object format entity

customDdlEntity - the optional custom DDL entity

Returns:

the create table Hive DDL

generateReplaceColumnsStatement

public String generateReplaceColumnsStatement(BusinessObjectFormatDdlRequest request,
                                              BusinessObjectFormatEntity businessObjectFormatEntity)

Specified by:

generateReplaceColumnsStatement in class DdlGenerator

getDdlCharacterValue

public String getDdlCharacterValue(String string)

Gets the DDL character value based on the specified configured character value. This method supports UTF-8 encoded strings and will "Hive" escape any non-ASCII printable characters using '\(value)'.

Parameters:

string - the configured character value.

Returns:

the DDL character value.

getDdlCharacterValue

public String getDdlCharacterValue(String string,
                                   boolean escapeSingleBackslash)

Gets the DDL character value based on the specified configured character value. This method supports UTF-8 encoded strings and will "Hive" escape any non-ASCII printable characters using '\(value)'.

Parameters:

string - the configured character value.

escapeSingleBackslash - specifies if we need to escape a single backslash character with an extra backslash

Returns:

the DDL character value.

getDdlOutputFormat

public BusinessObjectDataDdlOutputFormatEnum getDdlOutputFormat()

Description copied from class: DdlGenerator

Gets the supported DDL output format that this generator can support.

Specified by:

getDdlOutputFormat in class DdlGenerator

Returns:

BusinessObjectDataDdlOutputFormatEnum supported output format.

getHivePartitions

public List<HivePartitionDto> getHivePartitions(BusinessObjectDataKey businessObjectDataKey,
                                                List<SchemaColumn> autoDiscoverableSubPartitionColumns,
                                                String s3KeyPrefix,
                                                Collection<String> storageFiles,
                                                String storageName)

Gets a list of Hive partitions. For single level partitioning, no auto-discovery of sub-partitions (sub-directories) is needed - the business object data will be represented by a single Hive partition instance. For multiple level partitioning, this method performs an auto-discovery of all sub-partitions (sub-directories) and creates a Hive partition object instance for each partition.

Parameters:

businessObjectDataKey - the business object data key

autoDiscoverableSubPartitionColumns - the auto-discoverable sub-partition columns

s3KeyPrefix - the S3 key prefix

storageFiles - the storage files

storageName - the storage name

Returns:

the list of Hive partitions

getHivePathPattern

public Pattern getHivePathPattern(List<SchemaColumn> partitionColumns)

Gets a pattern to match Hive partition sub-directories.

Parameters:

partitionColumns - the list of partition columns

Returns:

the newly created pattern to match Hive partition sub-directories.

excludeDuplicateBusinessObjectData

protected List<StorageUnitAvailabilityDto> excludeDuplicateBusinessObjectData(List<StorageUnitAvailabilityDto> storageUnitAvailabilityDtos,
                                                                              List<String> storageNames)
                                                                       throws IllegalArgumentException

Eliminate storage units that belong to the same business object data by picking storage unit registered in a storage listed earlier in the list of storage names specified in the request. If storage names are not specified, simply fail on business object data instances registered with multiple storage.

Parameters:

storageUnitAvailabilityDtos - the list of storage unit availability DTOs

storageNames - the list of storage names

Returns:

the updated list of storage unit availability DTOs

Throws:

IllegalArgumentException - on business object data being registered in multiple storage and storage names are not specified to resolve this

notAllowNonAvailableRegisteredSubPartitions

protected void notAllowNonAvailableRegisteredSubPartitions(BusinessObjectFormatKey businessObjectFormatKey,
                                                           List<List<String>> matchedAvailablePartitionFilters,
                                                           List<List<String>> availablePartitions,
                                                           List<String> storageNames)

Searches for and fails on any of "non-available" registered sub-partitions as per list of "matched" partition filters.

Parameters:

businessObjectFormatKey - the business object format key

matchedAvailablePartitionFilters - the list of "matched" partition filters

availablePartitions - the list of already discovered "available" partitions, where each partition consists of primary and optional sub-partition values

storageNames - the list of storage names