org.finra.herd.service.helper

Class Hive13DdlGenerator

java.lang.Object

org.finra.herd.service.helper.DdlGenerator

org.finra.herd.service.helper.Hive13DdlGenerator

@Component
public class Hive13DdlGenerator
extends DdlGenerator

The DDL generator for Hive 13.

Field Summary

Fields

Modifier and Type	Field and Description
static String	JSON_HIVE_FILE_FORMAT Hive file format for JSON files.
static String	NO_PARTITIONING_PARTITION_KEY The partition key value for business object data without partitioning.
static String	NO_PARTITIONING_PARTITION_VALUE The partition value for business object data without partitioning.
static String	ORC_HIVE_FILE_FORMAT Hive file format for ORC files.
static String	PARQUET_HIVE_FILE_FORMAT Hive file format for PARQUET files.
static String	REGEX_S3_EMPTY_PARTITION The regular expression that represents an empty partition in S3, this is because hadoop file system implements directory support in S3 by creating empty files with the "directoryname_$folder$" suffix.
static String	TEXT_HIVE_FILE_FORMAT Hive file format for text files.

Fields inherited from class org.finra.herd.service.helper.DdlGenerator

NON_PARTITIONED_TABLE_LOCATION_CUSTOM_DDL_TOKEN, PARTITIONED_TABLE_LOCATION_CUSTOM_DDL_TOKEN, TABLE_NAME_CUSTOM_DDL_TOKEN

Constructor Summary

Constructors

Constructor and Description
Hive13DdlGenerator()

Method Summary

Modifier and Type	Method and Description
String	escapeSingleQuotes(String string) Escapes single quote characters, if not already escaped, with an extra backslash.
String	generateCreateTableDdl(BusinessObjectDataDdlRequest request, BusinessObjectFormatEntity businessObjectFormatEntity, CustomDdlEntity customDdlEntity, BusinessObjectDataStatusEntity businessObjectDataStatusEntity, List<String> storageNames, List<StorageEntity> requestedStorageEntities, Map<String,StorageEntity> cachedStorageEntities, Map<String,String> cachedS3BucketNames) Generates the create table Hive 13 DDL as per specified business object data DDL request.
String	generateCreateTableDdl(BusinessObjectFormatDdlRequest request, BusinessObjectFormatEntity businessObjectFormatEntity, CustomDdlEntity customDdlEntity) Generates the create table Hive 13 DDL as per specified business object format DDL request.
String	generateReplaceColumnsStatement(BusinessObjectFormatDdlRequest request, BusinessObjectFormatEntity businessObjectFormatEntity)
String	getDdlCharacterValue(String string) Gets the DDL character value based on the specified configured character value.
String	getDdlCharacterValue(String string, boolean escapeSingleBackslash) Gets the DDL character value based on the specified configured character value.
BusinessObjectDataDdlOutputFormatEnum	getDdlOutputFormat() Gets the supported DDL output format that this generator can support.
List<HivePartitionDto>	getHivePartitions(BusinessObjectDataKey businessObjectDataKey, List<SchemaColumn> autoDiscoverableSubPartitionColumns, String s3KeyPrefix, Collection<String> storageFiles, String storageName) Gets a list of Hive partitions.
Pattern	getHivePathPattern(List<SchemaColumn> partitionColumns) Gets a pattern to match Hive partition sub-directories.
String	getHivePathRegex(List<SchemaColumn> partitionColumns) Gets a regex to match Hive partition sub-directories.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NO_PARTITIONING_PARTITION_KEY

public static final String NO_PARTITIONING_PARTITION_KEY

The partition key value for business object data without partitioning.

See Also:

Constant Field Values

NO_PARTITIONING_PARTITION_VALUE

public static final String NO_PARTITIONING_PARTITION_VALUE

The partition value for business object data without partitioning.

See Also:

Constant Field Values

ORC_HIVE_FILE_FORMAT

public static final String ORC_HIVE_FILE_FORMAT

Hive file format for ORC files.

See Also:

Constant Field Values

PARQUET_HIVE_FILE_FORMAT

public static final String PARQUET_HIVE_FILE_FORMAT

Hive file format for PARQUET files.

See Also:

Constant Field Values

TEXT_HIVE_FILE_FORMAT

public static final String TEXT_HIVE_FILE_FORMAT

Hive file format for text files.

See Also:

Constant Field Values

JSON_HIVE_FILE_FORMAT

public static final String JSON_HIVE_FILE_FORMAT

Hive file format for JSON files.

See Also:

Constant Field Values

REGEX_S3_EMPTY_PARTITION

public static final String REGEX_S3_EMPTY_PARTITION

The regular expression that represents an empty partition in S3, this is because hadoop file system implements directory support in S3 by creating empty files with the "directoryname_$folder$" suffix.

See Also:

Constant Field Values

Constructor Detail

Hive13DdlGenerator

public Hive13DdlGenerator()

Method Detail

escapeSingleQuotes

public String escapeSingleQuotes(String string)

Escapes single quote characters, if not already escaped, with an extra backslash.

Parameters:

string - the input text

Returns:

the output text with all single quote characters escaped by an extra backslash

generateCreateTableDdl

public String generateCreateTableDdl(BusinessObjectDataDdlRequest request,
                                     BusinessObjectFormatEntity businessObjectFormatEntity,
                                     CustomDdlEntity customDdlEntity,
                                     BusinessObjectDataStatusEntity businessObjectDataStatusEntity,
                                     List<String> storageNames,
                                     List<StorageEntity> requestedStorageEntities,
                                     Map<String,StorageEntity> cachedStorageEntities,
                                     Map<String,String> cachedS3BucketNames)

Generates the create table Hive 13 DDL as per specified business object data DDL request.

Specified by:

generateCreateTableDdl in class DdlGenerator

Parameters:

request - the business object data DDL request

businessObjectFormatEntity - the business object format entity

customDdlEntity - the optional custom DDL entity

businessObjectDataStatusEntity - the business object data status for available business object data

storageNames - the list of storage names

requestedStorageEntities - the list of storage entities per storage names specified in the request

cachedStorageEntities - the map of storage names in upper case to the relative storage entities

cachedS3BucketNames - the map of storage names in upper case to the relative S3 bucket names

Returns:

the create table Hive DDL

generateCreateTableDdl

public String generateCreateTableDdl(BusinessObjectFormatDdlRequest request,
                                     BusinessObjectFormatEntity businessObjectFormatEntity,
                                     CustomDdlEntity customDdlEntity)

Generates the create table Hive 13 DDL as per specified business object format DDL request.

Specified by:

generateCreateTableDdl in class DdlGenerator

Parameters:

request - the business object format DDL request

businessObjectFormatEntity - the business object format entity

customDdlEntity - the optional custom DDL entity

Returns:

the create table Hive DDL

generateReplaceColumnsStatement

public String generateReplaceColumnsStatement(BusinessObjectFormatDdlRequest request,
                                              BusinessObjectFormatEntity businessObjectFormatEntity)

Specified by:

generateReplaceColumnsStatement in class DdlGenerator

getDdlCharacterValue

public String getDdlCharacterValue(String string)

Gets the DDL character value based on the specified configured character value. This method supports UTF-8 encoded strings and will "Hive" escape any non-ASCII printable characters using '\(value)'.

Parameters:

string - the configured character value.

Returns:

the DDL character value.

getDdlCharacterValue

public String getDdlCharacterValue(String string,
                                   boolean escapeSingleBackslash)

Gets the DDL character value based on the specified configured character value. This method supports UTF-8 encoded strings and will "Hive" escape any non-ASCII printable characters using '\(value)'.

Parameters:

string - the configured character value.

escapeSingleBackslash - specifies if we need to escape a single backslash character with an extra backslash

Returns:

the DDL character value.

getDdlOutputFormat

public BusinessObjectDataDdlOutputFormatEnum getDdlOutputFormat()

Description copied from class: DdlGenerator

Gets the supported DDL output format that this generator can support.

Specified by:

getDdlOutputFormat in class DdlGenerator

Returns:

BusinessObjectDataDdlOutputFormatEnum supported output format.

getHivePartitions

public List<HivePartitionDto> getHivePartitions(BusinessObjectDataKey businessObjectDataKey,
                                                List<SchemaColumn> autoDiscoverableSubPartitionColumns,
                                                String s3KeyPrefix,
                                                Collection<String> storageFiles,
                                                String storageName)

Gets a list of Hive partitions. For single level partitioning, no auto-discovery of sub-partitions (sub-directories) is needed - the business object data will be represented by a single Hive partition instance. For multiple level partitioning, this method performs an auto-discovery of all sub-partitions (sub-directories) and creates a Hive partition object instance for each partition.

Parameters:

businessObjectDataKey - the business object data key

autoDiscoverableSubPartitionColumns - the auto-discoverable sub-partition columns

s3KeyPrefix - the S3 key prefix

storageFiles - the storage files

storageName - the storage name

Returns:

the list of Hive partitions

getHivePathPattern

public Pattern getHivePathPattern(List<SchemaColumn> partitionColumns)

Gets a pattern to match Hive partition sub-directories.

Parameters:

partitionColumns - the list of partition columns

Returns:

the newly created pattern to match Hive partition sub-directories.

getHivePathRegex

public String getHivePathRegex(List<SchemaColumn> partitionColumns)

Gets a regex to match Hive partition sub-directories.

Parameters:

partitionColumns - the list of partition columns

Returns:

the newly created regex to match Hive partition sub-directories.