org.finra.herd.dao.helper

Class EmrHelper

java.lang.Object

org.finra.herd.dao.helper.AwsHelper

org.finra.herd.dao.helper.EmrHelper

@Component
public class EmrHelper
extends AwsHelper

A helper class that provides EMR functions.

Field Summary

Fields inherited from class org.finra.herd.dao.helper.AwsHelper

configurationHelper, herdStringHelper, retryPolicyFactory

Constructor Summary

Constructors

Constructor and Description
EmrHelper()

Method Summary

Modifier and Type	Method and Description
List<EmrClusterInstanceFleet>	buildEmrClusterInstanceFleetFromAwsResult(com.amazonaws.services.elasticmapreduce.model.ListInstanceFleetsResult awsInstanceFleetsResult) Returns EmrClusterInstanceFleet list from AWS call
String	buildEmrClusterName(String namespaceCd, String emrDefinitionName, String clusterName) Returns EMR cluster name constructed according to the template defined.
String	getActiveEmrClusterId(String emrClusterId, String emrClusterName, String accountId) Gets the ID of an active EMR cluster which matches the given criteria.
AwsParamsDto	getAwsParamsDtoByAccountId(String accountId) Get the AWS Params DTO for the account Id if no account id is specified, use the default
protected List<EmrClusterInstanceTypeConfiguration>	getConfigurations(List<com.amazonaws.services.elasticmapreduce.model.Configuration> configurations) Returns list of EmrClusterInstanceTypeConfiguration
protected List<EmrClusterEbsBlockDevice>	getEbsBlockDevices(List<com.amazonaws.services.elasticmapreduce.model.EbsBlockDevice> ebsBlockDevices) Returns list of EmrClusterEbsBlockDevice
protected EmrClusterInstanceFleetStatus	getEmrClusterInstanceFleetStatus(com.amazonaws.services.elasticmapreduce.model.InstanceFleetStatus instanceFleetStatus) Returns EmrClusterInstanceFleetStatus
EmrDao	getEmrDao()
com.amazonaws.services.elasticmapreduce.model.StepConfig	getEmrHadoopJarStepConfig(String stepName, String jarLocation, String mainClass, List<String> scriptArguments, Boolean isContinueOnError) Builds the StepConfig for the Hadoop jar step.
protected List<EmrClusterInstanceTypeSpecification>	getInstanceTypeSpecifications(List<com.amazonaws.services.elasticmapreduce.model.InstanceTypeSpecification> awsInstanceTypeConfigs) Returns list of EmrClusterInstanceTypeSpecification
protected EmrClusterInstanceFleetProvisioningSpecifications	getLaunchSpecifications(com.amazonaws.services.elasticmapreduce.model.InstanceFleetProvisioningSpecifications instanceFleetProvisioningSpecifications) Returns EmrClusterInstanceFleetProvisioningSpecifications
protected List<Parameter>	getParameterList(Map<String,String> properties) Returns parameter list
String	getS3StagingLocation() Get the S3_STAGING_RESOURCE full path from the bucket name as well as other details.
protected EmrClusterSpotProvisioningSpecification	getSpotSpecification(com.amazonaws.services.elasticmapreduce.model.SpotProvisioningSpecification spotProvisioningSpecification) Returns EmrClusterSpotProvisioningSpecification from AWS call
protected EmrClusterVolumeSpecification	getVolumeSpecification(com.amazonaws.services.elasticmapreduce.model.VolumeSpecification volumeSpecification) Returns EmrClusterVolumeSpecification
boolean	isActiveEmrState(String status) Returns true if the supplied EMR status is considered to be active.
boolean	isInstanceDefinitionsEmpty(InstanceDefinitions instanceDefinitions) Returns true if the supplied InstanceDefinitions is null or empty (contains no elements).
void	setEmrDao(EmrDao emrDao)

Methods inherited from class org.finra.herd.dao.helper.AwsHelper

getAwsParamsDto, getClientConfiguration, getTransferRateInKilobytesPerSecond, getTransferRateInMegabitsPerSecond

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EmrHelper

public EmrHelper()

Method Detail

buildEmrClusterName

public String buildEmrClusterName(String namespaceCd,
                                  String emrDefinitionName,
                                  String clusterName)

Returns EMR cluster name constructed according to the template defined.

Parameters:

namespaceCd - the namespace code value.

emrDefinitionName - the EMR definition name value.

clusterName - the cluster name value.

Returns:

the cluster name.

getActiveEmrClusterId

public String getActiveEmrClusterId(String emrClusterId,
                                    String emrClusterName,
                                    String accountId)

Gets the ID of an active EMR cluster which matches the given criteria. If both cluster ID and cluster name is specified, the name of the actual cluster with the given ID must match the specified name. For cases where the cluster is not found (does not exists or not active), the method fails. All parameters are case-insensitive and whitespace trimmed. Blank parameters are equal to null.

Parameters:

emrClusterId - EMR cluster ID

emrClusterName - EMR cluster name

accountId - the account Id that EMR cluster is running under

Returns:

The cluster ID

getAwsParamsDtoByAccountId

public AwsParamsDto getAwsParamsDtoByAccountId(String accountId)

Get the AWS Params DTO for the account Id if no account id is specified, use the default

Parameters:

accountId - account Id

Returns:

AwsParamsDto

getEmrDao

public EmrDao getEmrDao()

setEmrDao

public void setEmrDao(EmrDao emrDao)

getEmrHadoopJarStepConfig

public com.amazonaws.services.elasticmapreduce.model.StepConfig getEmrHadoopJarStepConfig(String stepName,
                                                                                          String jarLocation,
                                                                                          String mainClass,
                                                                                          List<String> scriptArguments,
                                                                                          Boolean isContinueOnError)

Builds the StepConfig for the Hadoop jar step.

Parameters:

stepName - the step name.

jarLocation - the location of jar.

mainClass - the main class.

scriptArguments - the arguments.

isContinueOnError - indicate what to do on error.

Returns:

the stepConfig.

getS3StagingLocation

public String getS3StagingLocation()

Get the S3_STAGING_RESOURCE full path from the bucket name as well as other details.

Returns:

the s3 managed location.

isActiveEmrState

public boolean isActiveEmrState(String status)

Returns true if the supplied EMR status is considered to be active.

Parameters:

status - the EMR status

Returns:

whether the given EMR status is active

isInstanceDefinitionsEmpty

public boolean isInstanceDefinitionsEmpty(InstanceDefinitions instanceDefinitions)

Returns true if the supplied InstanceDefinitions is null or empty (contains no elements).

Parameters:

instanceDefinitions - the instance group definitions from the EMR cluster definition

Returns:

whether the given InstanceDefinitions is empty

buildEmrClusterInstanceFleetFromAwsResult

public List<EmrClusterInstanceFleet> buildEmrClusterInstanceFleetFromAwsResult(com.amazonaws.services.elasticmapreduce.model.ListInstanceFleetsResult awsInstanceFleetsResult)

Returns EmrClusterInstanceFleet list from AWS call

Parameters:

awsInstanceFleetsResult - AWS Instance Fleets result

Returns:

list of EmrClusterInstanceFleet

getEmrClusterInstanceFleetStatus

protected EmrClusterInstanceFleetStatus getEmrClusterInstanceFleetStatus(com.amazonaws.services.elasticmapreduce.model.InstanceFleetStatus instanceFleetStatus)

Returns EmrClusterInstanceFleetStatus

Parameters:

instanceFleetStatus - AWS object

Returns:

EmrClusterInstanceFleetStatus

getLaunchSpecifications

protected EmrClusterInstanceFleetProvisioningSpecifications getLaunchSpecifications(com.amazonaws.services.elasticmapreduce.model.InstanceFleetProvisioningSpecifications instanceFleetProvisioningSpecifications)

Returns EmrClusterInstanceFleetProvisioningSpecifications

Parameters:

instanceFleetProvisioningSpecifications - AWS object

Returns:

EmrClusterInstanceFleetProvisioningSpecifications

getSpotSpecification

protected EmrClusterSpotProvisioningSpecification getSpotSpecification(com.amazonaws.services.elasticmapreduce.model.SpotProvisioningSpecification spotProvisioningSpecification)

Returns EmrClusterSpotProvisioningSpecification from AWS call

Parameters:

spotProvisioningSpecification - AWS object

Returns:

EmrClusterSpotProvisioningSpecification

getEbsBlockDevices

protected List<EmrClusterEbsBlockDevice> getEbsBlockDevices(List<com.amazonaws.services.elasticmapreduce.model.EbsBlockDevice> ebsBlockDevices)

Returns list of EmrClusterEbsBlockDevice

Parameters:

ebsBlockDevices - AWS object

Returns:

list of EmrClusterEbsBlockDevice

getVolumeSpecification

protected EmrClusterVolumeSpecification getVolumeSpecification(com.amazonaws.services.elasticmapreduce.model.VolumeSpecification volumeSpecification)

Returns EmrClusterVolumeSpecification

Parameters:

volumeSpecification - AWS object

Returns:

EmrClusterVolumeSpecification

getInstanceTypeSpecifications

protected List<EmrClusterInstanceTypeSpecification> getInstanceTypeSpecifications(List<com.amazonaws.services.elasticmapreduce.model.InstanceTypeSpecification> awsInstanceTypeConfigs)

Returns list of EmrClusterInstanceTypeSpecification

Parameters:

awsInstanceTypeConfigs - AWS object

Returns:

list of EmrClusterInstanceTypeSpecification

getConfigurations

protected List<EmrClusterInstanceTypeConfiguration> getConfigurations(List<com.amazonaws.services.elasticmapreduce.model.Configuration> configurations)

Returns list of EmrClusterInstanceTypeConfiguration

Parameters:

configurations - AWS configuration object list

Returns:

list of EmrClusterInstanceTypeConfiguration

getParameterList

protected List<Parameter> getParameterList(Map<String,String> properties)

Returns parameter list

Parameters:

properties - properties

Returns:

list of parameters