org.finra.herd.dao.impl

Class EmrDaoImpl

java.lang.Object

org.finra.herd.dao.impl.EmrDaoImpl

All Implemented Interfaces:

EmrDao

@Repository
public class EmrDaoImpl
extends Object
implements EmrDao

The EMR DAO implementation.

Constructor Summary

Constructors

Constructor and Description
EmrDaoImpl()

Method Summary

Modifier and Type	Method and Description
List<String>	addEmrMasterSecurityGroups(String clusterId, List<String> securityGroups, AwsParamsDto awsParams) Add Security groups to the master node of EMR cluster.
String	addEmrStep(String clusterId, com.amazonaws.services.elasticmapreduce.model.StepConfig emrStepConfig, AwsParamsDto awsParamsDto) Add an EMR Step.
String	createEmrCluster(String clusterName, EmrClusterDefinition emrClusterDefinition, AwsParamsDto awsParams) Create the EMR cluster.
com.amazonaws.services.elasticmapreduce.model.ClusterSummary	getActiveEmrClusterByName(String clusterName, AwsParamsDto awsParams) Get an Active EMR cluster by the cluster name.
List<com.amazonaws.services.elasticmapreduce.model.Application>	getApplications(List<EmrClusterDefinitionApplication> emrClusterDefinitionApplications) Converts the given list of EmrClusterDefinitionApplication into a list of Application
com.amazonaws.services.elasticmapreduce.model.StepSummary	getClusterActiveStep(String clusterId, AwsParamsDto awsParamsDto) Gets the active step on the cluster if any.
com.amazonaws.services.elasticmapreduce.model.Step	getClusterStep(String clusterId, String stepId, AwsParamsDto awsParamsDto) Gets the step on the cluster.
protected List<com.amazonaws.services.elasticmapreduce.model.Configuration>	getConfigurations(List<EmrClusterDefinitionConfiguration> emrClusterDefinitionConfigurations) Converts the given list of EmrClusterDefinitionConfiguration into a list of Configuration.
protected List<com.amazonaws.services.elasticmapreduce.model.EbsBlockDeviceConfig>	getEbsBlockDeviceConfigs(List<EmrClusterDefinitionEbsBlockDeviceConfig> emrClusterDefinitionEbsBlockDeviceConfigs) Creates a list of EbsBlockDeviceConfig from a given list of EmrClusterDefinitionEbsBlockDeviceConfig.
protected com.amazonaws.services.elasticmapreduce.model.EbsConfiguration	getEbsConfiguration(EmrClusterDefinitionEbsConfiguration emrClusterDefinitionEbsConfiguration) Creates an instance of EbsConfiguration from a given instance of EmrClusterDefinitionEbsConfiguration.
com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient	getEmrClient(AwsParamsDto awsParamsDto) Create the EMR client with the given proxy and access key details.
com.amazonaws.services.elasticmapreduce.model.Cluster	getEmrClusterById(String clusterId, AwsParamsDto awsParams) Get EMR cluster by cluster Id.
String	getEmrClusterStatusById(String clusterId, AwsParamsDto awsParams) Get EMR cluster status by cluster Id.
com.amazonaws.services.elasticmapreduce.model.Instance	getEmrMasterInstance(String clusterId, AwsParamsDto awsParams) Gets the master instance of the EMR cluster.
protected List<com.amazonaws.services.elasticmapreduce.model.InstanceFleetConfig>	getInstanceFleets(List<EmrClusterDefinitionInstanceFleet> emrClusterDefinitionInstanceFleets) Creates an instance fleet configuration that describes the EC2 instances and instance configurations for clusters that use this feature.
protected com.amazonaws.services.elasticmapreduce.model.InstanceGroupConfig	getInstanceGroupConfig(com.amazonaws.services.elasticmapreduce.model.InstanceRoleType roleType, String instanceType, Integer instanceCount, BigDecimal bidPrice, EmrClusterDefinitionEbsConfiguration emrClusterDefinitionEbsConfiguration) Creates an instance group configuration.
protected List<com.amazonaws.services.elasticmapreduce.model.InstanceGroupConfig>	getInstanceGroupConfigs(InstanceDefinitions instanceDefinitions) Create the instance group configuration for MASTER/CORE/TASK nodes as per the input parameters.
protected List<com.amazonaws.services.elasticmapreduce.model.InstanceTypeConfig>	getInstanceTypeConfigs(List<EmrClusterDefinitionInstanceTypeConfig> emrClusterDefinitionInstanceTypeConfigs) Creates a list of InstanceTypeConfig from a given list of EmrClusterDefinitionInstanceTypeConfig.
protected com.amazonaws.services.elasticmapreduce.model.KerberosAttributes	getKerberosAttributes(EmrClusterDefinitionKerberosAttributes emrClusterDefinitionKerberosAttributes) Creates an instance of KerberosAttributes from a given instance of EmrClusterDefinitionKerberosAttributes.
protected com.amazonaws.services.elasticmapreduce.model.InstanceFleetProvisioningSpecifications	getLaunchSpecifications(EmrClusterDefinitionLaunchSpecifications emrClusterDefinitionLaunchSpecifications) Creates an instance of InstanceFleetProvisioningSpecifications from a given instance of EmrClusterDefinitionLaunchSpecifications.
com.amazonaws.services.elasticmapreduce.model.ListInstanceFleetsResult	getListInstanceFleetsResult(String clusterId, AwsParamsDto awsParams) Get the instance fleets
protected Map<String,String>	getMap(List<Parameter> parameters) Converts the given list of Parameter into a Map of String, String
protected com.amazonaws.services.elasticmapreduce.model.SpotProvisioningSpecification	getSpotSpecification(EmrClusterDefinitionSpotSpecification emrClusterDefinitionSpotSpecification) Creates an instance of SpotProvisioningSpecification from a given instance of EmrClusterDefinitionSpotSpecification.
protected com.amazonaws.services.elasticmapreduce.model.VolumeSpecification	getVolumeSpecification(EmrClusterDefinitionVolumeSpecification emrClusterDefinitionVolumeSpecification) Creates an instance of VolumeSpecification from a given instance of EmrClusterDefinitionVolumeSpecification.
void	terminateEmrCluster(String clusterId, boolean overrideTerminationProtection, AwsParamsDto awsParams) Terminates the EMR cluster.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EmrDaoImpl

public EmrDaoImpl()

Method Detail

addEmrMasterSecurityGroups

public List<String> addEmrMasterSecurityGroups(String clusterId,
                                               List<String> securityGroups,
                                               AwsParamsDto awsParams)
                                        throws Exception

Description copied from interface: EmrDao

Add Security groups to the master node of EMR cluster.

Specified by:

addEmrMasterSecurityGroups in interface EmrDao

Parameters:

clusterId - EMR cluster Id.

securityGroups - the security groups list.

awsParams - the proxy details.

Returns:

the security groups that were added.

Throws:

Exception

addEmrStep

public String addEmrStep(String clusterId,
                         com.amazonaws.services.elasticmapreduce.model.StepConfig emrStepConfig,
                         AwsParamsDto awsParamsDto)
                  throws Exception

Description copied from interface: EmrDao

Add an EMR Step. This method adds the step to EMR cluster based on the input.

Specified by:

addEmrStep in interface EmrDao

Parameters:

clusterId - EMR cluster ID.

emrStepConfig - the EMR step config to be added.

awsParamsDto - the proxy details.

There are four serializable objects supported currently. They are 1: ShellStep - For shell scripts 2: HiveStep - For hive scripts 3: HadoopJarStep - For Custom Map Reduce Jar files and 4: PigStep - For Pig scripts.

Returns:

the step id

Throws:

Exception

createEmrCluster

public String createEmrCluster(String clusterName,
                               EmrClusterDefinition emrClusterDefinition,
                               AwsParamsDto awsParams)

Description copied from interface: EmrDao

Create the EMR cluster.

Specified by:

createEmrCluster in interface EmrDao

Parameters:

clusterName - the cluster name value.

emrClusterDefinition - the EMR cluster definition that contains all the EMR parameters.

awsParams - AWS related parameters for access/secret keys and proxy details.

Returns:

the cluster Id.

getActiveEmrClusterByName

public com.amazonaws.services.elasticmapreduce.model.ClusterSummary getActiveEmrClusterByName(String clusterName,
                                                                                              AwsParamsDto awsParams)

Description copied from interface: EmrDao

Get an Active EMR cluster by the cluster name. Cluster only in following states are returned: ClusterState.BOOTSTRAPPING, ClusterState.RUNNING, ClusterState.STARTING, ClusterState.WAITING

Specified by:

getActiveEmrClusterByName in interface EmrDao

Parameters:

clusterName - the cluster name value.

awsParams - AWS related parameters for access/secret keys and proxy details.

Returns:

the ClusterSummary object.

getApplications

public List<com.amazonaws.services.elasticmapreduce.model.Application> getApplications(List<EmrClusterDefinitionApplication> emrClusterDefinitionApplications)

Converts the given list of EmrClusterDefinitionApplication into a list of Application

Parameters:

emrClusterDefinitionApplications - list of EmrClusterDefinitionApplication

Returns:

list Application

getClusterActiveStep

public com.amazonaws.services.elasticmapreduce.model.StepSummary getClusterActiveStep(String clusterId,
                                                                                      AwsParamsDto awsParamsDto)

Description copied from interface: EmrDao

Gets the active step on the cluster if any.

Specified by:

getClusterActiveStep in interface EmrDao

Parameters:

clusterId - the cluster id.

awsParamsDto - AWS related parameters for access/secret keys and proxy details.

Returns:

the step summary object.

getClusterStep

public com.amazonaws.services.elasticmapreduce.model.Step getClusterStep(String clusterId,
                                                                         String stepId,
                                                                         AwsParamsDto awsParamsDto)

Description copied from interface: EmrDao

Gets the step on the cluster.

Specified by:

getClusterStep in interface EmrDao

Parameters:

clusterId - the cluster id.

stepId - the step id to get details of.

awsParamsDto - AWS related parameters for access/secret keys and proxy details.

Returns:

the step object.

getEmrClient

public com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient getEmrClient(AwsParamsDto awsParamsDto)

Description copied from interface: EmrDao

Create the EMR client with the given proxy and access key details.

Specified by:

getEmrClient in interface EmrDao

Parameters:

awsParamsDto - AWS related parameters for access/secret keys and proxy details.

Returns:

the AmazonElasticMapReduceClient object.

getEmrClusterById

public com.amazonaws.services.elasticmapreduce.model.Cluster getEmrClusterById(String clusterId,
                                                                               AwsParamsDto awsParams)

Description copied from interface: EmrDao

Get EMR cluster by cluster Id.

Specified by:

getEmrClusterById in interface EmrDao

Parameters:

clusterId - the job Id returned by EMR for the cluster.

awsParams - AWS related parameters for access/secret keys and proxy details.

Returns:

the cluster status.

getEmrClusterStatusById

public String getEmrClusterStatusById(String clusterId,
                                      AwsParamsDto awsParams)

Description copied from interface: EmrDao

Get EMR cluster status by cluster Id.

Specified by:

getEmrClusterStatusById in interface EmrDao

Parameters:

clusterId - the job Id returned by EMR for the cluster.

awsParams - AWS related parameters for access/secret keys and proxy details.

Returns:

the cluster status.

getEmrMasterInstance

public com.amazonaws.services.elasticmapreduce.model.Instance getEmrMasterInstance(String clusterId,
                                                                                   AwsParamsDto awsParams)
                                                                            throws Exception

Description copied from interface: EmrDao

Gets the master instance of the EMR cluster.

Specified by:

getEmrMasterInstance in interface EmrDao

Parameters:

clusterId - EMR cluster id.

awsParams - the proxy details.

Returns:

the master instance of the cluster.

Throws:

Exception

getListInstanceFleetsResult

public com.amazonaws.services.elasticmapreduce.model.ListInstanceFleetsResult getListInstanceFleetsResult(String clusterId,
                                                                                                          AwsParamsDto awsParams)

Description copied from interface: EmrDao

Get the instance fleets

Specified by:

getListInstanceFleetsResult in interface EmrDao

Parameters:

clusterId - the cluster Id.

awsParams - AWS related parameters for access/secret keys and proxy details.

Returns:

list instance fleets result

terminateEmrCluster

public void terminateEmrCluster(String clusterId,
                                boolean overrideTerminationProtection,
                                AwsParamsDto awsParams)

Description copied from interface: EmrDao

Terminates the EMR cluster.

Specified by:

terminateEmrCluster in interface EmrDao

Parameters:

clusterId - the cluster Id.

awsParams - AWS related parameters for access/secret keys and proxy details.

getConfigurations

protected List<com.amazonaws.services.elasticmapreduce.model.Configuration> getConfigurations(List<EmrClusterDefinitionConfiguration> emrClusterDefinitionConfigurations)

Converts the given list of EmrClusterDefinitionConfiguration into a list of Configuration.

Parameters:

emrClusterDefinitionConfigurations - list of EmrClusterDefinitionConfiguration

Returns:

list of Configuration

getEbsBlockDeviceConfigs

protected List<com.amazonaws.services.elasticmapreduce.model.EbsBlockDeviceConfig> getEbsBlockDeviceConfigs(List<EmrClusterDefinitionEbsBlockDeviceConfig> emrClusterDefinitionEbsBlockDeviceConfigs)

Creates a list of EbsBlockDeviceConfig from a given list of EmrClusterDefinitionEbsBlockDeviceConfig.

Parameters:

emrClusterDefinitionEbsBlockDeviceConfigs - the list of EmrClusterDefinitionEbsBlockDeviceConfig

Returns:

the list of EbsBlockDeviceConfig

getEbsConfiguration

protected com.amazonaws.services.elasticmapreduce.model.EbsConfiguration getEbsConfiguration(EmrClusterDefinitionEbsConfiguration emrClusterDefinitionEbsConfiguration)

Creates an instance of EbsConfiguration from a given instance of EmrClusterDefinitionEbsConfiguration.

Parameters:

emrClusterDefinitionEbsConfiguration - the instance of EmrClusterDefinitionEbsConfiguration

Returns:

the instance of EbsConfiguration

getInstanceFleets

protected List<com.amazonaws.services.elasticmapreduce.model.InstanceFleetConfig> getInstanceFleets(List<EmrClusterDefinitionInstanceFleet> emrClusterDefinitionInstanceFleets)

Creates an instance fleet configuration that describes the EC2 instances and instance configurations for clusters that use this feature.

Parameters:

emrClusterDefinitionInstanceFleets - the list of instance fleet configurations from the EMR cluster definition

Returns:

the instance fleet configuration

getInstanceGroupConfig

protected com.amazonaws.services.elasticmapreduce.model.InstanceGroupConfig getInstanceGroupConfig(com.amazonaws.services.elasticmapreduce.model.InstanceRoleType roleType,
                                                                                                   String instanceType,
                                                                                                   Integer instanceCount,
                                                                                                   BigDecimal bidPrice,
                                                                                                   EmrClusterDefinitionEbsConfiguration emrClusterDefinitionEbsConfiguration)

Creates an instance group configuration.

Parameters:

roleType - role type for the instance group (MASTER/CORE/TASK)

instanceType - EC2 instance type for the instance group

instanceCount - number of instances for the instance group

bidPrice - bid price in case of SPOT instance request

emrClusterDefinitionEbsConfiguration - the instance of EmrClusterDefinitionEbsConfiguration that contains EBS configurations that will be attached to each EC2 instance in this instance group

Returns:

the instance group config object

getInstanceGroupConfigs

protected List<com.amazonaws.services.elasticmapreduce.model.InstanceGroupConfig> getInstanceGroupConfigs(InstanceDefinitions instanceDefinitions)

Create the instance group configuration for MASTER/CORE/TASK nodes as per the input parameters.

Parameters:

instanceDefinitions - the instance group definitions from the EMR cluster definition

Returns:

the instance group config list with all the instance group definitions

getInstanceTypeConfigs

protected List<com.amazonaws.services.elasticmapreduce.model.InstanceTypeConfig> getInstanceTypeConfigs(List<EmrClusterDefinitionInstanceTypeConfig> emrClusterDefinitionInstanceTypeConfigs)

Creates a list of InstanceTypeConfig from a given list of EmrClusterDefinitionInstanceTypeConfig.

Parameters:

emrClusterDefinitionInstanceTypeConfigs - the list of EmrClusterDefinitionInstanceTypeConfig

Returns:

the list of InstanceTypeConfig

getKerberosAttributes

protected com.amazonaws.services.elasticmapreduce.model.KerberosAttributes getKerberosAttributes(EmrClusterDefinitionKerberosAttributes emrClusterDefinitionKerberosAttributes)

Creates an instance of KerberosAttributes from a given instance of EmrClusterDefinitionKerberosAttributes.

Parameters:

emrClusterDefinitionKerberosAttributes - the instance of EmrClusterDefinitionKerberosAttributes, may be null

Returns:

the instance of KerberosAttributes

getLaunchSpecifications

protected com.amazonaws.services.elasticmapreduce.model.InstanceFleetProvisioningSpecifications getLaunchSpecifications(EmrClusterDefinitionLaunchSpecifications emrClusterDefinitionLaunchSpecifications)

Creates an instance of InstanceFleetProvisioningSpecifications from a given instance of EmrClusterDefinitionLaunchSpecifications.

Parameters:

emrClusterDefinitionLaunchSpecifications - the instance of EmrClusterDefinitionLaunchSpecifications

Returns:

the instance of InstanceFleetProvisioningSpecifications

getMap

protected Map<String,String> getMap(List<Parameter> parameters)

Converts the given list of Parameter into a Map of String, String

Parameters:

parameters - List of Parameter

Returns:

Map

getSpotSpecification

protected com.amazonaws.services.elasticmapreduce.model.SpotProvisioningSpecification getSpotSpecification(EmrClusterDefinitionSpotSpecification emrClusterDefinitionSpotSpecification)

Creates an instance of SpotProvisioningSpecification from a given instance of EmrClusterDefinitionSpotSpecification.

Parameters:

emrClusterDefinitionSpotSpecification - the instance of EmrClusterDefinitionSpotSpecification

Returns:

the instance of SpotProvisioningSpecification

getVolumeSpecification

protected com.amazonaws.services.elasticmapreduce.model.VolumeSpecification getVolumeSpecification(EmrClusterDefinitionVolumeSpecification emrClusterDefinitionVolumeSpecification)

Creates an instance of VolumeSpecification from a given instance of EmrClusterDefinitionVolumeSpecification.

Parameters:

emrClusterDefinitionVolumeSpecification - the instance of EmrClusterDefinitionVolumeSpecification

Returns:

the instance of VolumeSpecification