AWS

Launcher integrates with AWS resources, to execute the tasks.

Currently only S3 and Elastic Mapreduce ( EMR ) is supported.

S3 Resource Definition

Following are the attributes for a S3 resource.

  1. alias: An identifier that can be used to reference while defining the tasks

  2. profile (Optional): The aws IAM profile, that will be used to connect to S3. This is an optional attributed, and the default value is "default"

resources.yml
- s3:
alias: s3-live
profile: data-sandbox

Elastic MapReduce ( EMR ) Definition

Following are the attributes for an EMR resource.

  1. alias: An identifier that can be used to reference while defining the tasks.

  2. keyFile: The private key file that's associated to the key pair, which is used to login to the master node. For more details about EC2 key pair, refer the aws document.

  3. clusterId: This is the job flow id created, once the EMR is launched. Refer the AWS EMR document.

resources.yml
- emr:
alias: test-cluster
keyFile: /path/to/key-pair.pem
clusterId: J-xxxxxxxxxxxx

‚Äč

Once you're done, link the resources definition in the launch configuration.

demo.flow.yml
project: Demo
resources: /path/to/resources.yml