Execute

Execute or exec in short is a generic task that supports executing a script defined locally or remotely.

Execute Task Definition

Following are the attributes of the task.

  • script: This denotes that a bash script would be executed.

  • path: Remote path of the script

  • value: The script content that will be executed.

  • target (Optional): The resource's alias where this script would be executed. If the path is specified, then the script location will be considered to be of the resource mentioned in the target attribute.

Either path or script needs to be specified. Supported resource targets are remote servers and EMR cluster.

Execute Script on EMR

The following launch configuration would add an EMR step in order to execute the script defined in the value attribute. To know more about EMR steps, refer the AWS document.

project: Demo
resources: /path/to/resources.yml
stages:
- exec_demo:
- exec:
script:
value: |
set -x
echo "Running the hadoop commands. This will take some time."
sleep 3m
echo "Successfully ran all the commands" > /home/hadoop/commands.out
target: playground-emr-cluster

Similarly, we can execute a script, that is located at a path say /opt/modules/scripts/exec_me.sh

project: Demo
resources: /path/to/resources.yml
stages:
- exec_demo:
- exec:
script:
path: /opt/modules/scripts/exec_me.sh
target: playground-emr-cluster

Corresponding resources configuration would be as below.

resources.yaml
- emr:
alias: test-cluster
keyFile: /path/to/key-pair.pem
clusterId: J-xxxxxxxxxxxx

Execute a script on Remotely

demo.flow.yml
project: Demo
resources: /path/to/resources.yml
stages:
- exec_demo:
- exec:
script:
path: /opt/modules/scripts/exec_me.sh
target: remote

Corresponding resource target remote would be defined as below.

resources.yml
resources:
- server:
alias: remote
host: xx.xx.xx.xxx
username: giga
keyFile: /path/to/keyFile.pem

Run the launch tasks

If the launch configuration demo.launch.yml is stored in the current working directory, then you can run the launch command as below.

$ teraflow demo

Else, you need to specify the path of the launch configuration.

$ teraflow -c /path/to/demo.flow.yml