Copy

Copy or cp in short, as the name suggests, allows users to copy files.

Copy task definition

Copy task, supports variety of resources. Following are the attributes of the cp task.

  1. from: The location of the file stored in the local system

  2. to: The path to the destination

  3. target (Optional): If not specified, it will just attempt to copy to same local system, else the specified target would be the alias name of the resource, defined in the resources.yml.

Format

- cp:
from: <local path of file>
to: <target path>
target: <alias of resource>

Copy to Remote server

demo.flow.yml
project: Demo
resources: /path/to/resources.yml
stages:
- cp:
from: /path/to/source/file
to: /path/to/destination/
target: remote

The corresponding resource definition for the remote server is as follows.

resources.yml
resources:
- server:
alias: remote
host: xx.xx.xx.xxx
username: giga
keyFile: /path/to/keyFile.pem

Copy to s3

Copy the file specified by the from attribute to the destination specified as to

demo.flow.yml
project: Demo
resources: /path/to/resources.yml
stages:
- cp:
from: /path/to/source/file
to: s3://bucket-name/key/path/to/destination/
target: s3-lab

Corresponding S3 resource definition example

resources.yml
resources:
- s3:
alias: remote
profile: data-sandbox

Copy to EMR master

Copy the file specified by the from attribute to the EMR master node. In order for this to work, make sure you've added the private key file while defining the EMR resource in resources.yml

demo.flow.yml
project: Demo
resources: /path/to/resources.yml
stages:
- cp_demo:
- cp:
from: /path/to/local/file
to: /path/in/master/node
target: playground-emr-cluster

Corresponding EMR resource definition

resources.yml
- emr:
alias: playground-emr-cluster
keyFile: /path/to/key-pair.pem
clusterId: J-xxxxxxxxxxxx

Run the launch tasks

If the launch configuration demo.launch.yml is stored in the current working directory, then you can run the launch command as below.

$ teraflow demo

Else, you need to specify the path of the launch configuration.

$ teraflow -c /path/to/demo.flow.yml