-
Jedrzej Rybicki authoredJedrzej Rybicki authored
Requests documentation for YORCA integration
This document provides a list and structure of request to start a data transfer with DLS. The minimal workflow currently defined in the project assumes that the DLS part will be about moving data from a EUDAT service (B2SHARE) into a HPC through ssh.
Prerequisites
Host address with api e.g., export DLS=http://HOST:PORT/api/v1/
Also credentials USER:PASS (which are sent as http authentication).
A full description of the airflow API can be found in Documentation.
A connection to B2SHARE instance needs to be set-up. This can be done either in manual fashion or with following request (please note that we use testing instance of B2SHARE):
curl -X POST -u USER:PASS -H "Content-Type: application/json" \
--data '{"connection_id": "default_b2share","conn_type":"https", "host": "/b2share-testing.fz-juelich.de", "schema":"https"}' \
$DLS/connections
There should be an object created in B2SHARE, each object in B2SHARE is identified by a id, which needs to be passed to the DLS workflow as a parameter (see below).
Also a connection to the SSH target needs to be created (where the data will be copied to). You are free to use own conneciton name here, and pass it over to during the execution (defult is 'default_ssh')
curl -X POST -u USER:PASS -H "Content-Type: application/json" \
--data '{"connection_id": "default_ssh", "conn_type": "ssh", "host": "SSH_HOST","login": "LOGIN", "port": PORT, "password": "PASSWORD"}' \
$DLS/connections
or for key-based access:
curl -X POST -u USER:PASS -H "Content-Type: application/json" --data '{"connection_id": "default_ssh", "conn_type": "ssh", "host": "SSH_HOST", "login": "LOGIN", "port": PORT, "extra": "{\"private_key\": \"-----BEGIN OPENSSH PRIVATE KEY-----\\nb3BlbnNzaC1rZXktdjgEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAlwAAAAdzc2gtcn\\nNhAAAAAwEAAQAAAIEAv1w/pNTlbh8Kvmu3+NMt5Kp7yT2FxplbPHi7sQEG80tPNNLb1oDa\\n1JaqKIN1Jq+U1895TTRr1nZBz6rKubssjdQ5H3AcO5ZbNRfiE4tGYbqHPAmpi4kTumQpy5\\nf0JkCHBNuK2LAEOV6pg1jukSmI34Z/HSoBpCCqmDPTQdT9Xq0AAAIIJKwpKCSsKSgAAAAH\\nc3NoLXJzYQAAAIEAv1w/pNTlbh8Kvmu3+NMt5Kp7yT2FxplbPHi7sQEG80tPNNLb1oDa1J\\naqKIN1Jq+U1895TTRr1nZBz6rKubssjdQ5H3AcO5ZbNRfiE4tGYbqHPAmpi4kTumQpy5f0\\nJkCHBNuK2LAEOV6pg1jukSmI34Z/HSoBpCCqmDPTQdT9Xq0AAAADAQABAAAAgEAFdu2IpL\\nGxBQEsPMKstH/6Yau0P5twF0wmAHV5qH+hRIChwxcCyTOyrH8dbAZjW+LP8P9ZeHKp4d6+\\nf1CgRIkhrKj2IYqXgIRUnbH3POBPzmxcEXSYrzc9zOriMhEEdsUraR0C20eFxShyVRHQRv\\nYjnvbYdcZjVnP09TLndZRpAAAAQAXtoENeyFzGxTpNlwqoggeeSvvXTIq8EiLFT8tdF2Lc\\nCXv/6VSDo53f3NmnC45sCNX3/vUq8Hqdu4SHm4y1EGEAAABBAPpNPhi2OvnN6wCiuRMbx1\\nD/nXdtI9LnPwwUmYcKZ+TDIx3mqpyZEJogIKA6gUlG1h1L1VUrtkr/e5XJGLP33ksAAABB\\nAMO3fvQIbOcNiVwKPu1nQG6KrS7y0Uf1O8sb+5kQMGBkJVcyLJTr3fYafOd7Sxo66OYv0b\\nQ649nEtohPPD75u+cAAAARcm9vdEBiY2JiZjhhZDdhMjQBAg==\\n-----END OPENSSH PRIVATE KEY-----\"}"}' \
$DLS/connections
Starting data transfer
To start a transfer following request needs to be sent it includes B2SHARE object id as a parameter. For testing purposes one can use b38609df2b334ea296ea1857e568dbea which includes one 100MB file. The target parameter is optional with default value /tmp/.
curl -X POST -u USER:PASS -H "Content-Type: application/json" \
--data '{"conf": {"oid": ID, "target": PATH}}' \
$DLS/dags/taskflow_example/dagRuns
If you want to use your own connection ('myown_con'):
curl -X POST -u USER:PASS -H "Content-Type: application/json" \
--data '{"conf": {"oid": ID, "target": PATH, "connection": "myown_con"}}' \
$DLS/dags/taskflow_example/dagRuns