-
Jedrzej Rybicki authoredJedrzej Rybicki authored
Requests documentation for YORCA integration
This document provides a list and structure of request to start a data transfer with DLS. The minimal workflow currently defined in the project assumes that the DLS part will be about moving data from a EUDAT service (B2SHARE) into a HPC through ssh.
Prerequisites
Host address with api e.g., export DLS=http://HOST:PORT/api/v1/
Also credentials USER:PASS (which are sent as http authentication).
A full description of the airflow API can be found in Documentation.
A connection to B2SHARE instance needs to be set-up. This can be done either in manual fashion or with following request (please note that we use testing instance of B2SHARE):
curl -X POST -u USER:PASS -H "Content-Type: application/json" \
--data '{"connection_id": "default_b2share","conn_type":"https", "host": "/b2share-testing.fz-juelich.de", "schema":"https"}' \
$DLS/connections
There should be an object created in B2SHARE, each object in B2SHARE is identified by a id, which needs to be passed to the DLS workflow as a parameter (see below).
Also a connection to the SSH target needs to be created (where the data will be copied to):
curl -X POST -u USER:PASS -H "Content-Type: application/json" \
--data '{"connection_id": "default_ssh", "conn_type": "ssh", "host": "SSH_HOST","login": "LOGIN", "port": PORT, "password": "PASSWORD"}' \
$DLS/connections
or for key-based access:
curl -X POST -u USER:PASS -H "Content-Type: application/json" --data '{"connection_id": "default_ssh", "conn_type": "ssh", "host": "SSH_HOST", "login": "LOGIN", "port": PORT, "extra": "{\"private_key\": \"-----BEGIN OPENSSH PRIVATE KEY-----\\nb3BlbnNzaC1rZXktdjgEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAlwAAAAdzc2gtcn\\nNhAAAAAwEAAQAAAIEAv1w/pNTlbh8Kvmu3+NMt5Kp7yT2FxplbPHi7sQEG80tPNNLb1oDa\\n1JaqKIN1Jq+U1895TTRr1nZBz6rKubssjdQ5H3AcO5ZbNRfiE4tGYbqHPAmpi4kTumQpy5\\nf0JkCHBNuK2LAEOV6pg1jukSmI34Z/HSoBpCCqmDPTQdT9Xq0AAAIIJKwpKCSsKSgAAAAH\\nc3NoLXJzYQAAAIEAv1w/pNTlbh8Kvmu3+NMt5Kp7yT2FxplbPHi7sQEG80tPNNLb1oDa1J\\naqKIN1Jq+U1895TTRr1nZBz6rKubssjdQ5H3AcO5ZbNRfiE4tGYbqHPAmpi4kTumQpy5f0\\nJkCHBNuK2LAEOV6pg1jukSmI34Z/HSoBpCCqmDPTQdT9Xq0AAAADAQABAAAAgEAFdu2IpL\\nGxBQEsPMKstH/6Yau0P5twF0wmAHV5qH+hRIChwxcCyTOyrH8dbAZjW+LP8P9ZeHKp4d6+\\nf1CgRIkhrKj2IYqXgIRUnbH3POBPzmxcEXSYrzc9zOriMhEEdsUraR0C20eFxShyVRHQRv\\nYjnvbYdcZjVnP09TLndZRpAAAAQAXtoENeyFzGxTpNlwqoggeeSvvXTIq8EiLFT8tdF2Lc\\nCXv/6VSDo53f3NmnC45sCNX3/vUq8Hqdu4SHm4y1EGEAAABBAPpNPhi2OvnN6wCiuRMbx1\\nD/nXdtI9LnPwwUmYcKZ+TDIx3mqpyZEJogIKA6gUlG1h1L1VUrtkr/e5XJGLP33ksAAABB\\nAMO3fvQIbOcNiVwKPu1nQG6KrS7y0Uf1O8sb+5kQMGBkJVcyLJTr3fYafOd7Sxo66OYv0b\\nQ649nEtohPPD75u+cAAAARcm9vdEBiY2JiZjhhZDdhMjQBAg==\\n-----END OPENSSH PRIVATE KEY-----\"}"}' \
$DLS/connections
Starting data transfer
To start a transfer following request needs to be sent it includes B2SHARE object id as a parameter. For testing purposes one can use b38609df2b334ea296ea1857e568dbea which includes one 100MB file. The target parameter is optional with default value /tmp/.
curl -X POST -u USER:PASS -H "Content-Type: application/json" \
--data '{"conf": {"oid": ID}, "target": PATH}' \
$DLS/dags/taskflow_example/dagRuns