Skip to content
Snippets Groups Projects
Commit 0a4035be authored by Jedrzej Rybicki's avatar Jedrzej Rybicki
Browse files

api requests for the upload

parent 69a5a2dd
Branches
Tags
No related merge requests found
Pipeline #92427 passed
== Requests documentation for YORCA integration == Requests documentation for YORC/TOSCA integration
This document provides a list and structure of request to start a data transfer with DLS. The minimal workflow currently defined in the project assumes that This document provides a list and structure of request to start a data transfer with DLS. The minimal workflow currently defined in the project assumes that
the DLS part will be about moving data from a EUDAT service (B2SHARE) into a HPC through +ssh+. the DLS part will be about moving data from a EUDAT service (B2SHARE) into a HPC through +ssh+.
...@@ -20,23 +20,30 @@ curl -X POST -u USER:PASS -H "Content-Type: application/json" \ ...@@ -20,23 +20,30 @@ curl -X POST -u USER:PASS -H "Content-Type: application/json" \
There should be an object created in B2SHARE, each object in B2SHARE is identified by a +id+, which needs to be passed to the DLS workflow as a parameter (see below). There should be an object created in B2SHARE, each object in B2SHARE is identified by a +id+, which needs to be passed to the DLS workflow as a parameter (see below).
Also a connection to the SSH target needs to be created (where the data will be copied to). You are free to use own conneciton name here, and pass it over to during the execution (defult is 'default_ssh')
---- === Credentials handling === [[credentials]]
curl -X POST -u USER:PASS -H "Content-Type: application/json" \ Credentials needed to access SSH-based storages should be passed over to the pipelines (both for up- and download) as pipelines parameters. Basically each DLS pipelines can be extended by tasks to handle those parameters and convert them into connections that can be used in the remainings of the pipeline. The code for that can be found in <a href='dags/conn_deco.py'>+dags/conn_deco.py+</a>.
--data '{"connection_id": "default_ssh", "conn_type": "ssh", "host": "SSH_HOST","login": "LOGIN", "port": PORT, "password": "PASSWORD"}' \
$DLS/connections ==== Passing credentials directly ====
---- Following parameters are expected:
or for key-based access:
---- ----
curl -X POST -u USER:PASS -H "Content-Type: application/json" --data '{"connection_id": "default_ssh", "conn_type": "ssh", "host": "SSH_HOST", "login": "LOGIN", "port": PORT, "extra": "{\"private_key\": \"-----BEGIN OPENSSH PRIVATE KEY-----\\nb3BlbnNzaC1rZXktdjgEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAlwAAAAdzc2gtcn\\nNhAAAAAwEAAQAAAIEAv1w/pNTlbh8Kvmu3+NMt5Kp7yT2FxplbPHi7sQEG80tPNNLb1oDa\\n1JaqKIN1Jq+U1895TTRr1nZBz6rKubssjdQ5H3AcO5ZbNRfiE4tGYbqHPAmpi4kTumQpy5\\nf0JkCHBNuK2LAEOV6pg1jukSmI34Z/HSoBpCCqmDPTQdT9Xq0AAAIIJKwpKCSsKSgAAAAH\\nc3NoLXJzYQAAAIEAv1w/pNTlbh8Kvmu3+NMt5Kp7yT2FxplbPHi7sQEG80tPNNLb1oDa1J\\naqKIN1Jq+U1895TTRr1nZBz6rKubssjdQ5H3AcO5ZbNRfiE4tGYbqHPAmpi4kTumQpy5f0\\nJkCHBNuK2LAEOV6pg1jukSmI34Z/HSoBpCCqmDPTQdT9Xq0AAAADAQABAAAAgEAFdu2IpL\\nGxBQEsPMKstH/6Yau0P5twF0wmAHV5qH+hRIChwxcCyTOyrH8dbAZjW+LP8P9ZeHKp4d6+\\nf1CgRIkhrKj2IYqXgIRUnbH3POBPzmxcEXSYrzc9zOriMhEEdsUraR0C20eFxShyVRHQRv\\nYjnvbYdcZjVnP09TLndZRpAAAAQAXtoENeyFzGxTpNlwqoggeeSvvXTIq8EiLFT8tdF2Lc\\nCXv/6VSDo53f3NmnC45sCNX3/vUq8Hqdu4SHm4y1EGEAAABBAPpNPhi2OvnN6wCiuRMbx1\\nD/nXdtI9LnPwwUmYcKZ+TDIx3mqpyZEJogIKA6gUlG1h1L1VUrtkr/e5XJGLP33ksAAABB\\nAMO3fvQIbOcNiVwKPu1nQG6KrS7y0Uf1O8sb+5kQMGBkJVcyLJTr3fYafOd7Sxo66OYv0b\\nQ649nEtohPPD75u+cAAAARcm9vdEBiY2JiZjhhZDdhMjQBAg==\\n-----END OPENSSH PRIVATE KEY-----\"}"}' \ host = params.get('host')
$DLS/connections port = params.get('port', 2222)
user = params.get('login')
key = params.get('key')
---- ----
Those will be used to create a temporary connection (with randomized name). The connection will be deleted after the pipline run.
==== Passing vault connection id ====
(not implemented yet)
=== Starting data transfer === === Starting data transfer ===
To start a transfer following request needs to be sent it includes B2SHARE object id as a parameter. For testing purposes one can use +b38609df2b334ea296ea1857e568dbea+ which To start a transfer following request needs to be sent it includes B2SHARE object id as a parameter. For testing purposes one can use +b38609df2b334ea296ea1857e568dbea+ which
includes one 100MB file. The target parameter is optional with default value +/tmp/+. includes one 100MB file. The target parameter is optional with default value +/tmp/+ and determines location to which the files from B2SHARE object will be uploaded.
The parameters should be passed along the credentials as described in <<credentials>>.
---- ----
curl -X POST -u USER:PASS -H "Content-Type: application/json" \ curl -X POST -u USER:PASS -H "Content-Type: application/json" \
...@@ -44,19 +51,60 @@ curl -X POST -u USER:PASS -H "Content-Type: application/json" \ ...@@ -44,19 +51,60 @@ curl -X POST -u USER:PASS -H "Content-Type: application/json" \
$DLS/dags/taskflow_example/dagRuns $DLS/dags/taskflow_example/dagRuns
---- ----
If you want to use your own connection ('myown_con'):
=== Checking status of a pipeline ===
---- ----
curl -X POST -u USER:PASS -H "Content-Type: application/json" \ curl -X GET -u USER:PASS -H "Content-Type: application/json" $DLS/dags/taskflow_example/dagRuns
--data '{"conf": {"oid": ID, "target": PATH, "connection": "myown_con"}}' \
$DLS/dags/taskflow_example/dagRuns
---- ----
=== Uploading example ===
To upload to B2SHARE use +dags/upload_example.py+ pipeline.
Comparing to the download case, for the upload the b2share connection must include the authenitcation token in extra field.
=== Checking status ===
---- ----
curl -X GET -u USER:PASS -H "Content-Type: application/json" $DLS/dags/taskflow_example/dagRuns {"access_token": "foo"}
---- ----
For information on how to generate access token, please refer to B2SHARE documenation: https://eudat.eu/services/userdoc/b2share-http-rest-api
Also connection to data catlog (with name +datacat+) needs to be defined in the DLS.
The upload target is defined by parameter +mid+ which points to a metadata template in datacatalog. The metadata template should comprise following information:
----
{
"name": "registration",
"url": "https://b2share-testing.fz-juelich.de/",
"metadata": {
"title":"Result of the computation",
"creator_name": "eflows4HPC",
"description": "Output registered by eflows4HPC DLS",
"community": "a9217684-945b-4436-8632-cac271f894ed",
"helmholtz centre": "Forschungszentrum Jülich",
"open_access": "True"
}
}
----
The values in the metadata field can be adjusted to the user needs. Once such a record is created in the data catalog, its
identifier can be used as the value for +mid+ parameter of the upload pipeline. To find out more about how to create records in data catalog check https://datacatalog.fz-juelich.de/docs.
An example upload template can be found at: https://datacatalog.fz-juelich.de/storage.html?type=storage_target&oid=71e863ac-aee6-4680-a57c-de318530b71e Thus the +mid+ = 71e863ac-aee6-4680-a57c-de318530b71e.
The upload piplenes requires input parameter +source+ which defines input directory. Similarly to the download case SSH credentials need to be passed to the execution (see <<credentials>> for details) (not included in the example).
----
curl -X POST -u USER:PASS -H "Content-Type: application/json" \
--data '{"conf": {"mid": MID, "source": PATH}}' \
$DLS/dags/upload_example/dagRuns
----
=== Comments === === Comments ===
I could image that a name of DLS pipeline (+taskflow_example+) can change and needs to be passed as parameter to YORC. I could image that a name of DLS pipeline (+taskflow_example+) can change and needs to be passed as parameter to YORC.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment