Skip to content
Snippets Groups Projects
Select Git revision
  • main default protected
  • airflow-2.7.0 protected
  • airflow253 protected
  • air251
  • test_docker_op
  • airflow225
  • mptest
  • https-deployment
  • datacat_integration protected
  • datacatalog-integration
  • stable-2.2.2 protected
  • stable-2.2.1 protected
  • stable-2.2.0 protected
  • stable-2.1.4 protected
  • stable-2.1.3 protected
  • stable-2.1.2 protected
  • stable-2.1.1 protected
  • stable-2.1.0 protected
  • stable-2.0.2 protected
  • stable-2.0.1 protected
  • stable-2.0.0 protected
  • stable-1.0.1 protected
  • stable-1.0 protected
  • stable-0.1 protected
24 results

data-logistics-service

  • Clone with SSH
  • Clone with HTTPS
  • Data Logistics Service

    eFlows4HPC Data Logistics Service

    mkdir ./logs ./plugins
    echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env
    reqs=`cat requirements.txt | tr '\n' ' '`
    echo "_PIP_ADDITIONAL_REQUIREMENTS=$reqs" >> .env
    
    docker-compose -f dockers/docker-compose.yaml --project-directory . up airflow-init
    docker-compose -f dockers/docker-compose.yaml --project-directory . up -d

    Setup connection

    B2Share connection

    Here we use testing instance (check hostname)

    curl -X POST -u creds -H "Content-Type: application/json"  --data '{"connection_id": "default_b2share","conn_type":"https", "host": "b2share-testing.fz-juelich.de", "schema":""}' airflow:7001/api/v1/connections

    SSH

    Copy to target goes through scp (example with username/pass)

    curl -X POST -u creds -H "Content-Type: application/json"  --data '{"connection_id": "default_ssh", "conn_type": "ssh", "host": "ssh", "login": "user", "port": 2222, "password": "pass"}' airflow:7001/api/v1/connections

    Connections can also be added through env variables, like

    AIRFLOW_CONN_MY_PROD_DATABASE=my-conn-type://login:password@host:port/schema?param1=val1&param2=val2