Skip to content
Snippets Groups Projects
Select Git revision
  • dev
  • sabine_177-endpoint-timeseries_merged-is-available
  • testing
  • swt-project-filtering
  • master default protected
  • cherry-pick-1d0d89dc
  • sabine_issue169-netcdf-output-format-has-been-added-to-data-endpoint
  • sabine_issue153-tool-for-automatic-ontology-creation
  • 114-api-pages-make-use-of-redoc
  • search_function_filters
  • 141-cleanup-repo
  • ontology
  • paging_test
  • sabine_#52_webdo3seAPI
  • roll_out
15 results

toardb_fastapi

  • Clone with SSH
  • Clone with HTTPS
  • schroeder5's avatar
    #154: Increase the length of the database character string used to store the request ID
    Sabine Schröder authored
    4748c15d
    History

    REST API to TOAR-II database using FastAPI

    documentation (pages)

    http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/

    installation guide

    installation of toardb_fastapi:
    After cloning the repository (git clone https://gitlab.jsc.fz-juelich.de/esde/toar-data/toardb_fastapi.git)
    it is recommended to create a virtual environment and install the requirements therein:

    python3 –m venv venv
    source venv/bin/activate
    pip install –r requirements.txt

    If needed: installation and setup of database:

    If you do not have access to a TOAR-II-like database, you may want to install a small test database.

    This repository holds a simple test database toardb_dump.sql (not yet fully filled with example data in all tables).
    Also some testing examples for the running application are provided within file production_tests.sh, as well as corresponding pytests.
    Running the tests and pytest are described after the installation guide (you need to do this first).

    If you already have a PostGIS database server running, just load the toardb_dump (see below)
    -- you still may need to install the "toar_controlled_vocabulary"-extension (also see below).

    If you need to install a PostGIS database server, please follow exactly the instructions on page
    https://gitlab.version.fz-juelich.de/esde/toar-data/toar-db/-/blob/master/INSTALL_POSTGIS.md

    You also need the extension toar_controlled_vocabulary to be installed within your database server.
    How to install, is described here: https://gitlab.version.fz-juelich.de/esde/toar-data/toardb_fastapi/-/issues/3#note_51339

    After the above instructions execute the following command from shell to load the database dump:

    psql -U toaradmin -d toardb -h localhost -f toardb_dump.sql

    Setting passwords for database access

    The following file holds the credentials for database access and may need some adaptions:
    toardb/utils/database.py
    To make sure, that this sensitive data is not uploaded to the remote server, use the following command to prevent git from tracking this file:
    git update-index --assume-unchanged toardb/utils/database.py

    Running pytest

    After download and installation (see above) of toardb_fastapi you can check, if everything is working via

    pytest

    (due to an unknown error the database cannot be teared down correctly -- still under investigation (very last test will throw an error for teardown of database if run as ensemble, if every module is pytested stand-alone this error will not show up)).

    Workaround

    for MOD in contacts data stationmeta timeseries variables
    do
       pytest toardb/$MOD
    done

    Hint
    If you want to run just a selection of pytests, you can select this test by pattern (with the option -k); for example:

    pytest toardb/contacts -k test_get_special_person

    This command will run the following tests from test_contacts.py (because the pattern matches their names):

    • test_get_special_person
    • test_get_special_person_out_of_index
    • test_get_special_person_by_name

    Running the application

    complete documentation of REST API

    REST API
    /controlled_vocabulary/
    /controlled_vocabulary/{name}
    /database_statistics/
    /database_statistics/{name}
    /variables/
    /variables/{name}
    /variables/id/{variable_id}
    /contacts/persons/
    /contacts/persons/id/{person_id}
    /contacts/persons/{name}
    /contacts/organisations/
    /contacts/organisations/id/{organisation_id}
    /contacts/organisations/{name}
    /contacts/
    /contacts/id/{contact_id}
    /contacts/orga_name/{name}
    /stationmeta/
    /stationmeta/{station_code}
    /stationmeta/id/{station_id}
    /stationmeta_changelog/{station_id}
    /stationmeta/delete_field/{station_code}
    /timeseries/
    /timeseries/{timeseries_id}
    /timeseries/id/{timeseries_id}/timeseries/id/{timeseries_id}
    /timeseries/unique/
    /timeseries_changelog/{timeseries_id}
    /timeseries/delete_field/{timeseries_id}
    /data/
    /data/{timeseries_id
    /data/id/{timeseries_id}
    /data/record/

    After download and installation (see above) of toardb_fastapi you can check, if everything is working in the live system via the following:
    From the top directory of the project (toardb_fastapi) open two terminals.
    In the first terminal start the application via:

    source venv/bin/activate
    uvicorn toardb.toardb:app --reload

    In the second terminal start the live tests via:

    ./production_tests.sh

    or open a browser window (or use curl/wget) with the following URLs (some examples -- complete REST API see above):

    get variable information:

    http://127.0.0.1:8000/variables/
    http://127.0.0.1:8000/variables/o3

    get stationmeta information:

    http://127.0.0.1:8000/stationmeta"
    http://127.0.0.1:8000/stationmeta/China11"

    get 4 arbitrary rows of table data:

    http://127.0.0.1:8000/data/?limit=4  

    get all data of timeseries with id=2:

    http://127.0.0.1:8000/data/2

    Since POST commands need header information, the complete curl command is given here:
    upload a file to the table data of the TOAR database:

    1. file in current directory:
    curl -X POST -H 'Content-Type: multipart/form-data; charset=utf-8; boundary=__X_PAW_BOUNDARY__' -F "file=@o3_CO002_2012_2017_v1-0.dat" "http://127.0.0.1:8000/data/"
    1. file with absolute path:
    curl -X POST -H 'Content-Type: multipart/form-data; charset=utf-8; boundary=__X_PAW_BOUNDARY__' -F "file=@/home/sschroeder/fastAPIproject/upload_tests/o3_CO002_2012_2017_v1-0.dat" "http://127.0.0.1:8000/data/"