REST API to TOAR-II database using FastAPI

• REST API to TOAR-II database using FastAPI
  • documentation (pages)
  • installation guide
  • Setting passwords for database access
  • Running pytest
  • Running the application

documentation (pages)

http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/

installation guide

installation of toardb_fastapi:
After cloning the repository (git clone https://gitlab.jsc.fz-juelich.de/esde/toar-data/toardb_fastapi.git)
it is recommended to create a virtual environment and install the requirements therein:

python3 –m venv venv
source venv/bin/activate
pip install –r requirements.txt

If needed: installation and setup of database:

If you do not have access to a TOAR-II-like database, you may want to install a small test database.

This repository holds a simple test database toardb_dump.sql (not yet fully filled with example data in all tables).
Also some testing examples for the running application are provided within file production_tests.sh, as well as corresponding pytests.
Running the tests and pytest are described after the installation guide (you need to do this first).

If you already have a PostGIS database server running, just load the toardb_dump (see below)
-- you still may need to install the "toar_controlled_vocabulary"-extension (also see below).

If you need to install a PostGIS database server, please follow exactly the instructions on page
https://gitlab.version.fz-juelich.de/esde/toar-data/toar-db/-/blob/master/INSTALL_POSTGIS.md

You also need the extension toar_controlled_vocabulary to be installed within your database server.
How to install, is described here: https://gitlab.version.fz-juelich.de/esde/toar-data/toardb_fastapi/-/issues/3#note_51339

After the above instructions execute the following command from shell to load the database dump:

psql -U toaradmin -d toardb -h localhost -f toardb_dump.sql

Setting passwords for database access

The following file holds the credentials for database access and may need some adaptions:
toardb/utils/database.py
To make sure, that this sensitive data is not uploaded to the remote server, use the following command to prevent git from tracking this file:
git update-index --assume-unchanged toardb/utils/database.py

Running pytest

After download and installation (see above) of toardb_fastapi you can check, if everything is working via

pytest

(due to an unknown error the database cannot be teared down correctly -- still under investigation (very last test will throw an error for teardown of database if run as ensemble, if every module is pytested stand-alone this error will not show up)).

Workaround

for MOD in contacts data stationmeta timeseries variables
do
   pytest toardb/$MOD
done

Hint
If you want to run just a selection of pytests, you can select this test by pattern (with the option -k); for example:

pytest toardb/contacts -k test_get_special_person

This command will run the following tests from test_contacts.py (because the pattern matches their names):

• test_get_special_person
• test_get_special_person_out_of_index
• test_get_special_person_by_name

Running the application

complete documentation of REST API

REST API
/controlled_vocabulary/
/controlled_vocabulary/{name}
/database_statistics/
/database_statistics/{name}
/variables/
/variables/{name}
/variables/id/{variable_id}
/contacts/persons/
/contacts/persons/id/{person_id}
/contacts/persons/{name}
/contacts/organisations/
/contacts/organisations/id/{organisation_id}
/contacts/organisations/{name}
/contacts/
/contacts/id/{contact_id}
/contacts/orga_name/{name}
/stationmeta/
/stationmeta/{station_code}
/stationmeta/id/{station_id}
/stationmeta_changelog/{station_id}
/stationmeta/delete_field/{station_code}
/timeseries/
/timeseries/{timeseries_id}
/timeseries/id/{timeseries_id}/timeseries/id/{timeseries_id}
/timeseries/unique/
/timeseries_changelog/{timeseries_id}
/timeseries/delete_field/{timeseries_id}
/data/
/data/{timeseries_id
/data/id/{timeseries_id}
/data/record/

After download and installation (see above) of toardb_fastapi you can check, if everything is working in the live system via the following:
From the top directory of the project (toardb_fastapi) open two terminals.
In the first terminal start the application via:

source venv/bin/activate
uvicorn toardb.toardb:app --reload

In the second terminal start the live tests via:

./production_tests.sh

or open a browser window (or use curl/wget) with the following URLs (some examples -- complete REST API see above):

get variable information:

http://127.0.0.1:8000/variables/
http://127.0.0.1:8000/variables/o3

get stationmeta information:

http://127.0.0.1:8000/stationmeta"
http://127.0.0.1:8000/stationmeta/China11"

get 4 arbitrary rows of table data:

http://127.0.0.1:8000/data/?limit=4  

get all data of timeseries with id=2:

http://127.0.0.1:8000/data/2

Since POST commands need header information, the complete curl command is given here:
upload a file to the table data of the TOAR database:

1. file in current directory:

curl -X POST -H 'Content-Type: multipart/form-data; charset=utf-8; boundary=__X_PAW_BOUNDARY__' -F "file=@o3_CO002_2012_2017_v1-0.dat" "http://127.0.0.1:8000/data/"

2. file with absolute path:

curl -X POST -H 'Content-Type: multipart/form-data; charset=utf-8; boundary=__X_PAW_BOUNDARY__' -F "file=@/home/sschroeder/fastAPIproject/upload_tests/o3_CO002_2012_2017_v1-0.dat" "http://127.0.0.1:8000/data/"