README.md 8.91 KB
Newer Older
Sabine Schröder's avatar
Sabine Schröder committed
1
# REST API to TOAR-II database using FastAPI
Sabine Schröder's avatar
Sabine Schröder committed
2
3

[[_TOC_]]
4
 
5
6
## documentation (pages)

7
http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/
8

9
## installation guide
10

Sabine Schröder's avatar
Sabine Schröder committed
11
12
13
14
15
16
17
18
19
20
21
22
23
24
**installation of toardb_fastapi:**  
After cloning the repository (`git clone https://gitlab.version.fz-juelich.de/toar/toardb_fastapi.git`)  
it is recommended to create a virtual environment and install the requirements therein:  
```
python3 –m venv venv
source venv/bin/activate
pip install –r requirements.txt
```

**If needed: installation and setup of database:**  

If you do not have access to a TOAR-II-like database, you may want to install a small test database.  

This repository holds a simple test database *toardb_dump.sql* (not yet fully filled with example data in **all** tables).  
25
26
Also some testing examples for the running application are provided within file *production_tests.sh*, as well as corresponding pytests.  
Running the tests and pytest are described after the installation guide (you need to do this first).  
27

Sabine Schröder's avatar
Sabine Schröder committed
28
29
30
31
If you already have a PostGIS database server running, just load the toardb_dump (see below)  
-- you still may need to install the "toar_controlled_vocabulary"-extension (also see below).  

If you need to **install a PostGIS database server**, please follow exactly the instructions on page   
32
https://gitlab.version.fz-juelich.de/toar/toar-db/-/blob/master/INSTALL_POSTGIS.md  
Sabine Schröder's avatar
Sabine Schröder committed
33

Sabine Schröder's avatar
Sabine Schröder committed
34
35
36
37
You also need **the extension toar_controlled_vocabulary** to be installed within your database server.  
How to install, is described here: https://gitlab.version.fz-juelich.de/esde/toar-data/toardb_fastapi/-/issues/3#note_51339

After the above instructions execute the following command from shell **to load the database dump**:  
38
39
40
41
```
psql -U toaradmin -d toardb -h localhost -f toardb_dump.sql
```

Sabine Schröder's avatar
Sabine Schröder committed
42
43
44
45
46
47
48
## Setting passwords for database access

The following file holds the credentials for database access and may need some adaptions:  
**toardb/utils/database.py**  
To make sure, that this sensitive data is not uploaded to the remote server, use the following command to prevent git from tracking this file:  
`git update-index --assume-unchanged toardb/utils/database.py`

49
50
51
52
53
54
55
56
## Running pytest

After download **and installation (see above)** of toardb_fastapi you can check, if everything is working via
```
pytest
```
(due to an unknown error the database cannot be teared down correctly -- still under investigation (very last test will throw an error for teardown of database if run as ensemble, if every module is pytested stand-alone this error will not show up)).  

Sabine Schröder's avatar
Sabine Schröder committed
57
58
59
60
61
62
63
64
**Workaround**  
```
for MOD in contacts data stationmeta timeseries variables
do
   pytest toardb/$MOD
done
```

65
66
67
68
69
70
71
72
73
74
**Hint**  
If you want to run just a selection of pytests, you can select this test by pattern (with the option -k); for example:  
```
pytest toardb/contacts -k test_get_special_person
```
This command will run the following tests from test_contacts.py (because the pattern matches their names):  
- test_get_special_person  
- test_get_special_person_out_of_index  
- test_get_special_person_by_name  

Sabine Schröder's avatar
Sabine Schröder committed
75
76
77
78
79
## Running the application

[complete documentation of REST API](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#rest-api)

**REST API**  
Sabine Schröder's avatar
Sabine Schröder committed
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
[/controlled_vocabulary/](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#id1)  
[/controlled_vocabulary/{name}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#controlled-vocabulary-name)  
[/database_statistics/](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#database-statistics)  
[/database_statistics/{name}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#database-statistics-name)  
[/variables/](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#variables)  
[/variables/{name}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#variables-name)  
[/variables/id/{variable_id}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#variables-id-variable-id)  
[/contacts/persons/](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#contacts-persons)  
[/contacts/persons/id/{person_id}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#contacts-persons-id-person-id)  
[/contacts/persons/{name}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#contacts-persons-name)  
[/contacts/organisations/](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#contacts-organisations)  
[/contacts/organisations/id/{organisation_id}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#contacts-organisations-id-organisation-id)  
[/contacts/organisations/{name}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#contacts-organisations-name)  
[/contacts/](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#contacts)  
[/contacts/id/{contact_id}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#contacts-id-contact-id)  
[/contacts/orga_name/{name}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#contacts-orga-name-name)  
[/stationmeta/](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#id64)  
[/stationmeta/{station_code}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#stationmeta-station-code)  
[/stationmeta/id/{station_id}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#stationmeta-id-station-id)  
[/stationmeta_changelog/{station_id}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#stationmeta-changelog-station-id)  
[/stationmeta/delete_field/{station_code}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#stationmeta-delete-field-station-code)  
[/timeseries/](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#id85)  
[/timeseries/{timeseries_id}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#timeseries-timeseries-id)  
[/timeseries/id/{timeseries_id}/timeseries/id/{timeseries_id}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#timeseries-id-timeseries-id)  
[/timeseries/unique/](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#timeseries-unique)  
[/timeseries_changelog/{timeseries_id}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#timeseries-changelog-timeseries-id)  
[/timeseries/delete_field/{timeseries_id}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#timeseries-delete-field-timeseries-id)  
[/data/](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#id106)  
[/data/{timeseries_id](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#data-timeseries-id)  
[/data/id/{timeseries_id}](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#data-id-timeseries-id)  
[/data/record/](http://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#data-record)  
Sabine Schröder's avatar
Sabine Schröder committed
111

112
113
114
115
116
117
118
119
120
121
122
123
124

After download **and installation (see above)** of toardb_fastapi you can check, if everything is working in the live system via the following:  
From the top directory of the project (*toardb_fastapi*) open two terminals.  
In the first terminal start the application via:  
```
source venv/bin/activate
uvicorn toardb.toardb:app --reload
```
In the second terminal start the live tests via:
```
./production_tests.sh
```

Sabine Schröder's avatar
Sabine Schröder committed
125
or open a browser window (or use curl/wget) with the following URLs (some examples -- complete REST API see above):
126

Sabine Schröder's avatar
Sabine Schröder committed
127
get variable information:  
128
```
Sabine Schröder's avatar
Sabine Schröder committed
129
130
http://127.0.0.1:8000/variables/
http://127.0.0.1:8000/variables/o3
131
132
```

Sabine Schröder's avatar
Sabine Schröder committed
133
get stationmeta information:  
134
```
Sabine Schröder's avatar
Sabine Schröder committed
135
136
http://127.0.0.1:8000/stationmeta"
http://127.0.0.1:8000/stationmeta/China11"
137
```
138
139

get 4 arbitrary rows of table data:  
140
```
141
http://127.0.0.1:8000/data/?limit=4  
142
```
143
144

get all data of timeseries with id=2:  
145
```
146
http://127.0.0.1:8000/data/2
147
```
148

Sabine Schröder's avatar
Sabine Schröder committed
149
Since POST commands need header information, the complete curl command is given here:  
150
151
152
153
154
155
156
157
158
159
upload a file to the table data of the TOAR database: 
1) file in current directory:  
```
curl -X POST -H 'Content-Type: multipart/form-data; charset=utf-8; boundary=__X_PAW_BOUNDARY__' -F "file=@o3_CO002_2012_2017_v1-0.dat" "http://127.0.0.1:8000/data/"
```
2) file with absolute path:  
```
curl -X POST -H 'Content-Type: multipart/form-data; charset=utf-8; boundary=__X_PAW_BOUNDARY__' -F "file=@/home/sschroeder/fastAPIproject/upload_tests/o3_CO002_2012_2017_v1-0.dat" "http://127.0.0.1:8000/data/"
```