esde issueshttps://gitlab.jsc.fz-juelich.de/groups/esde/-/issues2024-02-28T09:15:15+01:00https://gitlab.jsc.fz-juelich.de/esde/toar-data/toardb_fastapi/-/issues/149data-endpoint accepts fields argument2024-02-28T09:15:15+01:00Max Lensingdata-endpoint accepts fields argumentThe data-endpoint should accept the use of the fields argument.
This should make it possible to query only certain data of a timeseries, e.g. only datetime and value of a timeseriesThe data-endpoint should accept the use of the fields argument.
This should make it possible to query only certain data of a timeseries, e.g. only datetime and value of a timeserieshttps://gitlab.jsc.fz-juelich.de/esde/toar-data/toardb_fastapi/-/issues/148new script "determine_database_statistics" has been written2024-02-23T15:26:46+01:00Sabine Schröders.schroeder@fz-juelich.denew script "determine_database_statistics" has been writtenSince the PostgreSQL server on toar-data has been shut down,
the script for determing the database statistics (users, stations, time-series, and data records) has to be adapted,
because it relies on `psql`-commands.Since the PostgreSQL server on toar-data has been shut down,
the script for determing the database statistics (users, stations, time-series, and data records) has to be adapted,
because it relies on `psql`-commands.https://gitlab.jsc.fz-juelich.de/esde/toar-data/toar-ii-dashboard/-/issues/12got rid of Plot_graphs.py2024-02-05T16:26:50+01:00Sabine Schröders.schroeder@fz-juelich.degot rid of Plot_graphs.pyThere are two source code files for creating the plots (`Plot_graphs.py` and `toar_plotting.py`), but only one of them (`toar_plotting.py`) is used. This has previously led to confusion and some changes were made in the wrong file (`Plot...There are two source code files for creating the plots (`Plot_graphs.py` and `toar_plotting.py`), but only one of them (`toar_plotting.py`) is used. This has previously led to confusion and some changes were made in the wrong file (`Plot_graphs.py`).
It is necessary to check again which changes were made in `Plot_graphs.py` and whether these were completely transferred to `toar_plotting.py`.
Then `Plot_graphs.py` can be deleted.https://gitlab.jsc.fz-juelich.de/esde/destine/mlworkflowinterface/-/issues/3All Plots are also provided in German2024-01-09T11:08:35+01:00Sabine Schröders.schroeder@fz-juelich.deAll Plots are also provided in GermanSince the user interface is also available in German, the plots for German settings should also be created completely in German, i.e. with German axis labels etc.Since the user interface is also available in German, the plots for German settings should also be created completely in German, i.e. with German axis labels etc.https://gitlab.jsc.fz-juelich.de/esde/destine/mlworkflowinterface/-/issues/2boundary box can be given to EURAD plotting method2024-01-03T15:20:03+01:00Sabine Schröders.schroeder@fz-juelich.deboundary box can be given to EURAD plotting methodAt the moment, the full map for the EURAD-IM output is plotted.
The areas of interest are nevertheless North Rhine-Westphalia and Berlin-Brandenburg.
Therefore, now a hardwired solution cuts the German data within the "eurad_vis.py"....At the moment, the full map for the EURAD-IM output is plotted.
The areas of interest are nevertheless North Rhine-Westphalia and Berlin-Brandenburg.
Therefore, now a hardwired solution cuts the German data within the "eurad_vis.py".
The data should be cut using the values of the CTM output files.
Furthermore, it should be possible to specify a bounding box and the cities to be plotted (via an input file) (so that you can also zoom to North Rhine-Westphalia and Berlin-Brandenburg within Germany).
Adjustments to the source code of eurad_vis.py to show only a map for Germany (hardwired and with a new file h09.csv, which includes coordinates for Düsseldorf and Berlin):
```
l.44
- if self.is_imported():
- self.grid = int(self.dataset.attrs["dx"])
+ # is_imported does not work as expected...
+ #if self.is_imported():
+ self.grid = int(self.dataset.attrs["dx"])
l.219
- if self.grid == 3 or self.grid == 9:
- # print("3")
+ if self.grid == 3:
self.add_cities_boolean = True
csv_file = "h03.csv"
+ if self.grid == 9:
+ self.add_cities_boolean = True
+ csv_file = "h09.csv"
l.288
- cenlat = self.dataset.cenlat
- cenlon = self.dataset.cenlon
+ cenlat = 51.0
+ cenlon = 10.5
- lat_values = self.dataset.lat.values
- lon_values = self.dataset.lon.values
+ lat_values = self.dataset.lat.values[150:250,230:305]
+ lon_values = self.dataset.lon.values[150:250,230:305]
l. 332
- area = self.dataset[s].isel(time=t, z=self.height)
+ area = self.dataset[s].isel(time=t, z=self.height)[150:250,230:305]
fig = plt.figure()
ax = plt.axes(projection=proj)
+ ax.set_extent([6.0, 15.0, 47.0, 55.0], crs=crs)
```https://gitlab.jsc.fz-juelich.de/esde/toar-data/toardb_fastapi/-/issues/144search endpoint can return station fields *and* use pydantic2024-01-17T14:14:06+01:00Sabine Schröders.schroeder@fz-juelich.desearch endpoint can return station fields *and* use pydanticcommit 31649a2acedffc676d35ca717d647799203aa6ac shows, what needs to be done.commit 31649a2acedffc676d35ca717d647799203aa6ac shows, what needs to be done.Sabine Schröders.schroeder@fz-juelich.deSabine Schröders.schroeder@fz-juelich.dehttps://gitlab.jsc.fz-juelich.de/esde/toar-data/toardb_fastapi/-/issues/143auth_user (faked API for user authorization) was replaced by Helmholtz AAI2024-01-17T14:14:06+01:00Sabine Schröders.schroeder@fz-juelich.deauth_user (faked API for user authorization) was replaced by Helmholtz AAIAs long as no AAI existed, this was faked via the `auth_user` model.
Now, with the Helmholtz AAI integrated into the dashboard, the REST API can also use this.
Both, dashboard (GUI) and toardb_fastapi (REST API) work with the same da...As long as no AAI existed, this was faked via the `auth_user` model.
Now, with the Helmholtz AAI integrated into the dashboard, the REST API can also use this.
Both, dashboard (GUI) and toardb_fastapi (REST API) work with the same database of registered users.
Our own user_ids (stored alongside the connected e-mail address during TOAR phase 2) are used within changelog entries.
This ensures to keep provenance information within the TOAR-II database even if users might have deleted their consent within the Helmholtz AAI.Sabine Schröders.schroeder@fz-juelich.deSabine Schröders.schroeder@fz-juelich.dehttps://gitlab.jsc.fz-juelich.de/esde/toar-data/toar-ii-dashboard/-/issues/7marker size adjustment callback is now client-side2023-11-29T15:46:27+01:00Max Lensingmarker size adjustment callback is now client-sideUpdating the marker size to match the zoom level is currently a server-side callback.
To improve the speed of map interaction, this callback should be changed to a client-side callback.Updating the marker size to match the zoom level is currently a server-side callback.
To improve the speed of map interaction, this callback should be changed to a client-side callback.Till HauerTill Hauerhttps://gitlab.jsc.fz-juelich.de/esde/toar-data/toar-ii-dashboard/-/issues/5TOAR endpoint is stored in config file2023-08-22T11:57:46+02:00Max LensingTOAR endpoint is stored in config fileAt the moment the TOAR endpoint used is set in various files.
This leads to inconsistencies when changing the endpoint.
The aim is to create a central config file to set the TOAR endpoint, which will be used by all necessary files.At the moment the TOAR endpoint used is set in various files.
This leads to inconsistencies when changing the endpoint.
The aim is to create a central config file to set the TOAR endpoint, which will be used by all necessary files.Max LensingMax Lensinghttps://gitlab.jsc.fz-juelich.de/esde/toar-public/toarqc/-/issues/3SaQC package is used2023-07-13T13:57:56+02:00Niklas SelkeSaQC package is usedIt might be better to stop our own development efforts for a quality control tool and switch to the SaQC tool (https://www.hifis.net/spotlights/saqc).It might be better to stop our own development efforts for a quality control tool and switch to the SaQC tool (https://www.hifis.net/spotlights/saqc).https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/issues/455Skip harmonize history and target on demand2023-06-30T11:43:02+02:00Ghost UserSkip harmonize history and target on demandTo issue a real-time forecast, it is required to not harmonize history and target data as target data is not available at this time.
* [ ] add a parameter that stores unharmonized history data in separate variable `self.full_history`.
*...To issue a real-time forecast, it is required to not harmonize history and target data as target data is not available at this time.
* [ ] add a parameter that stores unharmonized history data in separate variable `self.full_history`.
* [ ] add method that forecasts also on full_history parameter and stores forecasts as `forecast_full.nc`https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/issues/446improve prediction speed2023-06-19T12:27:59+02:00Ghost Userimprove prediction speed
Improve prediction speed of DL model by indicating batch size in `.predict` call.
https://www.tensorflow.org/api_docs/python/tf/keras/Model#predict
```python
def make_prediction(self, subset):
"""
Create predictio...
Improve prediction speed of DL model by indicating batch size in `.predict` call.
https://www.tensorflow.org/api_docs/python/tf/keras/Model#predict
```python
def make_prediction(self, subset):
"""
Create predictions for NN, OLS, and persistence and add true observation as reference.
Predictions are filled in an array with full index range. Therefore, predictions can have missing values. All
predictions for a single station are stored locally under `<forecast/forecast_norm>_<station>_test.nc` and can
be found inside `forecast_path`.
"""
subset_type = subset.name
logging.info(f"start make_prediction for {subset_type}")
time_dimension = self.data_store.get("time_dim")
window_dim = self.data_store.get("window_dim")
for i, data in enumerate(subset):
input_data = data.get_X()
target_data = data.get_Y(as_numpy=False)
observation_data = data.get_observation()
# get scaling parameters
transformation_func = data.apply_transformation
nn_output = self.model.predict(input_data)
```https://gitlab.jsc.fz-juelich.de/esde/toar-data/toardb_fastapi/-/issues/138range parameters allowed for population_density2023-02-13T15:25:57+01:00Sabine Schröders.schroeder@fz-juelich.derange parameters allowed for population_densityjust implement a quick simple solution:
population_density=min_range,max_rangejust implement a quick simple solution:
population_density=min_range,max_rangeSabine Schröders.schroeder@fz-juelich.deSabine Schröders.schroeder@fz-juelich.dehttps://gitlab.jsc.fz-juelich.de/esde/toar-data/geopeas/-/issues/64services provide unified missing values2023-02-06T18:00:43+01:00Sabine Schröders.schroeder@fz-juelich.deservices provide unified missing valuesunify missing values:
-999.0 for float values
-1 for categorial values (like HTAP region)
## GEOPEAS:
(see views.py of related service)
* climatic_zones: default_value=0.
* ecoregions: default_value=-1.
* htap_regions_tier1: defau...unify missing values:
-999.0 for float values
-1 for categorial values (like HTAP region)
## GEOPEAS:
(see views.py of related service)
* climatic_zones: default_value=0.
* ecoregions: default_value=-1.
* htap_regions_tier1: default_value=-1
* landcover: default_value=0.
* major_roads: null
* nox_emissions: default_value=-999.
* population_density: (see population_number)
* population_number: default_value=-999.
* stable_nightlights: default_value=-99.
* topography_srtm: default_value=-9999.
GEOPEAS also provides out-of-bound values (f. ex. -888);
this is illustrated in
https://toar-data.fz-juelich.de/api/v2/stationmeta/id/7
GEOPEAS should provide the unified missing values.Niklas SelkeNiklas Selkehttps://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/issues/443Time Series Plot for all Competitors2023-02-06T10:15:48+01:00Ghost UserTime Series Plot for all Competitors* [ ] Add separate plot for each competitor
* [ ] also plot all models in single plot (with different shaded colors)* [ ] Add separate plot for each competitor
* [ ] also plot all models in single plot (with different shaded colors)https://gitlab.jsc.fz-juelich.de/esde/toar-data/geopeas/-/issues/63services calculate values even if coordinates exceed the boundaries of a glob...2023-01-14T09:41:52+01:00Sabine Schröders.schroeder@fz-juelich.deservices calculate values even if coordinates exceed the boundaries of a global mapThe problem is described using the service for the ECO region:
Some stations are not assigned an ECO region, even though those are defined globally at each point.
The query
http://zam10192.zam.kfa-juelich.de:7003/ecoregion/?descrip...The problem is described using the service for the ECO region:
Some stations are not assigned an ECO region, even though those are defined globally at each point.
The query
http://zam10192.zam.kfa-juelich.de:7003/ecoregion/?description=true&lat=-89.996&lon=24.8&radius=25000
returns:
```
The given range ('Lat', -90.015999999999, -89.976) is out of the range provided by server side. Please select values with: -89.901874828 < Lat < 83.628125172
```
Technically this is correct, but the earth is round :wink: , which is why the data does not stop at the left/right/top/bottom edge, but continues at the opposite/same edge.https://gitlab.jsc.fz-juelich.de/esde/toar-data/geopeas/-/issues/62service for IPCC regions has been added2022-12-15T14:50:38+01:00Sabine Schröders.schroeder@fz-juelich.deservice for IPCC regions has been addedAnother resource which could become useful for GEO-PEAS later:
https://github.com/IPCC-WG1/Atlas/tree/devel/reference-regions
(The TOAR SC is discussing a guideline document where also common region definitions are defined.
One op...Another resource which could become useful for GEO-PEAS later:
https://github.com/IPCC-WG1/Atlas/tree/devel/reference-regions
(The TOAR SC is discussing a guideline document where also common region definitions are defined.
One option could be to use the IPCC regions, which are posted here.)https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/issues/439other error metrics line plot2023-06-19T12:28:59+02:00Ghost Userother error metrics line plotCreate plot in similar manner to line plot version of `PlotTimeEvolutionMetric` but for other metrics than MSE. In particular, bias/mean error would be interesting.Create plot in similar manner to line plot version of `PlotTimeEvolutionMetric` but for other metrics than MSE. In particular, bias/mean error would be interesting.https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/issues/438error plot per month2023-06-19T12:28:55+02:00Ghost Usererror plot per monthCreate plot in similar manner to line plot version of `PlotTimeEvolutionMetric` that highlights error for each month summarized on all years in test set.Create plot in similar manner to line plot version of `PlotTimeEvolutionMetric` that highlights error for each month summarized on all years in test set.https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/issues/437combined plot of ME, MSE and R2023-06-19T12:28:50+02:00Ghost Usercombined plot of ME, MSE and RMaybe create another plot version that combines ME, MSE, and R in single graph. Can only display single value (e.g. overall average MSE, ...) and not entire distribution
![grafik](/uploads/a85c337feedc44353ea1b5d186e545c4/grafik.png)Maybe create another plot version that combines ME, MSE, and R in single graph. Can only display single value (e.g. overall average MSE, ...) and not entire distribution
![grafik](/uploads/a85c337feedc44353ea1b5d186e545c4/grafik.png)