esde issueshttps://gitlab.jsc.fz-juelich.de/groups/esde/-/issues2024-03-25T08:19:47+01:00https://gitlab.jsc.fz-juelich.de/esde/toar-data/toar-ii-analysis-service/-/issues/13request does not work properly for JSON files2024-03-25T08:19:47+01:00Till Hauerrequest does not work properly for JSON filesNormally, users should specify csv as the format when requesting quantile regression, but the standard format is JSON. If the user doesn't specify a format, the first line will always contain a comment that begins with a hashtag. However...Normally, users should specify csv as the format when requesting quantile regression, but the standard format is JSON. If the user doesn't specify a format, the first line will always contain a comment that begins with a hashtag. However, there is no way to write comments in JSON and when the user tries to read the JSON file, he gets an error messagehttps://gitlab.jsc.fz-juelich.de/esde/toar-data/toar-data-landing-page/-/issues/6carsten_idea_#6_crationOfDDDAlandingPage2024-03-21T14:15:18+01:00Carsten Hinzcarsten_idea_#6_crationOfDDDAlandingPageAs part of the discussion for an overhaul of the TOAR website a new design with Dashboard, Download, Documentation, Analysis came up.
This is a first suggestion to modify the existing landing page.As part of the discussion for an overhaul of the TOAR website a new design with Dashboard, Download, Documentation, Analysis came up.
This is a first suggestion to modify the existing landing page.Carsten HinzCarsten Hinzhttps://gitlab.jsc.fz-juelich.de/esde/toar-data/toar-data-landing-page/-/issues/5carsten_idea_#5_restructureTOC_onLandingPage2024-03-21T14:11:00+01:00Carsten Hinzcarsten_idea_#5_restructureTOC_onLandingPageRestructure landing page to improve visibility of main parts for TOAR v2Restructure landing page to improve visibility of main parts for TOAR v2Carsten HinzCarsten Hinzhttps://gitlab.jsc.fz-juelich.de/esde/machine-learning/downscaling_maelstrom/-/issues/96Enabling generator pretraining for WGAN2024-03-11T16:54:55+01:00Michael LangguthEnabling generator pretraining for WGANTBDTBDhttps://gitlab.jsc.fz-juelich.de/esde/machine-learning/downscaling_maelstrom/-/issues/95Implementing Deep500 for D2.62024-03-09T13:48:21+01:00Michael LangguthImplementing Deep500 for D2.6In scope of the MAELSTROM project, the WGAN will be benchmarked again with Deep500 similar to #53.
A guideline for the API integration is provided on [github](https://github.com/sashkboos/Deep500-for-MAELSTROM).In scope of the MAELSTROM project, the WGAN will be benchmarked again with Deep500 similar to #53.
A guideline for the API integration is provided on [github](https://github.com/sashkboos/Deep500-for-MAELSTROM).https://gitlab.jsc.fz-juelich.de/esde/machine-learning/downscaling_maelstrom/-/issues/94Bug with built-in Keras loss in WGAN2024-03-09T13:33:19+01:00Michael LangguthBug with built-in Keras loss in WGANWhen tyring to use Keras' built-in losses for the reconstruction loss in the WGAN model (e.g.. setting `recon_loss = "mae"` in `config_sha_wgan.json`), the training breaks when checkpointing the model. In particular, the monitored recons...When tyring to use Keras' built-in losses for the reconstruction loss in the WGAN model (e.g.. setting `recon_loss = "mae"` in `config_sha_wgan.json`), the training breaks when checkpointing the model. In particular, the monitored reconstruction loss `val_recon_loss` looks as expected, but when checkpointing, a tensor is retrieved instead of a loss-value. \
An example logfile is attached [here](/uploads/bc222c4af41b0590edd29d298dc9bed0/train_sha_wgan_euradim_nox-out.9462963).https://gitlab.jsc.fz-juelich.de/esde/machine-learning/downscaling_maelstrom/-/issues/93Merge recent branches and clean-up repo2024-03-09T13:38:41+01:00Michael LangguthMerge recent branches and clean-up repoIn this branch, recent branches will be merged und the code in the repository will be largely cleand up. The aim is to come up with a `develop`-branch version that comprises the functionality of recent advances. This comprises benchmarki...In this branch, recent branches will be merged und the code in the repository will be largely cleand up. The aim is to come up with a `develop`-branch version that comprises the functionality of recent advances. This comprises benchmarking with JUBE, downloading ERA5-data from MARS, modular model set-up from MLAir and preprocessing of EURAD-IM data.
The following branches will be included:
- preprocessing EURAD-IM data: #65
- downloading ERA5 source data: #89
- modular model set-up from MLAir: #90
- benchmarking with JUBE: #91https://gitlab.jsc.fz-juelich.de/esde/toar-data/toardb_fastapi/-/issues/149data-endpoint accepts fields argument2024-02-28T09:15:15+01:00Max Lensingdata-endpoint accepts fields argumentThe data-endpoint should accept the use of the fields argument.
This should make it possible to query only certain data of a timeseries, e.g. only datetime and value of a timeseriesThe data-endpoint should accept the use of the fields argument.
This should make it possible to query only certain data of a timeseries, e.g. only datetime and value of a timeserieshttps://gitlab.jsc.fz-juelich.de/esde/toar-data/toardb_fastapi/-/issues/148new script "determine_database_statistics" has been written2024-02-23T15:26:46+01:00Sabine Schröders.schroeder@fz-juelich.denew script "determine_database_statistics" has been writtenSince the PostgreSQL server on toar-data has been shut down,
the script for determing the database statistics (users, stations, time-series, and data records) has to be adapted,
because it relies on `psql`-commands.Since the PostgreSQL server on toar-data has been shut down,
the script for determing the database statistics (users, stations, time-series, and data records) has to be adapted,
because it relies on `psql`-commands.https://gitlab.jsc.fz-juelich.de/esde/toar-data/toar-ii-dashboard/-/issues/12got rid of Plot_graphs.py2024-02-05T16:26:50+01:00Sabine Schröders.schroeder@fz-juelich.degot rid of Plot_graphs.pyThere are two source code files for creating the plots (`Plot_graphs.py` and `toar_plotting.py`), but only one of them (`toar_plotting.py`) is used. This has previously led to confusion and some changes were made in the wrong file (`Plot...There are two source code files for creating the plots (`Plot_graphs.py` and `toar_plotting.py`), but only one of them (`toar_plotting.py`) is used. This has previously led to confusion and some changes were made in the wrong file (`Plot_graphs.py`).
It is necessary to check again which changes were made in `Plot_graphs.py` and whether these were completely transferred to `toar_plotting.py`.
Then `Plot_graphs.py` can be deleted.https://gitlab.jsc.fz-juelich.de/esde/destine/mlworkflowinterface/-/issues/3All Plots are also provided in German2024-01-09T11:08:35+01:00Sabine Schröders.schroeder@fz-juelich.deAll Plots are also provided in GermanSince the user interface is also available in German, the plots for German settings should also be created completely in German, i.e. with German axis labels etc.Since the user interface is also available in German, the plots for German settings should also be created completely in German, i.e. with German axis labels etc.https://gitlab.jsc.fz-juelich.de/esde/toar-data/toar-ii-analysis-service/-/issues/12UTC-conversion to local time or sun time2024-01-08T13:25:26+01:00Simon GrasseUTC-conversion to local time or sun timeCurrently the results of are returned in local time, however this becomes an issue when spatially aggregating over timeseries to get a global map, since timezones might break up areas into discontinous patches. Maybe Solar time should be...Currently the results of are returned in local time, however this becomes an issue when spatially aggregating over timeseries to get a global map, since timezones might break up areas into discontinous patches. Maybe Solar time should be used since the spatial shift in time is continous then. Thus results become less prone to computational outliers/errors due to the "jump" between timezones (espacially the international date border)https://gitlab.jsc.fz-juelich.de/esde/machine-learning/downscaling_maelstrom/-/issues/91Benchmarking deliverable 3.72024-03-09T13:38:42+01:00Michael LangguthBenchmarking deliverable 3.7Following up on the benchmarking deliverable 3.6 (cf. #61), this benchmarking deliverable will test several new hardware:
- JSC JURECA H100, A100, Mi250
- E4 Grace Hopper and NVIDIA A2
- IPU Graphcore (AP1,3,~~5~~)
- (E4 AMD + H100 if so...Following up on the benchmarking deliverable 3.6 (cf. #61), this benchmarking deliverable will test several new hardware:
- JSC JURECA H100, A100, Mi250
- E4 Grace Hopper and NVIDIA A2
- IPU Graphcore (AP1,3,~~5~~)
- (E4 AMD + H100 if someone wants to try and there's time)
See Mattia's e-mail from the 15th December, 12:08. <br><br>
Note that the IPU cannot be tested on Graphcore's IPUs since the used Keras extension do not support overwriting the train_step-method as required by the WGAN architecture.https://gitlab.jsc.fz-juelich.de/esde/destine/mlworkflowinterface/-/issues/2boundary box can be given to EURAD plotting method2024-01-03T15:20:03+01:00Sabine Schröders.schroeder@fz-juelich.deboundary box can be given to EURAD plotting methodAt the moment, the full map for the EURAD-IM output is plotted.
The areas of interest are nevertheless North Rhine-Westphalia and Berlin-Brandenburg.
Therefore, now a hardwired solution cuts the German data within the "eurad_vis.py"....At the moment, the full map for the EURAD-IM output is plotted.
The areas of interest are nevertheless North Rhine-Westphalia and Berlin-Brandenburg.
Therefore, now a hardwired solution cuts the German data within the "eurad_vis.py".
The data should be cut using the values of the CTM output files.
Furthermore, it should be possible to specify a bounding box and the cities to be plotted (via an input file) (so that you can also zoom to North Rhine-Westphalia and Berlin-Brandenburg within Germany).
Adjustments to the source code of eurad_vis.py to show only a map for Germany (hardwired and with a new file h09.csv, which includes coordinates for Düsseldorf and Berlin):
```
l.44
- if self.is_imported():
- self.grid = int(self.dataset.attrs["dx"])
+ # is_imported does not work as expected...
+ #if self.is_imported():
+ self.grid = int(self.dataset.attrs["dx"])
l.219
- if self.grid == 3 or self.grid == 9:
- # print("3")
+ if self.grid == 3:
self.add_cities_boolean = True
csv_file = "h03.csv"
+ if self.grid == 9:
+ self.add_cities_boolean = True
+ csv_file = "h09.csv"
l.288
- cenlat = self.dataset.cenlat
- cenlon = self.dataset.cenlon
+ cenlat = 51.0
+ cenlon = 10.5
- lat_values = self.dataset.lat.values
- lon_values = self.dataset.lon.values
+ lat_values = self.dataset.lat.values[150:250,230:305]
+ lon_values = self.dataset.lon.values[150:250,230:305]
l. 332
- area = self.dataset[s].isel(time=t, z=self.height)
+ area = self.dataset[s].isel(time=t, z=self.height)[150:250,230:305]
fig = plt.figure()
ax = plt.axes(projection=proj)
+ ax.set_extent([6.0, 15.0, 47.0, 55.0], crs=crs)
```https://gitlab.jsc.fz-juelich.de/esde/toar-data/toar-ii-dashboard/-/issues/10Show the Provider next to the variable2023-12-19T16:15:19+01:00Tom OhlmeyerShow the Provider next to the variableTom OhlmeyerTom Ohlmeyerhttps://gitlab.jsc.fz-juelich.de/esde/toar-data/toar-ii-dashboard/-/issues/9Reenable display/retrieval of OpenAQ data2023-12-19T14:27:32+01:00Simon GrasseReenable display/retrieval of OpenAQ dataProvide an option to reenable display of OpenAQ data when the default will be changed as described [here](https://gitlab.jsc.fz-juelich.de/esde/toar-data/toardb_fastapi/-/issues/146). This should be a checkbox that is unchecked by default.Provide an option to reenable display of OpenAQ data when the default will be changed as described [here](https://gitlab.jsc.fz-juelich.de/esde/toar-data/toardb_fastapi/-/issues/146). This should be a checkbox that is unchecked by default.https://gitlab.jsc.fz-juelich.de/esde/toar-data/toardb_fastapi/-/issues/146disable OpenAQ data by default2023-12-19T14:27:32+01:00Simon Grassedisable OpenAQ data by defaultOpenAQ data is often unreliable, but TOAR should provide high quality data. Therefor the default should be changed, such that OpenAQ data will only be included if explicitly stated.OpenAQ data is often unreliable, but TOAR should provide high quality data. Therefor the default should be changed, such that OpenAQ data will only be included if explicitly stated.https://gitlab.jsc.fz-juelich.de/esde/toar-data/do3se-web-service/-/issues/44final_handover2023-12-19T09:43:52+01:00Clara Betancourtfinal_handoverhttps://gitlab.jsc.fz-juelich.de/esde/toar-data/toar-ii-analysis-service/-/issues/10Cleanup zip results2023-12-18T09:44:25+01:00Simon GrasseCleanup zip resultsEach request for a given analysis result produces a separate status-url but also a separate zip-result. Currently these results will be stored indefinitely. Since they are only used to get data out of existing status URLs (which should b...Each request for a given analysis result produces a separate status-url but also a separate zip-result. Currently these results will be stored indefinitely. Since they are only used to get data out of existing status URLs (which should be usually retrieved soon after the results exists), this may unnecessarily take up storage space. Therefor a policy for storage and cleanup of these results should be decided and implemented.https://gitlab.jsc.fz-juelich.de/esde/machine-learning/downscaling_maelstrom/-/issues/90Integration of modular model-handling from MLAir2024-03-09T13:38:42+01:00Michael LangguthIntegration of modular model-handling from MLAirTo enable a modular model handling of Keras models, the model class developed in MLAir will be integrated.
More information on the model class can be obtained in the [repository](https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mla...To enable a modular model handling of Keras models, the model class developed in MLAir will be integrated.
More information on the model class can be obtained in the [repository](https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/blob/master/mlair/model_modules/GUIDE.md?ref_type=heads) or in the accompanying paper](https://doi.org/10.5194/gmd-14-1553-2021). <br>
With this issue, the DeepRU-architecture will also be introduced that has already been set-up in issue-branch #73.