TOAR Gridding Tool

About

The TOARgridding projects data from the TOAD database (https://toar-data.fz-juelich.de/) onto a grid. The request to the database also allows a statistical analysis of the requested value. The mean and standard deviation of all stations within a cell are computed.

The tool handles the request to the database over the REST API and the subsequent processing. The results of the gridding are provided as xarray objects for subsequent processing by the user.

This project is in beta with the intended basic functionalities. The documentation is work in progress.

Requirements

TBD, see pyproject.toml

Installation

Move to the folder you want to create download this project to. We now need to download the source code (https://gitlab.jsc.fz-juelich.de/esde/toar-public/toargridding/-/tree/dev?ref_type=heads). Either as ZIP folder or via git:

Download with GIT

Clone the project from its git repository:

git clone https://gitlab.jsc.fz-juelich.de/esde/toar-public/toargridding.git 

With git we need to checkout the development branch. Therefore we need to change to the project directory first:

cd toargridding
git checkout dev

Installing Dependencies

The handling of required packages is done with poetry. So run poetry in the project directory:

poetry install

Example

There are at the moment three example provided as jupyter notebooks (https://jupyter.org/).

Running them with the python environment produced by poetry can be done by

poetry run jupyter notebook

High level function

tests/produce_data.ipynb 

Provides an example on how to download data, apply gridding and save the results as netCDF files. A possible improvement for is the exchange of the AnalysisService with AnalysisServiceDownload, which caches requests from the TOARDB. This allows different griddings without the necessity to repeat the request to the TOARDB and subsequent download.

The example uses a dictionary to pass additional arguments to the request to the TAOR database. A detailed list can be found at https://toar-data.fz-juelich.de/api/v2/#stationmeta

Retrieving data

tests/get_sample_data.ipynb 

Downloads data from the TOAR database. The extracted data are written to disc. No further processing or gridding is done.

Retrieving data

tests/get_sample_data_manual.ipynb 

Downloads data from the TOAR database with a manual creation of the request to the TOAR database. This example does not perform any gridding.

Retriving data and visualization

tests/quality_controll.ipynb

Notebook for downloading and visualization of data. The data are downloaded and reused for subsequent executions of this notebook. The gridding is done on the downloaded data. Gridded data are not saved to disc.

Benchmarks Requests to TOAR Database

tests/benchmark.py

This script requests datasets with different durations (days to month) from the TOAR Database and saves them to disc. It reports the duration for the different requests. There is no gridding involved. CAVE: This script can run several hours.

Supported Grids

The first supported grid is a regular grid with longitude and latitude.

Supported Variables

At the moment only a limited number of variables from the TOAR database is supported.

Supported Time intervals

At the moment time differences larger than one day are working, i.e. start and end=start+1day leads to crashes.

Documentation of Source Code:

At the moment Carsten Hinz is working on a documentation of the source code, while getting familiar with it. The aim is a brief overview on the functionalities and the arguments. As he personally does not like repetitions, the documentations might not match other style guides. It will definitely be possible to extend the documentation:-)

class example:
    """An example class

    A more detailed explanation of the purpose of this example class.
    """

    def __init__(self, varA : int, varB : str):
        """Constructor

        Attributes:
        varA:
            brief details and more context
        varB:
            same here.
        """
        [implementation]    
    
    def func1(self, att1, att2):
        """Brief

        details

        Attributes:
        -----------
        att1:
            brief/details
        att2:
            brief/details
        """

        [implementation]
        

@dataclass
class dataClass:
    """Brief description

    optional details

    Parameters
    ----------
    anInt:
        brief description
    anStr:
        brief description
    secStr:
        brief description (explanation of default value, if this seems necessary)
    """
    anInt : int
    anStr : str
    secStr : str = "Default value"