add license, updated readme and get-started for docu

LICENSE

+MIT License
+
+Copyright (c) 2020 Lukas Leufen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
\ No newline at end of file

README.md

@@ -5,45 +5,213 @@ learning (ML) models for the analysis and forecasting of meteorological and air
 
 # Installation
 
-* Install __proj__ on your machine using the console. E.g. for opensuse / leap `zypper install proj`
-* A c++ compiler is required for the installation of the program __cartopy__
-* Install all requirements from `requirements.txt` preferably in a virtual environment
-* Installation of MLAir:
-    * Either clone MLAir from its repository in gitlab (link??) and use it without installation 
+MLAir is based on several python frameworks. To work properly, you have to install all packages from the 
+`requirements.txt` file. Additionally to support the geographical plotting part it is required to install geo
+packages built for your operating system. Name names of these package may differ for different systems, we refer
+here to the opensuse / leap OS. The geo plot can be removed from the `plot_list`, in this case there is no need to 
+install the geo packages.
+
+* (geo) Install **proj** on your machine using the console. E.g. for opensuse / leap `zypper install proj`
+* (geo) A c++ compiler is required for the installation of the program **cartopy**
+* Install all requirements from [`requirements.txt`](https://gitlab.version.fz-juelich.de/toar/machinelearningtools/-/blob/master/requirements.txt)
+  preferably in a virtual environment
+* (tf) Currently, TensorFlow-1.13 is mentioned in the requirements. We already tested the TensorFlow-1.15 version and couldn't
+  find any compatibility errors. Please note, that tf-1.13 and 1.15 have two distinct branches each, the default branch 
+  for CPU support, and the "-gpu" branch for GPU support. If the GPU version is installed, MLAir will make use of the GPU
+  device.
+* Installation of **MLAir**:
+    * Either clone MLAir from the [gitlab repository](https://gitlab.version.fz-juelich.de/toar/machinelearningtools.git) 
+      and use it without installation (beside the requirements) 
     * or download the distribution file (?? .whl) and install it via `pip install <??>`. In this case, you can simply
       import MLAir in any python script inside your virtual environment using `import mlair`.
 
-## Special instructions for installation on Jülich HPC systems
+# How to start with MLAir
 
-_Please note, that the HPC setup is customised for JUWELS and HDFML. When using another HPC system, you can use the HPC 
-setup files as a skeleton and customise it to your needs._
+In this section, we show three examples how to work with MLAir.
 
-The following instruction guide you through the installation on JUWELS and HDFML. 
-* Clone the repo to HPC system (we recommend to place it in `/p/projects/<project name>`).
-* Setup venv by executing `source setupHPC.sh`. This script loads all pre-installed modules and creates a venv for 
-all other packages. Furthermore, it creates slurm/batch scripts to execute code on compute nodes. <br> 
-You have to enter the HPC project's budget name (--account flag).
-* The default external data path on JUWELS and HDFML is set to `/p/project/deepacf/intelliaq/<user>/DATA/toar_<sampling>`. 
-<br>To choose a different location open `run.py` and add the following keyword argument to `ExperimentSetup`: 
-`data_path=<your>/<custom>/<path>`. 
-* Execute `python run.py` on a login node to download example data. The program will throw an OSerror after downloading.
-* Execute either `sbatch run_juwels_develgpus.bash` or `sbatch run_hdfml_batch.bash` to verify that the setup went well.
-* Currently cartopy is not working on our HPC system, therefore PlotStations does not create any output.
+## Example 1
 
-Note: The method `PartitionCheck` currently only checks if the hostname starts with `ju` or `hdfmll`. 
-Therefore, it might be necessary to adopt the `if` statement in `PartitionCheck._run`.
+We start MLAir in a dry run without any modification. Just import mlair and run it.
+```python
+import mlair
 
+# just give it a dry run without any modification 
+mlair.run()
+```
+The logging output will show you many informations. Additional information (including debug messages) are collected 
+inside the experiment path in the logging folder.
+```log
+INFO: mlair started
+INFO: ExperimentSetup started
+INFO: Experiment path is: /home/<usr>/mlair/testrun_network 
+...
+INFO: load data for DEBW001 from JOIN 
+...
+INFO: Training started
+...
+INFO: mlair finished after 00:00:12 (hh:mm:ss)
+```
 
-# Security
+## Example 2
 
-* To use hourly data from ToarDB via JOIN interface, a private token is required. Request your personal access token and
-add it to `src/join_settings.py` in the hourly data section. Replace the `TOAR_SERVICE_URL` and the `Authorization` 
-value. To make sure, that this **sensitive** data is not uploaded to the remote server, use the following command to
-prevent git from tracking this file: `git update-index --assume-unchanged src/join_settings.py`
+Now we update the stations and customise the window history size parameter.
+
+```python
+import mlair
+
+# our new stations to use
+stations = ['DEBW030', 'DEBW037', 'DEBW031', 'DEBW015', 'DEBW107']
+
+# expanded temporal context to 14 (days, because of default sampling="daily")
+window_history_size = 14
+
+# restart the experiment with little customisation
+mlair.run(stations=stations, 
+          window_history_size=window_history_size)
+```
+The output looks similar, but we can see, that the new stations are loaded.
+```log
+INFO: mlair started
+INFO: ExperimentSetup started
+...
+INFO: load data for DEBW030 from JOIN 
+INFO: load data for DEBW037 from JOIN 
+...
+INFO: Training started
+...
+INFO: mlair finished after 00:00:24 (hh:mm:ss)
+```
+
+## Example 3
+
+Let's just apply our trained model to new data. Therefore we keep the window history size parameter but change the stations.
+In the run method, we need to disable the trainable and create new model parameters. MLAir will use the model we have
+trained before. Note, this only works if the experiment path has not changed or a suitable trained model is placed 
+inside the experiment path.
+```python
+import mlair
+
+# our new stations to use
+stations = ['DEBY002', 'DEBY079']
+
+# same setting for window_history_size
+window_history_size = 14
+
+# run experiment without training
+mlair.run(stations=stations, 
+          window_history_size=window_history_size, 
+          create_new_model=False, 
+          trainable=False)
+```
+We can see from the terminal that no training was performed. Analysis is now made on the new stations.
+```log
+INFO: mlair started
+...
+INFO: No training has started, because trainable parameter was false. 
+...
+INFO: mlair finished after 00:00:06 (hh:mm:ss)
+```
+
+# Customised workflows and models
+
+# Custom Workflow
+
+MLAir provides a default workflow. If additional steps are to be performed, you have to append custom run modules to 
+the workflow.
+
+```python
+import mlair
+import logging
+
+class CustomStage(mlair.RunEnvironment):
+    """A custom MLAir stage for demonstration."""
+
+    def __init__(self, test_string):
+        super().__init__()  # always call super init method
+        self._run(test_string)  # call a class method
         
-# Customise your experiment
+    def _run(self, test_string):
+        logging.info("Just running a custom stage.")
+        logging.info("test_string = " + test_string)
+        epochs = self.data_store.get("epochs")
+        logging.info("epochs = " + str(epochs))
+
+        
+# create your custom MLAir workflow
+CustomWorkflow = mlair.Workflow()
+# provide stages without initialisation
+CustomWorkflow.add(mlair.ExperimentSetup, epochs=128)
+# add also keyword arguments for a specific stage
+CustomWorkflow.add(CustomStage, test_string="Hello World")
+# finally execute custom workflow in order of adding
+CustomWorkflow.run()
+```
+```log
+INFO: mlair started
+...
+INFO: ExperimentSetup finished after 00:00:12 (hh:mm:ss)
+INFO: CustomStage started
+INFO: Just running a custom stage.
+INFO: test_string = Hello World
+INFO: epochs = 128
+INFO: CustomStage finished after 00:00:01 (hh:mm:ss)
+INFO: mlair finished after 00:00:13 (hh:mm:ss)
+```
+
+## Custom Model
+
+Each model has to inherit from the abstract model class to ensure a smooth training and evaluation behaviour. It is 
+required to implement the set model and set compile options methods. The later has to set the loss at least.
+
+```python
+
+import keras
+from keras.losses import mean_squared_error as mse
+from keras.optimizers import SGD
+
+from mlair.model_modules import AbstractModelClass
+
+class MyLittleModel(AbstractModelClass):
+    """
+    A customised model with a 1x1 Conv, and 3 Dense layers (32, 16
+    window_lead_time). Dropout is used after Conv layer.
+    """
+    def __init__(self, window_history_size, window_lead_time, channels):
+        super().__init__()
+        # settings
+        self.window_history_size = window_history_size
+        self.window_lead_time = window_lead_time
+        self.channels = channels
+        self.dropout_rate = 0.1
+        self.activation = keras.layers.PReLU
+        self.lr = 1e-2
+        # apply to model
+        self.set_model()
+        self.set_compile_options()
+        self.set_custom_objects(loss=self.compile_options['loss'])
+
+    def set_model(self):
+        # add 1 to window_size to include current time step t0
+        shape = (self.window_history_size + 1, 1, self.channels)
+        x_input = keras.layers.Input(shape=shape)
+        x_in = keras.layers.Conv2D(32, (1, 1), padding='same')(x_input)
+        x_in = self.activation()(x_in)
+        x_in = keras.layers.Flatten()(x_in)
+        x_in = keras.layers.Dropout(self.dropout_rate)(x_in)
+        x_in = keras.layers.Dense(32)(x_in)
+        x_in = self.activation()(x_in)
+        x_in = keras.layers.Dense(16)(x_in)
+        x_in = self.activation()(x_in)
+        x_in = keras.layers.Dense(self.window_lead_time)(x_in)
+        out = self.activation()(x_in)
+        self.model = keras.Model(inputs=x_input, outputs=[out])
+
+    def set_compile_options(self):
+        self.compile_options = {"optimizer": SGD(lr=self.lr),
+                                "loss": mse, 
+                                "metrics": ["mse"]}
+```
 
-This section summarises which parameters can be customised for a training.
 
 ## Transformation
 
@@ -98,8 +266,34 @@ scaling values instead of the calculation method. For method *centre*, std can s
 class: `xr.DataArray` with `dims=["variables"]` and one value for each variable.
 
 
-## Inception Model
 
-See a description [here](https://towardsdatascience.com/a-simple-guide-to-the-versions-of-the-inception-network-7fc52b863202)
-or take a look on the papers [Going Deeper with Convolutions (Szegedy et al., 2014)](https://arxiv.org/abs/1409.4842)
-and [Network In Network (Lin et al., 2014)](https://arxiv.org/abs/1312.4400).
+
+
+# Special Remarks
+
+## Special instructions for installation on Jülich HPC systems
+
+_Please note, that the HPC setup is customised for JUWELS and HDFML. When using another HPC system, you can use the HPC 
+setup files as a skeleton and customise it to your needs._
+
+The following instruction guide you through the installation on JUWELS and HDFML. 
+* Clone the repo to HPC system (we recommend to place it in `/p/projects/<project name>`).
+* Setup venv by executing `source setupHPC.sh`. This script loads all pre-installed modules and creates a venv for 
+all other packages. Furthermore, it creates slurm/batch scripts to execute code on compute nodes. <br> 
+You have to enter the HPC project's budget name (--account flag).
+* The default external data path on JUWELS and HDFML is set to `/p/project/deepacf/intelliaq/<user>/DATA/toar_<sampling>`. 
+<br>To choose a different location open `run.py` and add the following keyword argument to `ExperimentSetup`: 
+`data_path=<your>/<custom>/<path>`. 
+* Execute `python run.py` on a login node to download example data. The program will throw an OSerror after downloading.
+* Execute either `sbatch run_juwels_develgpus.bash` or `sbatch run_hdfml_batch.bash` to verify that the setup went well.
+* Currently cartopy is not working on our HPC system, therefore PlotStations does not create any output.
+
+Note: The method `PartitionCheck` currently only checks if the hostname starts with `ju` or `hdfmll`. 
+Therefore, it might be necessary to adopt the `if` statement in `PartitionCheck._run`.
+
+## Security using JOIN
+
+* To use hourly data from ToarDB via JOIN interface, a private token is required. Request your personal access token and
+add it to `src/join_settings.py` in the hourly data section. Replace the `TOAR_SERVICE_URL` and the `Authorization` 
+value. To make sure, that this **sensitive** data is not uploaded to the remote server, use the following command to
+prevent git from tracking this file: `git update-index --assume-unchanged src/join_settings.py`

docs/_source/get-started.rst

 Get started with MachineLearningTools
 =====================================
 
-<what is machinelearningtools?>
-
-MLT Module and Funtion Documentation
-------------------------------------
-
 Install MachineLearningTools
 ----------------------------
 
-Dependencies
+MLAir is based on several python frameworks. To work properly, you have to install all packages from the
+`requirements.txt` file. Additionally to support the geographical plotting part it is required to install geo
+packages built for your operating system. Name names of these package may differ for different systems, we refer
+here to the opensuse / leap OS. The geo plot can be removed from the `plot_list`, in this case there is no need to
+install the geo packages.
+
+* (geo) Install **proj** on your machine using the console. E.g. for opensuse / leap `zypper install proj`
+* (geo) A c++ compiler is required for the installation of the program **cartopy**
+* Install all requirements from [`requirements.txt`](https://gitlab.version.fz-juelich.de/toar/machinelearningtools/-/blob/master/requirements.txt)
+  preferably in a virtual environment
+* (tf) Currently, TensorFlow-1.13 is mentioned in the requirements. We already tested the TensorFlow-1.15 version and couldn't
+  find any compatibility errors. Please note, that tf-1.13 and 1.15 have two distinct branches each, the default branch
+  for CPU support, and the "-gpu" branch for GPU support. If the GPU version is installed, MLAir will make use of the GPU
+  device.
+* Installation of **MLAir**:
+    * Either clone MLAir from the [gitlab repository](https://gitlab.version.fz-juelich.de/toar/machinelearningtools.git)
+      and use it without installation (beside the requirements)
+    * or download the distribution file (?? .whl) and install it via `pip install <??>`. In this case, you can simply
+      import MLAir in any python script inside your virtual environment using `import mlair`.
+
+
+How to start with MLAir
+-----------------------
+
+In this section, we show three examples how to work with MLAir.
+
+Example 1
+~~~~~~~~~
+
+We start MLAir in a dry run without any modification. Just import mlair and run it.
+
+.. code-block:: python
+
+    import mlair
+
+    # just give it a dry run without any modification
+    mlair.run()
+
+
+The logging output will show you many informations. Additional information (including debug messages) are collected
+inside the experiment path in the logging folder.
+
+.. code-block::
+
+    INFO: mlair started
+    INFO: ExperimentSetup started
+    INFO: Experiment path is: /home/<usr>/mlair/testrun_network
+    ...
+    INFO: load data for DEBW001 from JOIN
+    ...
+    INFO: Training started
+    ...
+    INFO: mlair finished after 00:00:12 (hh:mm:ss)
+
+
+Example 2
+~~~~~~~~~
+
+Now we update the stations and customise the window history size parameter.
+
+.. code-block:: python
+
+    import mlair
+
+    # our new stations to use
+    stations = ['DEBW030', 'DEBW037', 'DEBW031', 'DEBW015', 'DEBW107']
+
+    # expanded temporal context to 14 (days, because of default sampling="daily")
+    window_history_size = 14
+
+    # restart the experiment with little customisation
+    mlair.run(stations=stations,
+              window_history_size=window_history_size)
+
+The output looks similar, but we can see, that the new stations are loaded.
+
+.. code-block::
+
+    INFO: mlair started
+    INFO: ExperimentSetup started
+    ...
+    INFO: load data for DEBW030 from JOIN
+    INFO: load data for DEBW037 from JOIN
+    ...
+    INFO: Training started
+    ...
+    INFO: mlair finished after 00:00:24 (hh:mm:ss)
+
+Example 3
+~~~~~~~~~
+
+Let's just apply our trained model to new data. Therefore we keep the window history size parameter but change the stations.
+In the run method, we need to disable the trainable and create new model parameters. MLAir will use the model we have
+trained before. Note, this only works if the experiment path has not changed or a suitable trained model is placed
+inside the experiment path.
+
+.. code-block:: python
+
+    import mlair
+
+    # our new stations to use
+    stations = ['DEBY002', 'DEBY079']
+
+    # same setting for window_history_size
+    window_history_size = 14
+
+    # run experiment without training
+    mlair.run(stations=stations,
+              window_history_size=window_history_size,
+              create_new_model=False,
+              trainable=False)
+
+We can see from the terminal that no training was performed. Analysis is now made on the new stations.
+
+.. code-block::
+
+    INFO: mlair started
+    ...
+    INFO: No training has started, because trainable parameter was false.
+    ...
+    INFO: mlair finished after 00:00:06 (hh:mm:ss)
+
+
+
+Customised workflows and models
+-------------------------------
+
+Custom Workflow
+~~~~~~~~~~~~~~~
+
+MLAir provides a default workflow. If additional steps are to be performed, you have to append custom run modules to
+the workflow.
+
+.. code-block:: python
+    import mlair
+    import logging
+
+    class CustomStage(mlair.RunEnvironment):
+        """A custom MLAir stage for demonstration."""
+
+        def __init__(self, test_string):
+            super().__init__()  # always call super init method
+            self._run(test_string)  # call a class method
+
+        def _run(self, test_string):
+            logging.info("Just running a custom stage.")
+            logging.info("test_string = " + test_string)
+            epochs = self.data_store.get("epochs")
+            logging.info("epochs = " + str(epochs))
+
+
+    # create your custom MLAir workflow
+    CustomWorkflow = mlair.Workflow()
+    # provide stages without initialisation
+    CustomWorkflow.add(mlair.ExperimentSetup, epochs=128)
+    # add also keyword arguments for a specific stage
+    CustomWorkflow.add(CustomStage, test_string="Hello World")
+    # finally execute custom workflow in order of adding
+    CustomWorkflow.run()
+
+.. code-block::
+
+    INFO: mlair started
+    ...
+    INFO: ExperimentSetup finished after 00:00:12 (hh:mm:ss)
+    INFO: CustomStage started
+    INFO: Just running a custom stage.
+    INFO: test_string = Hello World
+    INFO: epochs = 128
+    INFO: CustomStage finished after 00:00:01 (hh:mm:ss)
+    INFO: mlair finished after 00:00:13 (hh:mm:ss)
+
+Custom Model
 ~~~~~~~~~~~~
 
-Data
-~~~~
+Each model has to inherit from the abstract model class to ensure a smooth training and evaluation behaviour. It is
+required to implement the set model and set compile options methods. The later has to set the loss at least.
+
+.. code-block:: python
+
+    import keras
+    from keras.losses import mean_squared_error as mse
+    from keras.optimizers import SGD
+
+    from mlair.model_modules import AbstractModelClass
+
+    class MyLittleModel(AbstractModelClass):
+        """
+        A customised model with a 1x1 Conv, and 3 Dense layers (32, 16
+        window_lead_time). Dropout is used after Conv layer.
+        """
+        def __init__(self, window_history_size, window_lead_time, channels):
+            super().__init__()
+            # settings
+            self.window_history_size = window_history_size
+            self.window_lead_time = window_lead_time
+            self.channels = channels
+            self.dropout_rate = 0.1
+            self.activation = keras.layers.PReLU
+            self.lr = 1e-2
+            # apply to model
+            self.set_model()
+            self.set_compile_options()
+            self.set_custom_objects(loss=self.compile_options['loss'])
+
+        def set_model(self):
+            # add 1 to window_size to include current time step t0
+            shape = (self.window_history_size + 1, 1, self.channels)
+            x_input = keras.layers.Input(shape=shape)
+            x_in = keras.layers.Conv2D(32, (1, 1), padding='same')(x_input)
+            x_in = self.activation()(x_in)
+            x_in = keras.layers.Flatten()(x_in)
+            x_in = keras.layers.Dropout(self.dropout_rate)(x_in)
+            x_in = keras.layers.Dense(32)(x_in)
+            x_in = self.activation()(x_in)
+            x_in = keras.layers.Dense(16)(x_in)
+            x_in = self.activation()(x_in)
+            x_in = keras.layers.Dense(self.window_lead_time)(x_in)
+            out = self.activation()(x_in)
+            self.model = keras.Model(inputs=x_input, outputs=[out])
+    
+        def set_compile_options(self):
+            self.compile_options = {"optimizer": SGD(lr=self.lr),
+                                    "loss": mse,
+                                    "metrics": ["mse"]}