Skip to content
Snippets Groups Projects
Commit d3011f62 authored by Felix Kleinert's avatar Felix Kleinert
Browse files

update examples

parent 5686c88b
No related branches found
No related tags found
4 merge requests!156include current development into release,!155Resolve "new release v0.12.1",!148Resolve "Create Quick-Start Notebook",!139Draft: Resolve "KZ filter"
Pipeline #46305 passed
Stage: test
Stage: docs
Stage: pages
Stage: deploy
Hide whitespace changes
Inline Side-by-side
Examples_from_manuscript.ipynb
+ 8
− 4
View file @ d3011f62
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# MLAir (v1.0) - Examples # MLAir (v1.0) - Examples
This notebook contains all examples as provided in Leufen et al. (2020). This notebook contains all examples as provided in Leufen et al. (2020).
Please follow the installation instructions provided in the [README](https://gitlab.version.fz-juelich.de/toar/mlair/-/blob/master/README.md) on gitlab. Please follow the installation instructions provided in the [README](https://gitlab.version.fz-juelich.de/toar/mlair/-/blob/master/README.md) on gitlab.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Example 1 ## Example 1
The following cell imports MLAir and executes a minimalistic toy experiment. This cell is equivalent to Figure 2 in the manuscript. The following cell imports MLAir and executes a minimalistic toy experiment. This cell is equivalent to Figure 2 in the manuscript.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
import mlair import mlair
# just give it a dry run without any modifications # just give it a dry run without any modifications
mlair.run() mlair.run()
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Example 2 ## Example 2
In the following cell we use other station IDs provided as a list of strings (see also [JOIN-Web interface](https://join.fz-juelich.de/services/rest/surfacedata/) of the TOAR database for more details). In the following cell we use other station IDs provided as a list of strings (see also [JOIN-Web interface](https://join.fz-juelich.de/services/rest/surfacedata/) of the TOAR database for more details).
Moreover, we expand the `window_history_size` to 14 days and run the experiment. This cell is equivalent to Figure 3 in the manuscript. Moreover, we expand the `window_history_size` to 14 days and run the experiment. This cell is equivalent to Figure 3 in the manuscript.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# our new stations to use # our new stations to use
stations = ['DEBW030', 'DEBW037', 'DEBW031', 'DEBW015', 'DEBW107'] stations = ['DEBW030', 'DEBW037', 'DEBW031', 'DEBW015', 'DEBW107']
# expanded temporal context to 14 (days, because of default sampling="daily") # expanded temporal context to 14 (days, because of default sampling="daily")
window_history_size = 14 window_history_size = 14
# restart the experiment with little customisation # restart the experiment with little customisation
mlair.run(stations=stations, mlair.run(stations=stations,
window_history_size=window_history_size) window_history_size=window_history_size)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Example 3 ## Example 3
The following cell loads the trained model from Example 2 and generates predictions for the two specified stations. The following cell loads the trained model from Example 2 and generates predictions for the two specified stations.
To ensure that the model is not retrained the keywords `create_new_model` and `train_model` are set to `False`. This cell is equivalent to Figure 4 in the manuscript. To ensure that the model is not retrained the keywords `create_new_model` and `train_model` are set to `False`. This cell is equivalent to Figure 4 in the manuscript.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# our new stations to use # our new stations to use
stations = ['DEBY002', 'DEBY079'] stations = ['DEBY002', 'DEBY079']
# same setting for window_history_size # same setting for window_history_size
window_history_size = 14 window_history_size = 14
# run experiment without training # run experiment without training
mlair.run(stations=stations, mlair.run(stations=stations,
window_history_size=window_history_size, window_history_size=window_history_size,
create_new_model=False, create_new_model=False,
train_model=False) train_model=False)
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Example 4 ## Example 4
The following cell demonstrates how a user defined model can be implemented by inheriting from `AbstractModelClass`. Within the `__init__` method `super().__init__`, `set_model` and `set_compile_options` should be called. Moreover, it is possible to set custom objects by calling `set_custom_objects`. Those custom objects are used to re-load the model (see also Keras documentation). For demonstration, the loss is added as custom object which is not required because a Keras built-in function is used as loss. The following cell demonstrates how a user defined model can be implemented by inheriting from `AbstractModelClass`. Within the `__init__` method `super().__init__`, `set_model` and `set_compile_options` should be called. Moreover, it is possible to set custom objects by calling `set_custom_objects`. Those custom objects are used to re-load the model (see also Keras documentation). For demonstration, the loss is added as custom object which is not required because a Keras built-in function is used as loss.
The Keras-model itself is defined in `set_model` by using the sequential or functional Keras API. All compile options can be defined in `set_compile_options`. The Keras-model itself is defined in `set_model` by using the sequential or functional Keras API. All compile options can be defined in `set_compile_options`.
This cell is equivalent to Figure 5 in the manuscript. This cell is equivalent to Figure 5 in the manuscript.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# Figure 5
import keras import keras
from keras.losses import mean_squared_error as mse from keras.losses import mean_squared_error as mse
from keras.layers import PReLU, Input, Conv2D, Flatten, Dropout, Dense from keras.layers import PReLU, Input, Conv2D, Flatten, Dropout, Dense
from mlair.model_modules import AbstractModelClass from mlair.model_modules import AbstractModelClass
from mlair.workflows import DefaultWorkflow
class MyCustomisedModel(AbstractModelClass): class MyCustomisedModel(AbstractModelClass):
""" """
A customised model with a 1x1 Conv, and 2 Dense layers (16, A customised model with a 1x1 Conv, and 2 Dense layers (16,
output shape). Dropout is used after Conv layer. output shape). Dropout is used after Conv layer.
""" """
def __init__(self, input_shape: list, output_shape: list): def __init__(self, input_shape: list, output_shape: list):
# set attributes shape_inputs and shape_outputs # set attributes shape_inputs and shape_outputs
super().__init__(input_shape[0], output_shape[0]) super().__init__(input_shape[0], output_shape[0])
# apply to model # apply to model
self.set_model() self.set_model()
self.set_compile_options() self.set_compile_options()
self.set_custom_objects(loss=self.compile_options['loss']) self.set_custom_objects(loss=self.compile_options['loss'])
def set_model(self): def set_model(self):
x_input = Input(shape=self.shape_inputs) x_input = Input(shape=self._input_shape)
x_in = Conv2D(4, (1, 1))(x_input) x_in = Conv2D(4, (1, 1))(x_input)
x_in = PReLU()(x_in) x_in = PReLU()(x_in)
x_in = Flatten()(x_in) x_in = Flatten()(x_in)
x_in = Dropout(0.1)(x_in) x_in = Dropout(0.1)(x_in)
x_in = Dense(16)(x_in) x_in = Dense(16)(x_in)
x_in = PReLU()(x_in) x_in = PReLU()(x_in)
x_in = Dense(self.shape_outputs)(x_in) x_in = Dense(self._output_shape)(x_in)
out = PReLU()(x_in) out = PReLU()(x_in)
self.model = keras.Model(inputs=x_input, outputs=[out]) self.model = keras.Model(inputs=x_input, outputs=[out])
def set_compile_options(self): def set_compile_options(self):
self.initial_lr = 1e-2 self.initial_lr = 1e-2
self.optimizer = keras.optimizers.SGD(lr=self.initial_lr, momentum=0.9) self.optimizer = keras.optimizers.SGD(lr=self.initial_lr, momentum=0.9)
self.loss = mse self.loss = mse
self.compile_options = {"metrics": ["mse", "mae"]} self.compile_options = {"metrics": ["mse", "mae"]}
# Make use of MyCustomisedModel within the DefaultWorkflow
workflow = DefaultWorkflow(model=MyCustomisedModel, epochs=2)
workflow.run()
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Example 5 ## Example 5
Embedding of a custom Run Module in a modified MLAir workflow. In comparison to examples 1 to 4, this code example works on a single step deeper regarding the level of abstraction. Instead of calling the run method of MLAir, the user needs to add all stages individually and is responsible for all dependencies between the stages. By using the `Workflow` class as context manager, all stages are automatically connected with the result that all stages can easily be plugged in. This cell is equivalent to Figure 6 in the manuscript. Embedding of a custom Run Module in a modified MLAir workflow. In comparison to examples 1 to 4, this code example works on a single step deeper regarding the level of abstraction. Instead of calling the run method of MLAir, the user needs to add all stages individually and is responsible for all dependencies between the stages. By using the `Workflow` class as context manager, all stages are automatically connected with the result that all stages can easily be plugged in. This cell is equivalent to Figure 6 in the manuscript.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
import logging import logging
class CustomStage(mlair.RunEnvironment): class CustomStage(mlair.RunEnvironment):
"""A custom MLAir stage for demonstration.""" """A custom MLAir stage for demonstration."""
def __init__(self, test_string): def __init__(self, test_string):
super().__init__() # always call super init method super().__init__() # always call super init method
self._run(test_string) # call a class method self._run(test_string) # call a class method
def _run(self, test_string): def _run(self, test_string):
logging.info("Just running a custom stage.") logging.info("Just running a custom stage.")
logging.info("test_string = " + test_string) logging.info("test_string = " + test_string)
epochs = self.data_store.get("epochs") epochs = self.data_store.get("epochs")
logging.info("epochs = " + str(epochs)) logging.info("epochs = " + str(epochs))
# create your custom MLAir workflow # create your custom MLAir workflow
CustomWorkflow = mlair.Workflow() CustomWorkflow = mlair.Workflow()
# provide stages without initialisation # provide stages without initialisation
CustomWorkflow.add(mlair.ExperimentSetup, epochs=128) CustomWorkflow.add(mlair.ExperimentSetup, epochs=128)
# add also keyword arguments for a specific stage # add also keyword arguments for a specific stage
CustomWorkflow.add(CustomStage, test_string="Hello World") CustomWorkflow.add(CustomStage, test_string="Hello World")
# finally execute custom workflow in order of adding # finally execute custom workflow in order of adding
CustomWorkflow.run() CustomWorkflow.run()
``` ```
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment