diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 32d8f138020e175ed4e17077713ed7cb26c5c533..75a77bb3e7eaf523aaf7f7bcf57b3b6a3e555058 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -43,7 +43,7 @@ tests (from scratch):
script:
- zypper --non-interactive install binutils libproj-devel gdal-devel
- zypper --non-interactive install proj geos-devel
- - pip install -r requirements.txt
+ - cat requirements.txt | cut -f1 -d"#" | sed '/^\s*$/d' | xargs -L 1 pip install
- chmod +x ./CI/run_pytest.sh
- ./CI/run_pytest.sh
after_script:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7d17fa89513192bf7d8ce0fa39e6e1449dc9b4f0..a276b5eada4abab9a64f7dfec2ae786cabbbbd03 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,14 @@
All notable changes to this project will be documented in this file.
+## v0.12.2 - 2020-10-01 - HDFML support
+
+### general:
+- HDFML support
+
+### technical:
+- installation script for HDFML adjusted, #183
+
## v0.12.1 - 2020-09-28 - examples in notebook
### general:
diff --git a/LICENSE b/LICENSE
index a79ea789a5b55f7328d1fd987293376838112048..27cf6cf57788b5ccf0c7aa32eb5b819af0a09600 100644
--- a/LICENSE
+++ b/LICENSE
@@ -1,6 +1,6 @@
MIT License
-Copyright (c) 2020 Lukas Leufen
+Copyright (c) 2020 Forschungszentrum Jülich GmbH
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
diff --git a/README.md b/README.md
index 6786f0ddb31e0616bf10161f94e80f09f8afeb1b..40954c7896811f364ae1efd4d0065aa6cad215f1 100644
--- a/README.md
+++ b/README.md
@@ -10,32 +10,86 @@ docs [here](http://toar.pages.jsc.fz-juelich.de/mlair/docs/).
MLAir is based on several python frameworks. To work properly, you have to install all packages from the
`requirements.txt` file. Additionally to support the geographical plotting part it is required to install geo
-packages built for your operating system. Name names of these package may differ for different systems, we refer
-here to the opensuse / leap OS. The geo plot can be removed from the `plot_list`, in this case there is no need to
-install the geo packages. For special instructions to install MLAir on the Juelich HPC systems, see
-[here](#special-instructions-for-installation-on-jülich-hpc-systems).
-
-* (geo) Install **proj** on your machine using the console. E.g. for opensuse / leap `zypper install proj`
-* (geo) A c++ compiler is required for the installation of the program **cartopy**
-* Install all requirements from [`requirements.txt`](https://gitlab.version.fz-juelich.de/toar/mlair/-/blob/master/requirements.txt)
- preferably in a virtual environment
-* (tf) Currently, TensorFlow-1.13 is mentioned in the requirements. We already tested the TensorFlow-1.15 version and couldn't
- find any compatibility errors. Please note, that tf-1.13 and 1.15 have two distinct branches each, the default branch
- for CPU support, and the "-gpu" branch for GPU support. If the GPU version is installed, MLAir will make use of the GPU
- device.
+packages built for your operating system. Unfortunately, the names of these package may differ for different systems.
+In this instruction, we try to address users of different operating systems namely openSUSE Leap, Ubuntu and macOS.
+If the installation is still not working, we recommend skipping the geographical plot. We have put together a small
+workaround [here](#workaround-to-skip-geographical-plot). For special instructions to install MLAir on the Juelich
+HPC systems, see [here](#special-instructions-for-installation-on-jülich-hpc-systems).
+
+* Make sure to have the **python3.6** version installed.
+* (geo) A **c++ compiler** is required for the installation of the program **cartopy**
+* (geo) Install **proj** and **GEOS** on your machine using the console.
+* Install the **python3.6 develop** libraries.
+* Install all **requirements** from [`requirements.txt`](https://gitlab.version.fz-juelich.de/toar/mlair/-/blob/master/requirements.txt)
+ preferably in a virtual environment. You can use `pip install -r requirements.txt` to install all requirements at
+ once. Note, we recently updated the version of Cartopy and there seems to be an ongoing
+ [issue](https://github.com/SciTools/cartopy/issues/1552) when installing numpy and Cartopy at the same time. If you
+ run into trouble, you could use `cat requirements.txt | cut -f1 -d"#" | sed '/^\s*$/d' | xargs -L 1 pip install`
+ instead.
* Installation of **MLAir**:
* Either clone MLAir from the [gitlab repository](https://gitlab.version.fz-juelich.de/toar/mlair.git)
and use it without installation (beside the requirements)
* or download the distribution file ([current version](https://gitlab.version.fz-juelich.de/toar/mlair/-/blob/master/dist/mlair-0.12.2-py3-none-any.whl))
and install it via `pip install <dist_file>.whl`. In this case, you can simply import MLAir in any python script
inside your virtual environment using `import mlair`.
-
+* (tf) Currently, TensorFlow-1.13 is mentioned in the requirements. We already tested the TensorFlow-1.15 version and couldn't
+ find any compatibility errors. Please note, that tf-1.13 and 1.15 have two distinct branches each, the default branch
+ for CPU support, and the "-gpu" branch for GPU support. If the GPU version is installed, MLAir will make use of the GPU
+ device.
+
+## openSUSE Leap 15.1
+
+* c++ compiler
+
+`sudo zypper install gcc-c++`
+
+* geo packages
+
+`sudo zypper install proj geos-devel`
+
+* depending on the pre-installed packages it could be required to install further packages
+
+`sudo zypper install libproj-devel binutils gdal-devel graphviz`
+
+* python develop libraries
+
+`sudo zypper install python3-devel`
+
+## Ubuntu 20.04.1
+
+* c++ compiler
+
+`sudo apt install build-essential`
+
+* geo packages
+
+`sudo apt install proj-bin libgeos-dev libproj-dev`
+
+* depending on the pre-installed packages it could be required to install further packages
+
+`sudo apt install graphviz libgeos++-dev`
+
+* python develop libraries
+
+`sudo apt install python3.6-dev`
+
+## macOS & windows
+
+The installation on macOS is not tested yet. The following commands are possibly needed:
+
+`brew install geos`
+
+`sudo port install graphviz`
+
+The installation on Windows is not tested yet.
# How to start with MLAir
In this section, we show three examples how to work with MLAir. Note, that for these examples MLAir was installed using
the distribution file. In case you are using the git clone it is required to adjust the import path if not directly
-executed inside the source directory of MLAir.
+executed inside the source directory of MLAir. There is also a downloadable
+[Jupyter Notebook](https://gitlab.version.fz-juelich.de/toar/mlair/-/blob/master/supplement/Examples_from_manuscript.ipynb)
+provided in that you can run the following examples. Note that this notebook still requires an installation of MLAir.
## Example 1
@@ -421,6 +475,20 @@ class DummyDataHandler(AbstractDataHandler):
# Special Remarks
+## Workaround to skip geographical plot
+
+If it is not possible to install all required geo libraries on your system, a good compromise is to skip the creation
+of the geographical plot. Therefore, it is required to remove the plot from the `plot_list` manually. We recommend to
+use this code snippet as a starting point.
+
+```python
+from mlair.helpers import remove_items
+from mlair.configuration.defaults import DEFAULT_PLOT_LIST
+
+mlair.run(plot_list=remove_items(DEFAULT_PLOT_LIST, "PlotStationMap"))
+
+```
+
## Special instructions for installation on Jülich HPC systems
_Please note, that the HPC setup is customised for JUWELS and HDFML. When using another HPC system, you can use the HPC
@@ -429,10 +497,10 @@ setup files as a skeleton and customise it to your needs._
The following instruction guide you through the installation on JUWELS and HDFML.
* Clone the repo to HPC system (we recommend to place it in `/p/projects/<project name>`).
* Setup venv by executing `source setupHPC.sh`. This script loads all pre-installed modules and creates a venv for
-all other packages. Furthermore, it creates slurm/batch scripts to execute code on compute nodes. <br>
+all other packages. Furthermore, it creates slurm/batch scripts to execute code on compute nodes.
You have to enter the HPC project's budget name (--account flag).
* The default external data path on JUWELS and HDFML is set to `/p/project/deepacf/intelliaq/<user>/DATA/toar_<sampling>`.
-<br>To choose a different location open `run.py` and add the following keyword argument to `ExperimentSetup`:
+To choose a different location open `run.py` and add the following keyword argument to `ExperimentSetup`:
`data_path=<your>/<custom>/<path>`.
* Execute `python run.py` on a login node to download example data. The program will throw an OSerror after downloading.
* Execute either `sbatch run_juwels_develgpus.bash` or `sbatch run_hdfml_batch.bash` to verify that the setup went well.
diff --git a/docs/_source/get-started.rst b/docs/_source/get-started.rst
index 7dda0eccb41281f6d15b4434b7e13ed2aa332d35..ee4a53e9e0c7c2b4b06ef768c2122351da5cd121 100644
--- a/docs/_source/get-started.rst
+++ b/docs/_source/get-started.rst
@@ -11,19 +11,19 @@ Install MLAir
MLAir is based on several python frameworks. To work properly, you have to install all packages from the
:py:`requirements.txt` file. Additionally to support the geographical plotting part it is required to install geo
-packages built for your operating system. Name names of these package may differ for different systems, we refer
-here to the opensuse / leap OS. The geo plot can be removed from the :py:`plot_list`, in this case there is no need to
-install the geo packages.
+packages built for your operating system. Unfortunately, the names of these package may differ for different systems.
+In this instruction, we try to address users of different operating systems namely openSUSE Leap, Ubuntu and macOS.
+If the installation is still not working, we recommend skipping the geographical plot. We have put together a small
+workaround :ref:`here<Workaround to skip geographical plot>`. For special instructions to install MLAir on the Juelich
+HPC systems, see section :ref:`Installation on Jülich HPC systems`.
Pre-requirements
~~~~~~~~~~~~~~~~
-* (geo) Install **proj** on your machine using the console. E.g. for opensuse / leap :py:`zypper install proj`
-* (geo) A c++ compiler is required for the installation of the program **cartopy**
-* (tf) Currently, TensorFlow-1.13 is mentioned in the requirements. We already tested the TensorFlow-1.15 version and couldn't
- find any compatibility errors. Please note, that tf-1.13 and 1.15 have two distinct branches each, the default branch
- for CPU support, and the "-gpu" branch for GPU support. If the GPU version is installed, MLAir will make use of the GPU
- device.
+* Make sure to have the **python3.6** version installed.
+* (geo) A **c++ compiler** is required for the installation of the program **cartopy**
+* (geo) Install **proj** and **GEOS** on your machine using the console.
+* Install the **python3.6 develop** libraries.
Installation of MLAir
~~~~~~~~~~~~~~~~~~~~~
@@ -34,6 +34,100 @@ Installation of MLAir
* or download the distribution file (`current version <https://gitlab.version.fz-juelich.de/toar/mlair/-/blob/master/dist/mlair-0.12.2-py3-none-any.whl>`_)
and install it via :py:`pip install <dist_file>.whl`. In this case, you can simply
import MLAir in any python script inside your virtual environment using :py:`import mlair`.
+* (tf) Currently, TensorFlow-1.13 is mentioned in the requirements. We already tested the TensorFlow-1.15 version and couldn't
+ find any compatibility errors. Please note, that tf-1.13 and 1.15 have two distinct branches each, the default branch
+ for CPU support, and the "-gpu" branch for GPU support. If the GPU version is installed, MLAir will make use of the GPU
+ device.
+
+Special Instructions for Installation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+openSUSE Leap 15.1
+""""""""""""""""""
+
+* c++ compiler
+
+:py:`sudo zypper install gcc-c++`
+
+* geo packages
+
+:py:`sudo zypper install proj geos-devel`
+
+* depending on the pre-installed packages it could be required to install further packages
+
+:py:`sudo zypper install libproj-devel binutils gdal-devel graphviz`
+
+* python develop libraries
+
+:py:`sudo zypper install python3-devel`
+
+Ubuntu 20.04.1
+""""""""""""""
+
+* c++ compiler
+
+:py:`sudo apt install build-essential`
+
+* geo packages
+
+:py:`sudo apt install proj-bin libgeos-dev libproj-dev`
+
+* depending on the pre-installed packages it could be required to install further packages
+
+:py:`sudo apt install graphviz libgeos++-dev`
+
+* python develop libraries
+
+:py:`sudo apt install python3.6-dev`
+
+macOS & windows
+"""""""""""""""
+
+The installation on macOS is not tested yet. The following commands are possibly needed:
+
+:py:`brew install geos`
+
+:py:`sudo port install graphviz`
+
+The installation on Windows is not tested yet.
+
+Installation on Jülich HPC systems
+""""""""""""""""""""""""""""""""""
+
+*Please note, that the HPC setup is customised for JUWELS and HDFML. When using another HPC system, you can use the HPC
+setup files as a skeleton and customise it to your needs.*
+
+The following instruction guide you through the installation on JUWELS and HDFML.
+
+* Clone the repo to HPC system (we recommend to place it in :py:`/p/projects/<project name>`).
+* Setup venv by executing :py:`source setupHPC.sh`. This script loads all pre-installed modules and creates a venv for
+ all other packages. Furthermore, it creates slurm/batch scripts to execute code on compute nodes.
+ You have to enter the HPC project's budget name (--account flag).
+* The default external data path on JUWELS and HDFML is set to :py:`/p/project/deepacf/intelliaq/<user>/DATA/toar_<sampling>`.
+* To choose a different location open :py:`run.py` and add the following keyword argument to :py:`ExperimentSetup`:
+ :py:`data_path=<your>/<custom>/<path>`.
+* Execute :py:`python run.py` on a login node to download example data. The program will throw an OSerror after downloading.
+* Execute either :py:`sbatch run_juwels_develgpus.bash` or :py:`sbatch run_hdfml_batch.bash` to verify that the setup
+ went well.
+* Currently cartopy is not working on our HPC system, therefore PlotStations does not create any output.
+
+Note: The method :py:`PartitionCheck` currently only checks if the hostname starts with :py:`ju` or :py:`hdfmll`.
+Therefore, it might be necessary to adopt the :py:`if` statement in :py:`PartitionCheck._run`.
+
+
+Workaround to skip geographical plot
+""""""""""""""""""""""""""""""""""""
+
+If it is not possible to install all required geo libraries on your system, a good compromise is to skip the creation
+of the geographical plot. Therefore, it is required to remove the plot from the :py:`plot_list` manually. We recommend
+to use this code snippet as a starting point.
+
+.. code-block:: python
+
+ from mlair.helpers import remove_items
+ from mlair.configuration.defaults import DEFAULT_PLOT_LIST
+
+ mlair.run(plot_list=remove_items(DEFAULT_PLOT_LIST, "PlotStationMap"))
How to start with MLAir
diff --git a/docs/_source/index.rst b/docs/_source/index.rst
index 5777cf162545d81c11e3834c74a62271d05254d2..705ab91ddc1000edfa7961719db1ff913de26a93 100644
--- a/docs/_source/index.rst
+++ b/docs/_source/index.rst
@@ -6,6 +6,7 @@
Welcome to MLAir's documentation!
================================================
+This is the documation of the `MLAir package <https://gitlab.version.fz-juelich.de/toar/mlair>`_.
.. toctree::
:maxdepth: 2
diff --git a/mlair/__init__.py b/mlair/__init__.py
index c0f6183fc3c30cd91818915de12115fd9679f648..0332c7b92700377eb8b4a5039539c0ae6156de52 100644
--- a/mlair/__init__.py
+++ b/mlair/__init__.py
@@ -1,7 +1,7 @@
__version_info__ = {
'major': 0,
'minor': 12,
- 'micro': 1,
+ 'micro': 2,
}
from mlair.run_modules import RunEnvironment, ExperimentSetup, PreProcessing, ModelSetup, Training, PostProcessing
diff --git a/requirements.txt b/requirements.txt
index e2d8f5bc47e3ab1365814259b313b1a07e54fff3..be76eab5b74797b039682a292ae8890488c058ec 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,14 +1,13 @@
+Cython==0.29.15
absl-py==0.9.0
astor==0.8.1
atomicwrites==1.3.0
attrs==19.3.0
-Cartopy==0.17.0
certifi==2019.11.28
chardet==3.0.4
cloudpickle==1.3.0
coverage==5.0.3
cycler==0.10.0
-Cython==0.29.15
dask==2.11.0
fsspec==0.6.2
gast==0.3.3
@@ -50,7 +49,6 @@ PyYAML==5.3
requests==2.23.0
scipy==1.4.1
seaborn==0.10.0
---no-binary shapely Shapely==1.7.0
six==1.11.0
statsmodels==0.11.1
tabulate
@@ -66,4 +64,6 @@ Werkzeug==1.0.0
xarray==0.15.0
zipp==3.1.0
-setuptools~=49.6.0
\ No newline at end of file
+setuptools~=49.6.0
+Cartopy==0.18.0
+--no-binary shapely Shapely==1.7.0
\ No newline at end of file
diff --git a/requirements_gpu.txt b/requirements_gpu.txt
index 38598ff6b902da6d96eb870124f60c72737000c2..45e04819f355640dc4a85218440b95164dba280c 100644
--- a/requirements_gpu.txt
+++ b/requirements_gpu.txt
@@ -1,14 +1,13 @@
+Cython==0.29.15
absl-py==0.9.0
astor==0.8.1
atomicwrites==1.3.0
attrs==19.3.0
-#Cartopy==0.17.0
certifi==2019.11.28
chardet==3.0.4
cloudpickle==1.3.0
coverage==5.0.3
cycler==0.10.0
-Cython==0.29.15
dask==2.11.0
fsspec==0.6.2
gast==0.3.3
@@ -50,7 +49,6 @@ PyYAML==5.3
requests==2.23.0
scipy==1.4.1
seaborn==0.10.0
---no-binary shapely Shapely==1.7.0
six==1.11.0
statsmodels==0.11.1
tabulate
@@ -65,3 +63,6 @@ wcwidth==0.1.8
Werkzeug==1.0.0
xarray==0.15.0
zipp==3.1.0
+
+#Cartopy==0.18.0
+--no-binary shapely Shapely==1.7.0
\ No newline at end of file
diff --git a/run_zam347.py b/run_zam347.py
index 9027bec807ad9beafcdac573a70aa32d34491034..352f04177167441d3636359a9f6ade5f039c12c1 100644
--- a/run_zam347.py
+++ b/run_zam347.py
@@ -16,7 +16,7 @@ from mlair.workflows import DefaultWorkflowHPC
def load_stations():
try:
- filename = 'German_background_stations.json'
+ filename = 'supplement/German_background_stations.json'
with open(filename, 'r') as jfile:
stations = json.load(jfile)
logging.info(f"load station IDs from file: {filename} ({len(stations)} stations)")
diff --git a/Examples_from_manuscript.ipynb b/supplement/Examples_from_manuscript.ipynb
similarity index 100%
rename from Examples_from_manuscript.ipynb
rename to supplement/Examples_from_manuscript.ipynb
diff --git a/German_background_stations.json b/supplement/German_background_stations.json
similarity index 100%
rename from German_background_stations.json
rename to supplement/German_background_stations.json
diff --git a/development_guidelines.md b/supplement/development_guidelines.md
similarity index 100%
rename from development_guidelines.md
rename to supplement/development_guidelines.md