Skip to content
Snippets Groups Projects
Commit 161a8672 authored by leufen1's avatar leufen1
Browse files

use cartopy 0.18, added detailed installation instructions for openSUSE and Ubuntu

parent e89a2014
No related branches found
No related tags found
4 merge requests!164update release branch by recent developments,!162Resolve "new release v1.0.0",!161detailed install instructions,!139Draft: Resolve "KZ filter"
Pipeline #47870 passed
Stage: test
Stage: docs
Stage: pages
Stage: deploy
Show whitespace changes
Inline Side-by-side
.gitlab-ci.yml
+ 1
− 1
View file @ 161a8672
......@@ -72,7 +72,7 @@ tests (on GPU):
- chmod +x ./CI/update_badge.sh
- ./CI/update_badge.sh > /dev/null
script:
- pip install -r requirements.txt
- pip install -r requirements_gpu.txt
- chmod +x ./CI/run_pytest.sh
- ./CI/run_pytest.sh
after_script:
......
README.md
+ 77
− 15
View file @ 161a8672
......@@ -10,26 +10,74 @@ docs [here](http://toar.pages.jsc.fz-juelich.de/mlair/docs/).
MLAir is based on several python frameworks. To work properly, you have to install all packages from the
`requirements.txt` file. Additionally to support the geographical plotting part it is required to install geo
packages built for your operating system. Name names of these package may differ for different systems, we refer
here to the opensuse / leap OS. The geo plot can be removed from the `plot_list`, in this case there is no need to
install the geo packages. For special instructions to install MLAir on the Juelich HPC systems, see
[here](#special-instructions-for-installation-on-jülich-hpc-systems).
* (geo) Install **proj** on your machine using the console. E.g. for opensuse / leap `zypper install proj`
* (geo) A c++ compiler is required for the installation of the program **cartopy**
* Install all requirements from [`requirements.txt`](https://gitlab.version.fz-juelich.de/toar/mlair/-/blob/master/requirements.txt)
packages built for your operating system. Unfortunately, the names of these package may differ for different systems.
In this instruction, we try to address users of different operating systems namely openSUSE Leap, Ubuntu and macOS.
If the installation is still not working, we recommend skipping the geographical plot. We have put together a small
workaround [here](#workaround-to-skip-geographical-plot). For special instructions to install MLAir on the Juelich
HPC systems, see [here](#special-instructions-for-installation-on-jülich-hpc-systems).
* Make sure to have the **python3.6** version installed.
* (geo) A **c++ compiler** is required for the installation of the program **cartopy**
* (geo) Install **proj** and **GEOS** on your machine using the console.
* Install the **python3.6 develop** libraries.
* Install all **requirements** from [`requirements.txt`](https://gitlab.version.fz-juelich.de/toar/mlair/-/blob/master/requirements.txt)
preferably in a virtual environment
* (tf) Currently, TensorFlow-1.13 is mentioned in the requirements. We already tested the TensorFlow-1.15 version and couldn't
find any compatibility errors. Please note, that tf-1.13 and 1.15 have two distinct branches each, the default branch
for CPU support, and the "-gpu" branch for GPU support. If the GPU version is installed, MLAir will make use of the GPU
device.
* Installation of **MLAir**:
* Either clone MLAir from the [gitlab repository](https://gitlab.version.fz-juelich.de/toar/mlair.git)
and use it without installation (beside the requirements)
* or download the distribution file ([current version](https://gitlab.version.fz-juelich.de/toar/mlair/-/blob/master/dist/mlair-0.12.2-py3-none-any.whl))
and install it via `pip install <dist_file>.whl`. In this case, you can simply import MLAir in any python script
inside your virtual environment using `import mlair`.
* (tf) Currently, TensorFlow-1.13 is mentioned in the requirements. We already tested the TensorFlow-1.15 version and couldn't
find any compatibility errors. Please note, that tf-1.13 and 1.15 have two distinct branches each, the default branch
for CPU support, and the "-gpu" branch for GPU support. If the GPU version is installed, MLAir will make use of the GPU
device.
## openSUSE Leap 15.1
* c++ compiler
`sudo zypper install gcc-c++`
* geo packages
`sudo zypper install proj geos-devel`
* depending on the pre-installed packages it could be required to install further packages
`sudo zypper install libproj-devel binutils gdal-devel graphviz`
* python develop libraries
`sudo zypper install python3-devel`
## Ubuntu 20.04.1
* c++ compiler
`sudo apt install build-essential`
* geo packages
`sudo apt install proj-bin libgeos-dev libproj-dev`
* depending on the pre-installed packages it could be required to install further packages
`sudo apt install graphviz libgeos++-dev`
* python develop libraries
`sudo apt install python3.6-dev`
## macOS & windows
The installation on macOS is not tested yet. The following commands are possibly needed:
`brew install geos`
`sudo port install graphviz`
The installation on Windows is not tested yet.
# How to start with MLAir
......@@ -421,6 +469,20 @@ class DummyDataHandler(AbstractDataHandler):
# Special Remarks
## Workaround to skip geographical plot
If it is not possible to install all required geo libraries on your system, a good compromise is to skip the creation
of the geographical plot. Therefore, it is required to remove the plot from the `plot_list` manually. We recommend to
use this code snippet as a starting point.
```python
from mlair.helpers import remove_items
from mlair.configuration.defaults import DEFAULT_PLOT_LIST
mlair.run(plot_list=remove_items(DEFAULT_PLOT_LIST, "PlotStationMap"))
```
## Special instructions for installation on Jülich HPC systems
_Please note, that the HPC setup is customised for JUWELS and HDFML. When using another HPC system, you can use the HPC
......@@ -429,10 +491,10 @@ setup files as a skeleton and customise it to your needs._
The following instruction guide you through the installation on JUWELS and HDFML.
* Clone the repo to HPC system (we recommend to place it in `/p/projects/<project name>`).
* Setup venv by executing `source setupHPC.sh`. This script loads all pre-installed modules and creates a venv for
all other packages. Furthermore, it creates slurm/batch scripts to execute code on compute nodes. <br>
all other packages. Furthermore, it creates slurm/batch scripts to execute code on compute nodes.
You have to enter the HPC project's budget name (--account flag).
* The default external data path on JUWELS and HDFML is set to `/p/project/deepacf/intelliaq/<user>/DATA/toar_<sampling>`.
<br>To choose a different location open `run.py` and add the following keyword argument to `ExperimentSetup`:
To choose a different location open `run.py` and add the following keyword argument to `ExperimentSetup`:
`data_path=<your>/<custom>/<path>`.
* Execute `python run.py` on a login node to download example data. The program will throw an OSerror after downloading.
* Execute either `sbatch run_juwels_develgpus.bash` or `sbatch run_hdfml_batch.bash` to verify that the setup went well.
......
docs/_source/get-started.rst
+ 106
− 6
View file @ 161a8672
......@@ -15,15 +15,21 @@ packages built for your operating system. Name names of these package may differ
here to the opensuse / leap OS. The geo plot can be removed from the :py:`plot_list`, in this case there is no need to
install the geo packages.
MLAir is based on several python frameworks. To work properly, you have to install all packages from the
:py:`requirements.txt` file. Additionally to support the geographical plotting part it is required to install geo
packages built for your operating system. Unfortunately, the names of these package may differ for different systems.
In this instruction, we try to address users of different operating systems namely openSUSE Leap, Ubuntu and macOS.
If the installation is still not working, we recommend skipping the geographical plot. We have put together a small
workaround :ref:`Workaround to skip geographical plot`. For special instructions to install MLAir on the Juelich
HPC systems, see ref:`Installation on Jülich HPC systems`.
Pre-requirements
~~~~~~~~~~~~~~~~
* (geo) Install **proj** on your machine using the console. E.g. for opensuse / leap :py:`zypper install proj`
* (geo) A c++ compiler is required for the installation of the program **cartopy**
* (tf) Currently, TensorFlow-1.13 is mentioned in the requirements. We already tested the TensorFlow-1.15 version and couldn't
find any compatibility errors. Please note, that tf-1.13 and 1.15 have two distinct branches each, the default branch
for CPU support, and the "-gpu" branch for GPU support. If the GPU version is installed, MLAir will make use of the GPU
device.
* Make sure to have the **python3.6** version installed.
* (geo) A **c++ compiler** is required for the installation of the program **cartopy**
* (geo) Install **proj** and **GEOS** on your machine using the console.
* Install the **python3.6 develop** libraries.
Installation of MLAir
~~~~~~~~~~~~~~~~~~~~~
......@@ -34,6 +40,100 @@ Installation of MLAir
* or download the distribution file (`current version <https://gitlab.version.fz-juelich.de/toar/mlair/-/blob/master/dist/mlair-0.12.2-py3-none-any.whl>`_)
and install it via :py:`pip install <dist_file>.whl`. In this case, you can simply
import MLAir in any python script inside your virtual environment using :py:`import mlair`.
* (tf) Currently, TensorFlow-1.13 is mentioned in the requirements. We already tested the TensorFlow-1.15 version and couldn't
find any compatibility errors. Please note, that tf-1.13 and 1.15 have two distinct branches each, the default branch
for CPU support, and the "-gpu" branch for GPU support. If the GPU version is installed, MLAir will make use of the GPU
device.
Special Instructions for Installation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
openSUSE Leap 15.1
""""""""""""""""""
* c++ compiler
:py:`sudo zypper install gcc-c++`
* geo packages
:py:`sudo zypper install proj geos-devel`
* depending on the pre-installed packages it could be required to install further packages
:py:`sudo zypper install libproj-devel binutils gdal-devel graphviz`
* python develop libraries
:py:`sudo zypper install python3-devel`
Ubuntu 20.04.1
""""""""""""""
* c++ compiler
:py:`sudo apt install build-essential`
* geo packages
:py:`sudo apt install proj-bin libgeos-dev libproj-dev`
* depending on the pre-installed packages it could be required to install further packages
:py:`sudo apt install graphviz libgeos++-dev`
* python develop libraries
:py:`sudo apt install python3.6-dev`
macOS & windows
"""""""""""""""
The installation on macOS is not tested yet. The following commands are possibly needed:
:py:`brew install geos`
:py:`sudo port install graphviz`
The installation on Windows is not tested yet.
Installation on Jülich HPC systems
""""""""""""""""""""""""""""""""""
*Please note, that the HPC setup is customised for JUWELS and HDFML. When using another HPC system, you can use the HPC
setup files as a skeleton and customise it to your needs.*
The following instruction guide you through the installation on JUWELS and HDFML.
* Clone the repo to HPC system (we recommend to place it in :py:`/p/projects/<project name>`).
* Setup venv by executing :py:`source setupHPC.sh`. This script loads all pre-installed modules and creates a venv for
all other packages. Furthermore, it creates slurm/batch scripts to execute code on compute nodes.
You have to enter the HPC project's budget name (--account flag).
* The default external data path on JUWELS and HDFML is set to :py:`/p/project/deepacf/intelliaq/<user>/DATA/toar_<sampling>`.
* To choose a different location open :py:`run.py` and add the following keyword argument to :py:`ExperimentSetup`:
:py:`data_path=<your>/<custom>/<path>`.
* Execute :py:`python run.py` on a login node to download example data. The program will throw an OSerror after downloading.
* Execute either :py:`sbatch run_juwels_develgpus.bash` or :py:`sbatch run_hdfml_batch.bash` to verify that the setup
went well.
* Currently cartopy is not working on our HPC system, therefore PlotStations does not create any output.
Note: The method :py:`PartitionCheck` currently only checks if the hostname starts with :py:`ju` or :py:`hdfmll`.
Therefore, it might be necessary to adopt the :py:`if` statement in :py:`PartitionCheck._run`.
Workaround to skip geographical plot
""""""""""""""""""""""""""""""""""""
If it is not possible to install all required geo libraries on your system, a good compromise is to skip the creation
of the geographical plot. Therefore, it is required to remove the plot from the :py:`plot_list` manually. We recommend
to use this code snippet as a starting point.
.. code-block:: python
from mlair.helpers import remove_items
from mlair.configuration.defaults import DEFAULT_PLOT_LIST
mlair.run(plot_list=remove_items(DEFAULT_PLOT_LIST, "PlotStationMap"))
How to start with MLAir
......
docs/_source/index.rst
+ 1
− 0
View file @ 161a8672
......@@ -6,6 +6,7 @@
Welcome to MLAir's documentation!
================================================
This is the documation of the `MLAir package <https://gitlab.version.fz-juelich.de/toar/mlair>`_.
.. toctree::
:maxdepth: 2
......
requirements.txt
+ 1
− 1
View file @ 161a8672
......@@ -65,5 +65,5 @@ xarray==0.15.0
zipp==3.1.0
setuptools~=49.6.0
Cartopy==0.17.0
Cartopy==0.18.0
--no-binary shapely Shapely==1.7.0
\ No newline at end of file
requirements_gpu.txt
+ 1
− 1
View file @ 161a8672
......@@ -64,5 +64,5 @@ Werkzeug==1.0.0
xarray==0.15.0
zipp==3.1.0
#Cartopy==0.17.0
#Cartopy==0.18.0
--no-binary shapely Shapely==1.7.0
\ No newline at end of file
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment