AMBS

Atmopsheric Machine learning Benchmarking Systems (AMBS) aims to privde state-of-the-art benchmarking machine learning architectures for video prediction on HPC in the context of atmospheric domain, which is developed by Amirpasha, Michael, Bing, and Scarlet

Prerequisites

• Linux or macOS
• Python 3
• CPU or NVIDIA GPU + CUDA CuDNN
• MPI
• Tensorflow 1.13.1

Installation

• Clone this repo:

git clone https://gitlab.version.fz-juelich.de/toar/ambs.git

Set-up env on JUWELS and ZAM347

• Set up env and install packages on JUWELS

cd video_prediction_savp/env_setup
source create_env.sh <dir_name> <env_name>

• Setup env and install packages on zam347

cd video_prediction_savp/env_setup
source create_env_zam347.sh <env_name>

Run workflow on ZAM347

• Go to zam347_scripts directory

cd video_prediction_savp/Zam347_scripts

• Data Extraction

./DataExtraction.sh

• Data Preprocessing

./DataPreprocess.sh
./DataPreprocess_to_tf.sh

• Training

./train_era5.sh

• Postprocess

./generate_era5.sh

Recomendation for output folder structure and name convention

The details can be found name_convention

├── ExtractedData
│   ├── [Year]
│   │   ├── [Month]
│   │   │   ├── **/*.netCDF
├── PreprocessedData
│   ├── [Data_name_convention]
│   │   ├── hickle
│   │   │   ├── train
│   │   │   ├── val
│   │   │   ├── test
│   │   ├── tfrecords
│   │   │   ├── train
│   │   │   ├── val
│   │   │   ├── test
├── Models
│   ├── [Data_name_convention]
│   │   ├── [model_name]
│   │   ├── [model_name]
├── Results
│   ├── [Data_name_convention]
│   │   ├── [training_mode]
│   │   │   ├── [source_data_name_convention]
│   │   │   │   ├── [model_name]

Benchmarking architectures:

• convLSTM: paper,code
• Variational Autoencoder:paper
• Stochastic Adversarial Video Prediction (SAVP): paper,code
• Motion and Content Network (MCnet): paper, code

Contact

• Amirpash Mozafarri: a.mozafarri@fz-juelich.de
• Michael Langguth: m.langguth@fz-juelich.de
• Bing Gong: b.gong@fz-juelich.de
• Scarlet Stadtler: s.stadtler@fz-juelich.de