AMBS
Atmopsheric Machine learning Benchmarking Systems (AMBS) aims to privde state-of-the-art benchmarking machine learning architectures for video prediction on HPC in the context of atmospheric domain, which is developed by Amirpasha, Michael, Bing, and Scarlet
Prerequisites
- Linux or macOS
- Python 3
- CPU or NVIDIA GPU + CUDA CuDNN
- MPI
- Tensorflow 1.13.1
Installation
- Clone this repo:
git clone https://gitlab.version.fz-juelich.de/toar/ambs.git
Set-up env on JUWELS and ZAM347
- Set up env and install packages on JUWELS
cd video_prediction_savp/env_setup
source create_env.sh <dir_name> <env_name>
- Setup env and install packages on zam347
cd video_prediction_savp/env_setup
source create_env_zam347.sh <env_name>
Run workflow on ZAM347
- Go to zam347_scripts directory
cd video_prediction_savp/Zam347_scripts
- Data Extraction
./DataExtraction.sh
- Data Preprocessing
./DataPreprocess.sh
./DataPreprocess_to_tf.sh
- Training
./train_era5.sh
- Postprocess
./generate_era5.sh
Recomendation for output folder structure and name convention
The details can be found name_convention
├── ExtractedData
│ ├── [Year]
│ │ ├── [Month]
│ │ │ ├── **/*.netCDF
├── PreprocessedData
│ ├── [Data_name_convention]
│ │ ├── hickle
│ │ │ ├── train
│ │ │ ├── val
│ │ │ ├── test
│ │ ├── tfrecords
│ │ │ ├── train
│ │ │ ├── val
│ │ │ ├── test
├── Models
│ ├── [Data_name_convention]
│ │ ├── [model_name]
│ │ ├── [model_name]
├── Results
│ ├── [Data_name_convention]
│ │ ├── [training_mode]
│ │ │ ├── [source_data_name_convention]
│ │ │ │ ├── [model_name]
Benchmarking architectures:
- convLSTM: paper,code
- Variational Autoencoder:paper
- Stochastic Adversarial Video Prediction (SAVP): paper,code
- Motion and Content Network (MCnet): paper, code
Contact
- Amirpash Mozafarri: a.mozafarri@fz-juelich.de
- Michael Langguth: m.langguth@fz-juelich.de
- Bing Gong: b.gong@fz-juelich.de
- Scarlet Stadtler: s.stadtler@fz-juelich.de