Skip to content
Snippets Groups Projects
Select Git revision
  • 7398cd7aa8be2d83a6d45f970c84ce3fb5aef7a3
  • bing_issues#190_tf2
  • bing_tf2_convert
  • bing_issue#189_train_modular
  • simon_#172_integrate_weatherbench
  • develop
  • bing_issue#188_restructure_ambs
  • yan_issue#100_extract_prcp_data
  • bing_issue#170_data_preprocess_training_tf1
  • Gong2022_temperature_forecasts
  • bing_issue#186_clean_GMD1_tag
  • yan_issue#179_integrate_GZAWS_data_onfly
  • bing_issue#178_runscript_bug_postprocess
  • michael_issue#187_bugfix_setup_runscript_template
  • bing_issue#180_bugs_postprpocess_meta_postprocess
  • yan_issue#177_repo_for_CLGAN_gmd
  • bing_issue#176_integrate_weather_bench
  • michael_issue#181_eval_era5_forecasts
  • michael_issue#182_eval_subdomain
  • michael_issue#119_warmup_Horovod
  • bing_issue#160_test_zam347
  • ambs_v1
  • ambs_gmd_nowcasting_v1.0
  • GMD1
  • modular_booster_20210203
  • new_structure_20201004_v1.0
  • old_structure_20200930
27 results

ambs

user avatar
gong1 authored
Merge branch 'bing_issue#009_clean_up_postprocessing' of ssh://gitlab.version.fz-juelich.de:10022/toar/ambs into bing_issue#009_clean_up_postprocessing
7398cd7a
History

AMBS

Atmopsheric Machine learning Benchmarking Systems (AMBS) aims to privde state-of-the-art benchmarking machine learning architectures for video prediction on HPC in the context of atmospheric domain, which is developed by Amirpasha, Michael, Bing, and Scarlet

Prerequisites

  • Linux or macOS
  • Python 3
  • CPU or NVIDIA GPU + CUDA CuDNN
  • MPI
  • Tensorflow 1.13.1

Installation

  • Clone this repo:
git clone https://gitlab.version.fz-juelich.de/toar/ambs.git

Set-up env on JUWELS and ZAM347

  • Setup env and install packages
cd video_prediction_savp/env_setup
source create_env_zam347.sh <env_name>

Run workflow on JUWELS

  • Go to HPC_scripts directory
cd video_prediction_savp/HPC_scripts
  • Data Extraction
sbatch DataExtraction.sh
  • Data Preprocessing
sbatch /DataPreprocess.sh
sbatch /DataPreprocess_to_tf.sh
  • Setup hyperparams

This step will setup the hyper-parameters that used for training, and create a folder named "datetime_user" where save the trained model

source hyperparam_setup.sh
  • Training
sbatch train_era5.sh
  • Postprocess
sbatch generate_era5.sh
  • Reset all the generated path to origin state
source reset_dirs.sh

Run workflow on ZAM347

  • Go to zam347_scripts directory
cd video_prediction_savp/Zam347_scripts
  • Data Extraction
./DataExtraction.sh
  • Data Preprocessing
./DataPreprocess.sh
./DataPreprocess_to_tf.sh
  • Training
./train_era5.sh
  • Postprocess
./generate_era5.sh

Recomendation for output folder structure and name convention

The details can be found name_convention

├── ExtractedData
│   ├── [Year]
│   │   ├── [Month]
│   │   │   ├── **/*.netCDF
├── PreprocessedData
│   ├── [Data_name_convention]
│   │   ├── hickle
│   │   │   ├── train
│   │   │   ├── val
│   │   │   ├── test
│   │   ├── tfrecords
│   │   │   ├── train
│   │   │   ├── val
│   │   │   ├── test
├── Models
│   ├── [Data_name_convention]
│   │   ├── [model_name]
│   │   ├── [model_name]
├── Results
│   ├── [Data_name_convention]
│   │   ├── [training_mode]
│   │   │   ├── [source_data_name_convention]
│   │   │   │   ├── [model_name]

Benchmarking architectures:

Contact