Skip to content
Snippets Groups Projects
Select Git revision
  • bing_issues#190_tf2
  • bing_tf2_convert
  • bing_issue#189_train_modular
  • simon_#172_integrate_weatherbench
  • develop
  • bing_issue#188_restructure_ambs
  • yan_issue#100_extract_prcp_data
  • bing_issue#170_data_preprocess_training_tf1
  • Gong2022_temperature_forecasts
  • bing_issue#186_clean_GMD1_tag
  • yan_issue#179_integrate_GZAWS_data_onfly
  • bing_issue#178_runscript_bug_postprocess
  • michael_issue#187_bugfix_setup_runscript_template
  • bing_issue#180_bugs_postprpocess_meta_postprocess
  • yan_issue#177_repo_for_CLGAN_gmd
  • bing_issue#176_integrate_weather_bench
  • michael_issue#181_eval_era5_forecasts
  • michael_issue#182_eval_subdomain
  • michael_issue#119_warmup_Horovod
  • bing_issue#160_test_zam347
  • ambs_v1
  • ambs_gmd_nowcasting_v1.0
  • GMD1
  • modular_booster_20210203
  • new_structure_20201004_v1.0
  • old_structure_20200930
26 results

ambs

  • Clone with SSH
  • Clone with HTTPS
  • user avatar
    Bing Gong authored
    613dbb73
    History

    AMBS

    Atmopsheric Machine learning Benchmarking Systems (AMBS) aims to privde state-of-the-art benchmarking machine learning architectures for video prediction on HPC in the context of atmospheric domain, which is developed by Amirpasha, Michael, Bing, and Scarlet

    Prerequisites

    • Linux or macOS
    • Python 3
    • CPU or NVIDIA GPU + CUDA CuDNN
    • MPI
    • Tensorflow 1.13.1

    Installation

    • Clone this repo:
    git clone https://gitlab.version.fz-juelich.de/toar/ambs.git

    Set-up env on JUWELS and ZAM347

    • Set up env and install packages on JUWELS
    cd video_prediction_savp/env_setup
    source create_env.sh <dir_name> <env_name>
    • Setup env and install packages on zam347
    cd video_prediction_savp/env_setup
    source create_env_zam347.sh <env_name>

    Run workflow on ZAM347

    • Go to zam347_scripts directory
    cd video_prediction_savp/Zam347_scripts
    • Data Extraction
    ./DataExtraction.sh
    • Data Preprocessing
    ./DataPreprocess.sh
    ./DataPreprocess_to_tf.sh
    • Training
    ./train_era5.sh
    • Postprocess
    ./generate_era5.sh

    Recomendation for output folder structure and name convention

    The details can be found name_convention

    ├── ExtractedData
    │   ├── [Year]
    │   │   ├── [Month]
    │   │   │   ├── **/*.netCDF
    ├── PreprocessedData
    │   ├── [Data_name_convention]
    │   │   ├── hickle
    │   │   │   ├── train
    │   │   │   ├── val
    │   │   │   ├── test
    │   │   ├── tfrecords
    │   │   │   ├── train
    │   │   │   ├── val
    │   │   │   ├── test
    ├── Models
    │   ├── [Data_name_convention]
    │   │   ├── [model_name]
    │   │   ├── [model_name]
    ├── Results
    │   ├── [Data_name_convention]
    │   │   ├── [training_mode]
    │   │   │   ├── [source_data_name_convention]
    │   │   │   │   ├── [model_name]
    

    Benchmarking architectures:

    Contact