Skip to content
Snippets Groups Projects
Commit 816ba446 authored by gong1's avatar gong1
Browse files

Merge branch 'develop' into bing_issue#155_Non_JSC_HPC_scripts_prepration

parents cf736df2 37ccafc5
Branches
Tags
No related merge requests found
Show whitespace changes
Inline Side-by-side
README.md
+ 200
− 140
View file @ 816ba446
# AMBS
**A**tmopsheric **M**achine learning **B**enchmarking **S**ystem (AMBS)
aims to provide state-of-the-art video prediction methods applied to the meteorological domain.
In the scope of the current application, the hourly evolution of the 2m temperature
over a used-defined region is focused. <br>
Different Deep Learning video prediction architectures such as convLSTM and SAVP
are trained with ERA5 reanalysis to perform a prediction for 12 hours based on the previous 12 hours.
In addition to the 2m temperature (t2m) itself, other variables can be fed to the video frame prediction models to enhance their capability to learn the complex physical processes driving the diurnal cycle of temperature.
Currently, the recommended additional meteorological variables are the 850 hPa temperature (t850) and the total cloud cover (tcc).
Besides, training on other standard video frame prediction datasets (such as MovingMNIST) can be prerformed.
The project is currently developed by Bing Gong, Michael Langguth, Amirpasha Mozafarri, Yan Ji and Karim Mache.<br>
Former code developers are Scarlet Stadtler and Severin Hussmann.
<img src="ambs_logo.jpg" width="1000" height="400">
[Check our video](https://www.youtube.com/watch?v=Tf2BDDlSDeQ)
## Table of Contents
- [Introduction to Atmopsheric Machine learning Benchmarking System](#introduction-to-atmopsheric-machine-learning-benchmarking-system)
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Start with AMBS](#start-with-ambs)
* [Set-up virtual environment](#set-up-virtual-environment)
+ [On Jülich's HPC systems](#on-j-lich-s-hpc-systems)
+ [On other HPC systems](#on-other-hpc-systems)
* [Prepare your dataset](#prepare-your-dataset)
+ [Dry run with small samples (~15 GB)](#dry-run-with-small-samples---15-gb-)
+ [Access ERA5 dataset (~TB)](#access-era5-dataset---tb-)
+ [Climatological mean data](#climatological-mean-data)
* [Run the workflow](#run-the-workflow)
* [Preparation with NVIDIA's TF1.15 singularity containers](#preparation-with-nvidia-s-tf115-singularity-containers)
* [Create specific runscripts](#create-specific-runscripts)
* [Running the workflow substeps](#running-the-workflow-substeps)
* [Compare and visualize the results](#compare-and-visualize-the-results)
* [Input and Output folder structure and naming convention](#input-and-output-folder-structure-and-naming-convention)
- [Benchmarking architectures:](#benchmarking-architectures-)
- [Contributors and contact](#contributors-and-contact)
- [On-going work](#on-going-work)
## Introduction to Atmopsheric Machine learning Benchmarking System
### Prerequisites
**A**tmopsheric **M**achine learning **B**enchmarking **S**ystem (AMBS) aims to provide state-of-the-art video prediction methods applied to the meteorological domain. In the scope of the current application, the hourly evolution of the 2m temperature over a used-defined region is focused.
Different Deep Learning video prediction architectures such as convLSTM and SAVP are trained with ERA5 reanalysis to perform a prediction for 12 hours based on the previous 12 hours. In addition to the 2m temperature (t2m) itself, other variables can be fed to the video frame prediction models to enhance their capability to learn the complex physical processes driving the diurnal cycle of temperature. Currently, the recommended additional meteorological variables are the 850 hPa temperature (t850) and the total cloud cover (tcc) as described in our preprint GMD paper.
## Prerequisites
- Linux or macOS
- Python 3
- CPU or NVIDIA GPU + CUDA CuDNN
......@@ -22,186 +42,180 @@ Former code developers are Scarlet Stadtler and Severin Hussmann.
- Tensorflow 1.13.1 or CUDA-enabled NVIDIA TensorFlow 1.15 within a singularity container
- CDO >= 1.9.5
### Installation
## Installation
Clone this repo by typing the following command in your personal target dirctory:
```bash
git clone https://gitlab.jsc.fz-juelich.de/esde/machine-learning/ambs.git
```
This will create a directory called `ambs` under which this README-file and
two subdirectories are placed. The subdirectory `[...]/ambs/test/` contains unittest-scripts for
the workflow and is therefore of minor relevance for non-developers.
The subdirectory `[...]/ambs/video_prediction_tools` contains everything which is needed in the workflow and is
therefore called the top-level directory in the following.
Since the project is continuously developed and make the experiments described in the GMD paper reproducible, we also provide a frozen version:
```bash
git clone https://gitlab.jsc.fz-juelich.de/esde/machine-learning/ambs_gmd1.git
```
This will create a directory called `ambs` under which this README-file and two subdirectories are placed. The subdirectory `[...]/ambs/test/` contains unittest-scripts for the workflow and is therefore of minor relevance for non-developers. The subdirectory `[...]/ambs/video_prediction_tools` contains everything which is needed in the workflow and is, therefore, called the top-level directory in the following.
Thus, change into this subdirectory after cloning:
```bash
cd ambs/video_preditcion_tools/
```
### Set-up environment on Jülich's HPC systems or other computing systems
The following commands will setup a customized virtual environment
either on a known HPC-system at JSC (Juwels, Juwels Booster or HDF-ML). Setting up a virtual environment on other computing systems, e.g. the personal computer, is currently not supported, but targeted for the future.
The script `create_env.sh` automatically detects on which machine it is executed and loads/installs
all required Python (binary) modules and packages.
The virtual environment is set up in the top-level directory (`[...]/ambs/video_prediction_tools`)
under a subdirectory which gets the name of the virtual environment.
## Start with AMBS
### Set-up virtual environment
AMBS is the tool for HPC systems. Other computing systems sucha s personal computer is not supported. Currently, we provide two approaches to set up your virtual envionrment based on either Jülich HPC system or other HPC systems. The introduction is described as below.
#### On Jülich's HPC systems
The following commands will setup a customized virtual environment on a known HPC-system at JSC (Juwels, Juwels Booster or HDF-ML). The script `create_env.sh` automatically detects on which machine it is executed and loads/installs all required Python (binary) modules and packages. The virtual environment with the name provide by user is then set up in a subdirectory `[...]/ambs/video_prediction_tools/virtual_envs/<env_name>` the top-level directory (`[...]/ambs/video_prediction_tools`).
```bash
cd env_setup
source create_env.sh <env_name>
```
This also already sets up the runscript templates for you. By default, the runscript templates make use of the standard target base directory `/p/project/deepacf/deeprain/video_prediction_shared_folder/`. In case that you want to deviate from this, you may call `create_env.sh` as follows:
This also already sets up the runscript templates with regards to the five steps of the workflow for you under the folder `[...]/ambs/video_prediction_tools/HPC_scripts`.
By default, the runscript templates make use of the standard target base directory `/p/project/deepacf/deeprain/video_prediction_shared_folder/`. This directory will serve as your standard top-level direcotry to store the output of each step in the workflow (see details in #Input-and-output-folder-tructure-and-naming-convention). In case that you want to deviate from this, you may call `create_env.sh` as follows:
```bash
source create_env.sh <env_name> -base_dir=<my_target_dir>
```
Note that suifficient read-write permissions and a reasonable amount of memory space is mandatory for your alternative standard output directory.
**Note** that suifficient read-write permissions and a reasonable amount of memory space is mandatory for your alternative standard output directory.
#### On other HPC systems
If you are working on other HPC system, you are required to custermise the templates under the folder `nonJSC_HPC_scripts`. You can then replace the templates to the `HPC_scripts`.
### Prepare your dataset
#### Dry run with small samples (~15 GB)
In weatheer and Climate application, we are also dealing with the large dataset. However, we prepare rather small samples (3 months data with few variables) to help the users test the workflow.
- For the users of JSC HPC system: The data can be downloaded through the following link [link!!] .
- For the users of deepacf project: You can also access `cd /p/scratch/deepacf/deeprain/ji4/GMD_data_example`
#### Access ERA5 dataset (~TB)
The experiment described in the GMD paper relies on the rather large ERA5 dataset with 13 years data.
- For the users of JSC HPC system: You access the data from the followin path: /p/fastdata/slmet/slmet111/met_data/ecmwf/era5/grib. If you meet access permission issue please contact: Stein, Olaf <o.stein@fz-juelich.de>
- For the users of other HPC sytems: You can retrieve the ERA5 data from the ECMWF MARS archive by specifying a resolution of 0.3° in the retrieval script (keyword "GRID", "https://confluence.ecmwf.int/pages/viewpage.action?pageId=123799065 "). The variable names and the corresponding paramID can be found in the ECMWF documentaation website [ERA5 documentations](https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Howtoacknowledge,citeandrefertoERA5)
#### Climatological mean data
climatological mean which is inferred at each grid point from the ERA5 reanalysis data between 1990 and 2019 is used in the postprocess step. The data can be download [LINK!!]
### Run the workflow
Depending on the computing system you are working on, the workflow steps will be invoked
by dedicated runscripts either from the directory `HPC_scripts/` (on known HPC-systems, see above) or from
the directory `nonHPC_scripts/` (else, but not implemented yet).<br>
Each runscript can be set up conveniently with the help of the Python-script `generate_runscript.py`.
Its usage as well the workflow runscripts are described subsequently.
Depending on the computing system you are working on, the workflow steps will be invoked by dedicated runscripts either from the directory `HPC_scripts/` (on known HPC-systems, see above) or from the directory `JSC_scripts/`.
#### Preparation
To help the users conduct different experiments with different configuration (e.g. input variables, hyperparameters etc). Each runscript can be set up conveniently with the help of the Python-script `generate_runscript.py`. Its usage as well the workflow runscripts are described subsequently.
In case that a virtual environment has already been set up beforehand, the script `create_env.sh` can also be used for activating it. For this, simply do
```bash
cd env_setup
source create_env.sh <env_name>
```
similarly to the creation process of the virtual environment (see above). Note that this also loads some modules on the HPC-systems that are required for the runscript generator (see below). <br>
**Remark:** In case you want to change the target base directory _after_ having created the virtual environment,
you may run `setup_runscript_templates.sh` manually (the `script create_env.sh` does this internally):
```bash
source <top_level_dir>/utils/runscript_generator/setup_runscript_templates.sh <my_target_dir>
```
### Preparation with NVIDIA's TF1.15 singularity containers
Since 2022, JSC HPC does not support TF1.X in the current stack software system. As an intermediate solution before the TF2 version being ready,
a singularity container with a CUDA-enabled NVIDIA TensorFlow v1.15 was made available which has to be reflected when setting up the virtual environment and when submiiting the job.
Then, you can either download container image ([Link](https://docs.nvidia.com/deeplearning/frameworks/tensorflow-release-notes/rel_21-09.html#rel_21-09)) and place it under the folder`HPC_script`; Or you can access to the image though the symlink command as below, if you are part of the *deepacf*project (still link to the `HPC_scripts`-directory)
` ln -sf /p/project/deepacf/deeprain/video_prediction_shared_folder/containers_juwels_booster/nvidia_tensorflow_21.09-tf1-py3.sif`
Note that if you are the user of JSC HPC system, you need to log in [Judoor account] (https://judoor.fz-juelich.de/login) and specifically ask for the request to access to the restricted container software.
#### Create specific runscripts
### Create specific runscripts
Specific runscripts for each workfow substep (see below) are generated conventiently by keyboard interaction.
The interactive Python-script thereby has to be executed in an activated virtual environment with some addiional modules!
After prompting
The interactive Python-script thereby has to be executed in an activated virtual environment with some addiional modules! After prompting
```bash
python generate_runscript.py
```
you will be asked first which workflow runscript shall be generated. You can chose one of the workflow step name:
You will be asked first which workflow runscript shall be generated. You can chose one of the workflow step name:
- extract
- preprocess1
- preprocess2
- train
- postprocess
The subsequent keyboard interactions then allow the user to make individual settings to the workflow step at hand.
By pressing simply Enter, the user may receive some guidance for the keyboard interaction. <br>
The subsequent keyboard interactions then allow the user to make individual settings to the workflow step at hand. By pressing simply Enter, the user may receive some guidance for the keyboard interaction.
Note that the runscript creation of later workflow substeps depends on the preceding steps (i.e. by checking the arguments from keyboard interaction).
Thus, they should be created sequentially instead of all at once at the beginning.
#### Running the workflow substeps
Having created the runscript by keyboard interaction, the workflow substeps can be run sequentially.
Depending on the machine you are working on, change either to `HPC_scripts/` (on Juwels, Juwels Booster or HDF-ML) or to
`nonHPC347_scripts/` (not implemented yet).
### Running the workflow substeps
Having created the runscript by keyboard interaction, the workflow substeps can be run sequentially. Depending on the machine you are working on, change either to `HPC_scripts/` (on Juwels, Juwels Booster or HDF-ML).
There, the respective runscripts for all steps of the workflow are located
whose order is as follows. Note that `[sbatch]` only has to precede on one of the HPC systems.
Besides data extraction and preprocessing step 1 are onyl mandatory when ERA5 data is subject to the application.
whose order is as follows. Note that `[sbatch]` only has to precede on one of the HPC systems. Besides data extraction and preprocessing step 1 are onyl mandatory when ERA5 data is subject to the application.
Note we provide default configurations for each runscripts
that the users still need to manully configure flags based on which project and HPC systems you are currently working on. Particurly, you must replace the flag `#SBATCH --account =<your computing project name>` with your project name. For partitions `#SBATCH --partition=`, we refer the users to the following link [JUWELS/JUWELS Booster](https://apps.fz-juelich.de/jsc/hps/juwels/batchsystem.html#slurm-partitions) for further information. If you are using HDF-MLsystem, you can simply use `batch` as partition.
Now it is time to run the AMBS workflow
1. Data Extraction: This script retrieves the demanded variables for user-defined years from complete ERA% reanalysis grib-files and stors the data into netCDF-files.
```bash
[sbatch] ./data_extraction_era5.sh
```
2. Data Preprocessing: Crop the ERA 5-data (multiple years possible) to the region of interest (preprocesing step 1),
The TFrecord-files which are fed to the trained model (next workflow step) are created afterwards.
This is also the place where other datasets such as the MovingMNIST (link?) can be prepared.
Thus, two cases exist at this stage:
* **ERA 5 data**
The TFrecord-files which are fed to the trained model (next workflow step) are created afterwards. Thus, two cases exist at this stage:
```bash
[sbatch] ./preprocess_data_era5_step1.sh
[sbatch] ./preprocess_data_era5_step2.sh
```
* **MovingMNIST data**
```bash
[sbatch] ./preprocess_data_moving_mnist.sh
```
3. Training: Training of one of the available models with the preprocessed data. <br>
3. Training: Training of one of the available models with the preprocessed data.
Note that the `exp_id` is generated automatically when running `generate_runscript.py`.
* **ERA 5 data**
```bash
[sbatch] ./train_model_era5_<exp_id>.sh
```
* **MovingMNIST data**
```bash
[sbatch] ./train_model_moving_mnist_<exp_id>.sh
```
4. Postprocess: Create some plots and calculate the evaluation metrics for test dataset. <br>
Note that the `exp_id` is generated automatically when running `generate_runscript.py`.
* **ERA 5 data**
```bash
[sbatch] ./visualize_postprocess_era5_<exp_id>.sh
```
* **MovingMNIST data**
```bash
[sbatch] ./visualize_postprocess_moving_mnist_<exp_id>.sh
```
### Running the training in NVIDIA's TF1.15 singularity containers ###
**The following documentation is deprecated and must be updated.** <br><br>
The computionally expensive training of the Deep Learning video prediction architectures is supposed to benefit from the Booster module which is installed at JSC in autumn 2020. In order to test the potential speed-up on this state-of-the-art HPC system, optimized for massively parallel workloads, we selected the convLSTM-architecture as a test candidate in the scope of the Juwels Booster Early Access program.
To run the training on the Booster, change to the recent Booster related working branch of this git repository.
```bash
git checkout scarlet_issue#031_booster
[sbatch] ./visualize_postprocess_era5_<exp_id>.sh
```
Currently, there is no system installation of TensorFlow available on Juwels Booster. As an intermediate solution,
a singularity container with a CUDA-enabled NVIDIA TensorFlow v1.15 was made available which has to be reflected when setting up the virtual environment and when submiiting the job.
The corresponding singularity image file is shared under `/p/project/deepacf/deeprain/video_prediction_shared_folder/containers_juwels_booster/` and needs to be linked to the `HPC_scripts`-directory. If you are not part of the *deepacf*-project, the singularity file can be accessed alternatively from
### Compare and visualize the results
```bash
cd video_prediction_tools/HPC_scripts
ln -sf /p/project/deepacf/deeprain/video_prediction_shared_folder/containers_juwels_booster/tf-1.15-ofed-venv.sif tf-1.15-ofed-venv.sif
```
AMBS also provide the tool (called met_postprocess) for the users to compare different experiments results and visualize the results as shown in GMD paper through
`meta_postprocess` step. The runscript template are also prepared in the `HPC_scripts`.
Before setting up the virtual environment for the workflow, some required modules have to be loaded and an interactive shell has to be spawn within the container.
```bash
cd ../env_setup
source modules_train.sh
cd ../HPC_scripts/
singularity shell tf-1.15-ofed-venv.sif
```
### Input and Output folder structure and naming convention
To succesfully runt the workflow and enable to track the result from each step, inputs and output directories, and the file name convention should be constructed as below:
Now, the virtual environment can be created as usual, i.e. by doing the following steps within the container shell:
The example of inputs structure for ERA5 dataset. In detail, the data is recoredly hourly and stored into two grib files. The file with postfix `*_ml.grb` consists of multi layers of the variables, whereas `_sf.grb` only include the surface data.
```bash
cd ../env_setup
source create_env.sh <env_name> [<exp_id>]
```
├── ERA5 dataset
│ ├── [Year]
│ │ ├── [Month]
│ │ │ ├── *_ml.grb
│ │ │ ├── *_sf.grb
│ │ │ ├── ...
│ │ ├── [Month]
│ │ │ ├── *_ml.grb
│ │ │ ├── *_sf.grb
│ │ │ ├── ...
This creates a corresponding directory `video_prediction_tools/<env_name>`.
Note that `create_env.sh` is *system-aware* and therefore only creates an executable runscript for the training step,
e.g. `video_prediction_tools/HPC_scripts/train_model_era5_booster_exp1.sh`.
Since all other substeps of the workflow are supposed to be performed on other systems so far, the data in-and output directories cannot be determined automatically during the workflow.
Thus, the user has to modify the corresponding runscript by adapting manually the script varibales `source_dir` and `destination_dir` in line 19+20 of `train_model_era5_booster_<exp_id>.sh`.
Finally, the training job can be submitted to Juwels Booster after exiting from the cotainer instance by
``` bash
exit
cd ../HPC_scripts
salloc --partition=booster --nodes=1 --account=ea_deepacf --time=00:10:00 --reservation earlyaccess srun singularity exec --nv tf-1.15-ofed-venv.sif ./train_model_era5_booster_<exp_id>.sh
```
### Output folder structure and naming convention
The details can be found [name_convention](docs/structure_name_convention.md)
The root output directory should be set up when you run the workflow at the first time as aformentioned
The output strucutre for each step of the workflow along with the file name convention are described below:
```
├── ExtractedData
│ ├── [Year]
......@@ -210,37 +224,83 @@ The details can be found [name_convention](docs/structure_name_convention.md)
├── PreprocessedData
│ ├── [Data_name_convention]
│ │ ├── pickle
│ │ │ ├── train
│ │ │ ├── val
│ │ │ ├── test
│ │ │ ├── X_<Month>.pkl
│ │ │ ├── T_<Month>.pkl
│ │ │ ├── stat_<Month>.pkl
│ │ ├── tfrecords
│ │ │ ├── train
│ │ │ ├── val
│ │ │ ├── test
│ │ │ ├── sequence_Y_<Year>_M_<Month>.tfrecords
│ │ │── metadata.json
├── Models
│ ├── [Data_name_convention]
│ │ ├── [model_name]
│ │ ├── [model_name]
│ │ │ ├── <timestamp>_<user>_<exp_id>
│ │ │ │ ├── checkpoint_<iteration>
│ │ │ │ │ ├── model_*
│ │ │ │ │── timing_per_iteration_time.pkl
│ │ │ │ │── timing_total_time.pkl
│ │ │ │ │── timing_training_time.pkl
│ │ │ │ │── train_losses.pkl
│ │ │ │ │── val_losses.pkl
│ │ │ │ │── *.json
├── Results
│ ├── [Data_name_convention]
│ │ ├── [training_mode]
│ │ │ ├── [source_data_name_convention]
│ │ │ │ ├── [model_name]
│ │ │ │ │ ├── *.nc
├── meta_postprocoess
│ ├── [experiment ID]
```
### Benchmarking architectures:
| Arguments | Value |
|--- |--- |
| [Year] | 2005;2006;2007,...,2019|
| [Month] | 01;02;03 ...,12|
|[Data_name_convention]|Y[yyyy]to[yyyy]M[mm]to[mm]-[nx]_[ny]-[nn.nn]N[ee.ee]E-[var1]_[var2]_[var3]|
|[model_name]| convLSTM, savp, ...|
***Data name convention***
`Y[yyyy]to[yyyy]M[mm]to[mm]-[nx]_[ny]-[nn.nn]N[ee.ee]E-[var1]_[var2]_[var3]`
- Y[yyyy]to[yyyy]M[mm]to[mm]
- [nx]_[ny]: the size of images,e.g 64_64 means 64*64 pixels
- [nn.nn]N[ee.ee]E: the geolocation of selected regions with two decimal points. e.g : 0.00N11.50E
- [var1]_[var2]_[var3]: the abbrevation of selected variables
| Examples | Name abbrevation |
|--- |--- |
|all data from March to June of the years 2005-2015 | Y2005toY2015M03to06 |
|data from February to May of years 2005-2008 + data from March to June of year 2015| Y2005to2008M02to05_Y2015M03to06 |
|Data from February to May, and October to December of 2005 | Y2005M02to05_Y2015M10to12 |
|operational’ data base: whole year 2016 | Y2016M01to12 |
|add new whole year data of 2017 on the operational data base |Y2016to2017M01to12 |
|Note: Y2016to2017M01to12 = Y2016M01to12_Y2017M01to12|
## Benchmarking architectures:
- convLSTM: [paper](https://papers.nips.cc/paper/5955-convolutional-lstm-network-a-machine-learning-approach-for-precipitation-nowcasting.pdf),[code](https://github.com/loliverhennigh/Convolutional-LSTM-in-Tensorflow)
- Variational Autoencoder:[paper](https://arxiv.org/pdf/1312.6114.pdf)
- Stochastic Adversarial Video Prediction (SAVP): [paper](https://arxiv.org/pdf/1804.01523.pdf),[code](https://github.com/alexlee-gk/video_prediction)
- Motion and Content Network (MCnet): [paper](https://arxiv.org/pdf/1706.08033.pdf), [code](https://github.com/rubenvillegas/iclr2017mcnet)
- Variational Autoencoder:[paper](https://arxiv.org/pdf/1312.6114.pdf)
## Contributors and contact
### Contact
The project is currently developed by Bing Gong, Michael Langguth, Amirpasha Mozafarri, and Yan Ji.
- Amirpash Mozafarri: a.mozafarri@fz-juelich.de
- Michael Langguth: m.langguth@fz-juelich.de
- Bing Gong: b.gong@fz-juelich.de
- Scarlet Stadtler: s.stadtler@fz-juelich.de
- Michael Langguth: m.langguth@fz-juelich.de
- Amirpash Mozafarri: a.mozafarri@fz-juelich.de
- Yan Ji: y.ji@fz-juelich.de
Former code developers are Scarlet Stadtler and Severin Hussmann.
## On-going work
- Port to PyTorch version
- Parallel training neural network
- Integrate precipitation data and new architecture used in our submitted CVPR paper
- Integrate the ML benchmark datasets such as Moving MNIST
ambs_logo.jpg 0 → 100644
+ 0
− 0
View file @ 816ba446
ambs_logo.jpg

55.9 KiB

video_prediction_tools/data_preprocess/era5_varmapping.json 0 → 100644
+ 9
− 0
View file @ 816ba446
{
"surface": ["2t", "tcc","msl","10u","10v"],
"multi":{
"t" : {
"pl": 85000
}
}
}
video_prediction_tools/data_preprocess/era5_varmapping_template.json
+ 2
− 6
View file @ 816ba446
......@@ -10,11 +10,7 @@
# !!! This file should be only adapted if you are familiar with the ERA5 grib files!!!
{
"surface":{
"2t": "var167",
"tcc": "var164",
"msl": "var151",
"10u": "var165",
"10v": "var166"
["2t", "tcc","msl","10u","10v"]
},
"multi":{
......
video_prediction_tools/data_preprocess/prepare_era5_data.py
+ 12
− 9
View file @ 816ba446
......@@ -8,8 +8,8 @@ Functions required for extracting ERA5 data.
import os
import json
__email__ = "b.gong@fz-juelich.de"
__author__ = "Bing Gong, Scarlet Stadtler, Michael Langguth,Yanji"
__date__ = "unknown"
__author__ = "Bing Gong,Michael Langguth,Yanji"
__update_date__ = "2022-02-15"
# specify source and target directories
......@@ -64,21 +64,24 @@ class ERA5DataExtraction(object):
temp_path = os.path.join(self.target_dir, self.year, month)
os.makedirs(temp_path, exist_ok=True)
for var, value in self.varslist_surface.items():
for value in self.varslist_surface:
# surface variables
infile = os.path.join(self.src_dir, self.year, month, self.year+month+day+hour+'_sf.grb')
outfile_sf = os.path.join(self.target_dir, self.year, month, self.year+month+day+hour+'_'+var+'.nc')
os.system('cdo -f nc copy -selname,%s %s %s' % (value, infile, outfile_sf))
os.system('cdo -chname,%s,%s %s %s' % (value, var, outfile_sf, outfile_sf))
outfile_sf = os.path.join(self.target_dir, self.year, month, self.year+month+day+hour+'_'+value+'.nc')
os.system('cdo --eccodes -f nc copy -selname,%s %s %s' % (value, infile, outfile_sf))
# multi-level variables
for var, pl_dic in self.varslist_multi.items():
for pl, pl_value in pl_dic.items():
infile = os.path.join(self.src_dir, self.year, month, self.year+month+day+hour+'_ml.grb')
outfile_sf = os.path.join(self.target_dir, self.year, month, self.year+month+day+hour+'_'+var +
outfile_sf_temp = os.path.join(self.target_dir, self.year, month, self.year+month+day+hour+'_'+var +
str(pl_value) + '.nc')
os.system('cdo -f nc copy -selname,%s -ml2plx,%d %s %s' % (var,pl_value,infile,outfile_sf))
os.system('cdo -chname,%s,%s %s %s' % (var, var+"_{0:d}".format(int(pl_value/100.)), outfile_sf, outfile_sf))
outfile_sf = os.path.join(self.target_dir, self.year, month, self.year+month+day+hour+'_'+var +
str(int(pl_value/100.)) + '.nc')
os.system('cdo -f nc copy -selname,%s -ml2pl,%d %s %s' % (var,pl_value,infile,outfile_sf_temp))
os.system('cdo -chname,%s,%s %s %s' % (var, var+"_{0:d}".format(int(pl_value/100.)), outfile_sf_temp, outfile_sf))
os.system('rm %s' % (outfile_sf_temp))
# merge both variables
infile = os.path.join(self.target_dir, self.year, month, self.year+month+day+hour+'*.nc')
# change the output file name
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment