diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..28ea5b77e082085e1904ecc5a38d1e6730fa7cfe --- /dev/null +++ b/README.md @@ -0,0 +1,121 @@ +# Video Prediction by GAN + +This project aims to adopt the GAN-based architectures, which original proposed by [[Project Page]](https://alexlee-gk.github.io/video_prediction/) [[Paper]](https://arxiv.org/abs/1804.01523), to predict temperature based on ERA5 data + +## Getting Started ### +### Prerequisites +- Linux or macOS +- Python 3 +- CPU or NVIDIA GPU + CUDA CuDNN + +### Installation +This project need to work with [Workflow_parallel_frame_prediction project](https://gitlab.version.fz-juelich.de/gong1/workflow_parallel_frame_prediction) +- Clone this repo: +```bash +git clone master https://gitlab.version.fz-juelich.de/gong1/video_prediction_savp.git +git clone master https://gitlab.version.fz-juelich.de/gong1/workflow_parallel_frame_prediction.git +``` + +### Set-up env on JUWELS + +- Set up env and install packages + +```bash +cd video_prediction_savp +source env_setup/create_env.sh <dir_name> <env_name> +``` + +## Workflow by steps + +### Data Extraction + +```python +python3 ../workflow_video_prediction/DataExtraction/mpi_stager_v2.py --source_dir <input_dir1> --destination_dir <output_dir1> +``` + +e.g. +```python +python3 ../workflow_video_prediction/DataExtraction/mpi_stager_v2.py --source_dir /p/fastdata/slmet/slmet111/met_data/ecmwf/era5/nc/2017/ --destination_dir /p/scratch/deepacf/bing/extractedData +``` + +### Data Preprocessing +```python +python3 ../workflow_video_prediction/DataPreprocess/mpi_stager_v2_process_netCDF.py --source_dir <output_dir1> --destination_dir <output_dir2> + +python3 video_prediction/datasets/era5_dataset_v2.py --source_dir <output_dir2> --destination_dir <.data/exp_name> +``` + +Example +```python +python3 ../workflow_video_prediction/DataPreprocess/mpi_stager_v2_process_netCDF.py --source_dir /p/scratch/deepacf/bing/extractedData --destination_dir /p/scratch/deepacf/bing/preprocessedData + +python3 video_prediction/datasets/era5_dataset_v2.py /p/scratch/deepacf/bing/preprocessedData ./data/era5_64_64_3_3t_norm + ``` + +### Trarining + +```python +python3 scripts/train_v2.py --input_dir <./data/exp_name> --dataset era5 --model <savp> --model_hparams_dict hparams/kth/ours_savp/model_hparams.json --output_dir <./logs/{exp_name}/{mode}/> +``` + +Example +```python +python3 scripts/train_v2.py --input_dir ./data/era5_size_64_64_3_3t_norm --dataset era5 --model savp --model_hparams_dict hparams/kth/ours_savp/model_hparams.json --output_dir logs/era5_64_64_3_3t_norm/end_to_end +``` +### Postprocessing + +Generating prediction frames, model evaluation, and visulization +You can trained your own model from the training step , or you can copy the Bing's trained model + +```python +python3 scripts/generate_transfer_learning_finetune.py --input_dir <./data/exp_name> --dataset_hparams sequence_length=20 --checkpoint <./logs/{exp_name}/{mode}/{model}> --mode test --results_dir <./results/{exp_name}/{mode}> --batch_size <batch_size> --dataset era5 +``` + +- example: use end_to_end training model from bing for exp_name:era5_size_64_64_3_3t_norm +```python +python3 scripts/generate_transfer_learning_finetune.py --input_dir data/era5_size_64_64_3_3t_norm --dataset_hparams sequence_length=20 --checkpoint /p/project/deepacf/deeprain/bing/video_prediction_savp/logs/era5_size_64_64_3_3t_norm/end_to_end/ours_savp --mode test --results_dir results_test_samples/era5_size_64_64_3_3t_norm/end_to_end --batch_size 4 --dataset era5 +``` + + +# End-to-End run the entire workflow + +```bash +./bash/workflow_era5.sh <model> <train_mode> <exp_name> +``` + +example: +```bash +./bash/workflow_era5.sh savp end_to_end era5_size_64_64_3_3t_norm +``` + + + +### Recomendation for output folder structure and name convention +The details can be found [name_convention](docs/structure_name_convention.md) + +``` +├── ExtractedData +│ ├── [Year] +│ │ ├── [Month] +│ │ │ ├── **/*.netCDF +├── PreprocessedData +│ ├── [Data_name_convention] +│ │ ├── hickle +│ │ │ ├── train +│ │ │ ├── val +│ │ │ ├── test +│ │ ├── tfrecords +│ │ │ ├── train +│ │ │ ├── val +│ │ │ ├── test +├── Models +│ ├── [Data_name_convention] +│ │ ├── [model_name] +│ │ ├── [model_name] +├── Results +│ ├── [Data_name_convention] +│ │ ├── [training_mode] +│ │ │ ├── [source_data_name_convention] +│ │ │ │ ├── [model_name] + +``` \ No newline at end of file