FileExistsError in Cleanup / DataHandler Build

Bug

Error description

When running in JUWELS, MLAir returns an FileExistsError during data handling build.

Error message

Traceback (most recent call last):
  File "/gpfs/software/juwels/stages/2019a/software/Python/3.6.8-GCCcore-8.3.0/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/p/project/deepacf/intelliaq/kleinert1/mlair/mlair/run_modules/pre_processing.py", line 339, in f_proc
    res = data_handler.build(station, name_affix=name_affix, store_processed_data=store, **kwargs)
  File "/p/project/deepacf/intelliaq/kleinert1/mlair/mlair/data_handler/default_data_handler.py", line 66, in build
    return cls(sp, **dp_args)
  File "/p/project/deepacf/intelliaq/kleinert1/mlair/mlair/data_handler/default_data_handler.py", line 59, in __init__
    self._store(fresh_store=True, store_processed_data=store_processed_data)
  File "/p/project/deepacf/intelliaq/kleinert1/mlair/mlair/data_handler/default_data_handler.py", line 88, in _store
    self._cleanup() if fresh_store is True else None
  File "/p/project/deepacf/intelliaq/kleinert1/mlair/mlair/data_handler/default_data_handler.py", line 82, in _cleanup
    os.makedirs(directory)
  File "/gpfs/software/juwels/stages/2019a/software/Python/3.6.8-GCCcore-8.3.0/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/p/project/deepacf/intelliaq/kleinert1/mlair/2021-07-01_0738-20_2GPUS_network_daily/data'

First guess on error origin

The os.makedirs statement is encapsulated by an if statement to check if the folder already exists or not. It could be by chance that multiple processes are working almost simultaniously. This could lead to multiple checks that no folder exists. But after the first processes could create the folder, all other processes being just a little bit delayed are running into trouble. Try to fix this by adding the option to skip on exist error os.makedirs(mydir, exist_ok=True).

Error origin

guess confirmed ✅

Solution

Apply change in mlair/data_handler/default_data_handler.py, line 82, _cleanup()

os.makedirs(directory) ▶ os.makedirs(directory, exist_ok=True)