FileExistsError in Cleanup / DataHandler Build
Bug
Error description
When running in JUWELS, MLAir returns an FileExistsError during data handling build.
Error message
Traceback (most recent call last):
File "/gpfs/software/juwels/stages/2019a/software/Python/3.6.8-GCCcore-8.3.0/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/p/project/deepacf/intelliaq/kleinert1/mlair/mlair/run_modules/pre_processing.py", line 339, in f_proc
res = data_handler.build(station, name_affix=name_affix, store_processed_data=store, **kwargs)
File "/p/project/deepacf/intelliaq/kleinert1/mlair/mlair/data_handler/default_data_handler.py", line 66, in build
return cls(sp, **dp_args)
File "/p/project/deepacf/intelliaq/kleinert1/mlair/mlair/data_handler/default_data_handler.py", line 59, in __init__
self._store(fresh_store=True, store_processed_data=store_processed_data)
File "/p/project/deepacf/intelliaq/kleinert1/mlair/mlair/data_handler/default_data_handler.py", line 88, in _store
self._cleanup() if fresh_store is True else None
File "/p/project/deepacf/intelliaq/kleinert1/mlair/mlair/data_handler/default_data_handler.py", line 82, in _cleanup
os.makedirs(directory)
File "/gpfs/software/juwels/stages/2019a/software/Python/3.6.8-GCCcore-8.3.0/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/p/project/deepacf/intelliaq/kleinert1/mlair/2021-07-01_0738-20_2GPUS_network_daily/data'
First guess on error origin
The os.makedirs
statement is encapsulated by an if
statement to check if the folder already exists or not. It could be by chance that multiple processes are working almost simultaniously. This could lead to multiple checks that no folder exists. But after the first processes could create the folder, all other processes being just a little bit delayed are running into trouble. Try to fix this by adding the option to skip on exist error os.makedirs(mydir, exist_ok=True)
.
Error origin
guess confirmed
Solution
Apply change in mlair/data_handler/default_data_handler.py, line 82, _cleanup()
os.makedirs(directory)
os.makedirs(directory, exist_ok=True)
Edited by Ghost User