Error when running run.py with predownloaded data
Bug
Version
1.3
Error description
Error message
(venv) vincent@localhost:~/PycharmProjects/mlair> python run.py
Using TensorFlow backend.
/home/vincent/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/vincent/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/vincent/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/vincent/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/vincent/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/vincent/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
2021-03-02 16:34:19,831 - INFO: File logger: /home/vincent/PycharmProjects/mlair/logging/logging_2021-03-02_16-34-19.log [logger.py:__init__:32]
2021-03-02 16:34:19,831 - INFO: DefaultWorkflow started [run_environment.py:__init__:103]
2021-03-02 16:34:19,831 - INFO: ExperimentSetup started [run_environment.py:__init__:103]
2021-03-02 16:34:19,836 - INFO: Experiment path is: /home/vincent/PycharmProjects/mlair/testrun_network_daily [experiment_setup.py:__init__:265]
2021-03-02 16:34:19,852 - INFO: ExperimentSetup finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:118]
2021-03-02 16:34:19,852 - INFO: PreProcessing started [run_environment.py:__init__:103]
2021-03-02 16:34:19,852 - INFO: check valid stations started (preprocessing) [pre_processing.py:validate_station:256]
2021-03-02 16:34:19,858 - INFO: use serial validate station approach [pre_processing.py:validate_station:281]
2021-03-02 16:34:20,073 - INFO: load data for DEBW013 from JOIN [join.py:download_join:65]
2021-03-02 16:34:47,631 - INFO: setup_samples finished after 0:00:28 (hh:mm:ss) [time_tracking.py:__exit__:131]
2021-03-02 16:34:47,631 - ERROR: [Errno 13] Permission denied: b'/home/vincent/PycharmProjects/mlair/data/daily/DEBW013_cloudcover_no_no2_o3_pblheight_relhum_temp_u_v.nc' [run_environment.py:__exit__:137]
Traceback (most recent call last):
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 268, in load_data
self.check_station_meta(meta, station, station_type, network)
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 332, in check_station_meta
raise FileNotFoundError
FileNotFoundError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 199, in _acquire_with_cache_info
file = self._cache[self._key]
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/lru_cache.py", line 53, in __getitem__
value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/home/vincent/PycharmProjects/mlair/data/daily/DEBW013_cloudcover_no_no2_o3_pblheight_relhum_temp_u_v.nc',), 'a', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/vincent/PycharmProjects/mlair/mlair/workflows/abstract_workflow.py", line 30, in run
stage(**self._registry_kwargs[pos])
File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 57, in __init__
self._run()
File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 63, in _run
"preprocessing") # , store_processed_data=False)
File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 283, in validate_station
dh, s = f_proc(data_handler, station, set_name, store_processed_data, **kwargs)
File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 358, in f_proc
res = data_handler.build(station, name_affix=name_affix, store_processed_data=store, **kwargs)
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/default_data_handler.py", line 60, in build
sp = cls.data_handler(station, **sp_keys)
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 97, in __init__
self.setup_samples()
File "/home/vincent/PycharmProjects/mlair/mlair/helpers/time_tracking.py", line 40, in __call__
return self.__wrapped__(*args, **kwargs)
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 220, in setup_samples
self.start, self.end)
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 275, in load_data
store_data_locally=store_data_locally, data_origin=data_origin)
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 313, in download_data
iter_dim=self.iter_dim)
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 307, in download_data_from_join
xarr.to_netcdf(path=file_name)
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/core/dataarray.py", line 2664, in to_netcdf
return dataset.to_netcdf(*args, **kwargs)
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/core/dataset.py", line 1654, in to_netcdf
invalid_netcdf=invalid_netcdf,
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/api.py", line 1094, in to_netcdf
store = store_open(target, mode, format, group, **kwargs)
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 364, in open
return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 314, in __init__
self.format = self.ds.data_model
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 373, in ds
return self._acquire()
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 367, in _acquire
with self._manager.acquire_context(needs_lock) as root:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 187, in acquire_context
file, cached = self._acquire_with_cache_info(needs_lock)
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 205, in _acquire_with_cache_info
file = self._opener(*self._args, **kwargs)
File "netCDF4/_netCDF4.pyx", line 2357, in netCDF4._netCDF4.Dataset.__init__
File "netCDF4/_netCDF4.pyx", line 1925, in netCDF4._netCDF4._ensure_nc_success
PermissionError: [Errno 13] Permission denied: b'/home/vincent/PycharmProjects/mlair/data/daily/DEBW013_cloudcover_no_no2_o3_pblheight_relhum_temp_u_v.nc'
2021-03-02 16:34:47,642 - INFO: DefaultWorkflow finished after 0:00:28 (hh:mm:ss) [run_environment.py:__del__:118]
2021-03-02 16:34:50,356 - INFO: Copy tracker file to /home/vincent/PycharmProjects/mlair/testrun_network_daily/logging/tracking_002.json [run_environment.py:__save_tracking:155]
2021-03-02 16:34:50,358 - INFO: Move log file to /home/vincent/PycharmProjects/mlair/testrun_network_daily/logging/logging_002.log [run_environment.py:__move_log_file:143]
Traceback (most recent call last):
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 268, in load_data
self.check_station_meta(meta, station, station_type, network)
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 332, in check_station_meta
raise FileNotFoundError
FileNotFoundError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 199, in _acquire_with_cache_info
file = self._cache[self._key]
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/lru_cache.py", line 53, in __getitem__
value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/home/vincent/PycharmProjects/mlair/data/daily/DEBW013_cloudcover_no_no2_o3_pblheight_relhum_temp_u_v.nc',), 'a', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "run.py", line 40, in <module>
main(args)
File "run.py", line 32, in main
workflow.run()
File "/home/vincent/PycharmProjects/mlair/mlair/workflows/abstract_workflow.py", line 30, in run
stage(**self._registry_kwargs[pos])
File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 57, in __init__
self._run()
File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 63, in _run
"preprocessing") # , store_processed_data=False)
File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 283, in validate_station
dh, s = f_proc(data_handler, station, set_name, store_processed_data, **kwargs)
File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 358, in f_proc
res = data_handler.build(station, name_affix=name_affix, store_processed_data=store, **kwargs)
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/default_data_handler.py", line 60, in build
sp = cls.data_handler(station, **sp_keys)
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 97, in __init__
self.setup_samples()
File "/home/vincent/PycharmProjects/mlair/mlair/helpers/time_tracking.py", line 40, in __call__
return self.__wrapped__(*args, **kwargs)
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 220, in setup_samples
self.start, self.end)
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 275, in load_data
store_data_locally=store_data_locally, data_origin=data_origin)
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 313, in download_data
iter_dim=self.iter_dim)
File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 307, in download_data_from_join
xarr.to_netcdf(path=file_name)
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/core/dataarray.py", line 2664, in to_netcdf
return dataset.to_netcdf(*args, **kwargs)
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/core/dataset.py", line 1654, in to_netcdf
invalid_netcdf=invalid_netcdf,
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/api.py", line 1094, in to_netcdf
store = store_open(target, mode, format, group, **kwargs)
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 364, in open
return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 314, in __init__
self.format = self.ds.data_model
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 373, in ds
return self._acquire()
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 367, in _acquire
with self._manager.acquire_context(needs_lock) as root:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 187, in acquire_context
file, cached = self._acquire_with_cache_info(needs_lock)
File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 205, in _acquire_with_cache_info
file = self._opener(*self._args, **kwargs)
File "netCDF4/_netCDF4.pyx", line 2357, in netCDF4._netCDF4.Dataset.__init__
File "netCDF4/_netCDF4.pyx", line 1925, in netCDF4._netCDF4._ensure_nc_success
PermissionError: [Errno 13] Permission denied: b'/home/vincent/PycharmProjects/mlair/data/daily/DEBW013_cloudcover_no_no2_o3_pblheight_relhum_temp_u_v.nc'
Exception ignored in: <bound method RunEnvironment.__del__ of <mlair.run_modules.pre_processing.PreProcessing object at 0x7feb217cfac8>>
Traceback (most recent call last):
File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/run_environment.py", line 118, in __del__
logging.info(f"{self._name} finished after {self.time}")
File "/usr/lib64/python3.6/logging/__init__.py", line 1902, in info
root.info(msg, *args, **kwargs)
File "/usr/lib64/python3.6/logging/__init__.py", line 1308, in info
self._log(INFO, msg, args, **kwargs)
File "/usr/lib64/python3.6/logging/__init__.py", line 1444, in _log
self.handle(record)
File "/usr/lib64/python3.6/logging/__init__.py", line 1454, in handle
self.callHandlers(record)
File "/usr/lib64/python3.6/logging/__init__.py", line 1516, in callHandlers
hdlr.handle(record)
File "/usr/lib64/python3.6/logging/__init__.py", line 865, in handle
self.emit(record)
File "/usr/lib64/python3.6/logging/__init__.py", line 1071, in emit
self.stream = self._open()
File "/usr/lib64/python3.6/logging/__init__.py", line 1061, in _open
return open(self.baseFilename, self.mode, encoding=self.encoding)
FileNotFoundError: [Errno 2] No such file or directory: '/home/vincent/PycharmProjects/mlair/logging/logging_2021-03-02_16-34-19.log'
First guess on error origin
Error origin
Solution
Edited by Ghost User