Error when running run.py with predownloaded data

Bug

Version

1.3

Error description

Error message

(venv) vincent@localhost:~/PycharmProjects/mlair> python run.py
Using TensorFlow backend.
/home/vincent/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/vincent/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/vincent/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/vincent/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/vincent/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/vincent/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
2021-03-02 16:34:19,831 - INFO: File logger: /home/vincent/PycharmProjects/mlair/logging/logging_2021-03-02_16-34-19.log  [logger.py:__init__:32]
2021-03-02 16:34:19,831 - INFO: DefaultWorkflow started  [run_environment.py:__init__:103]
2021-03-02 16:34:19,831 - INFO: ExperimentSetup started  [run_environment.py:__init__:103]
2021-03-02 16:34:19,836 - INFO: Experiment path is: /home/vincent/PycharmProjects/mlair/testrun_network_daily  [experiment_setup.py:__init__:265]
2021-03-02 16:34:19,852 - INFO: ExperimentSetup finished after 0:00:01 (hh:mm:ss)  [run_environment.py:__del__:118]
2021-03-02 16:34:19,852 - INFO: PreProcessing started  [run_environment.py:__init__:103]
2021-03-02 16:34:19,852 - INFO: check valid stations started (preprocessing)  [pre_processing.py:validate_station:256]
2021-03-02 16:34:19,858 - INFO: use serial validate station approach  [pre_processing.py:validate_station:281]
2021-03-02 16:34:20,073 - INFO: load data for DEBW013 from JOIN  [join.py:download_join:65]
2021-03-02 16:34:47,631 - INFO: setup_samples finished after 0:00:28 (hh:mm:ss)  [time_tracking.py:__exit__:131]
2021-03-02 16:34:47,631 - ERROR: [Errno 13] Permission denied: b'/home/vincent/PycharmProjects/mlair/data/daily/DEBW013_cloudcover_no_no2_o3_pblheight_relhum_temp_u_v.nc'  [run_environment.py:__exit__:137]
Traceback (most recent call last):
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 268, in load_data
    self.check_station_meta(meta, station, station_type, network)
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 332, in check_station_meta
    raise FileNotFoundError
FileNotFoundError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 199, in _acquire_with_cache_info
    file = self._cache[self._key]
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/lru_cache.py", line 53, in __getitem__
    value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/home/vincent/PycharmProjects/mlair/data/daily/DEBW013_cloudcover_no_no2_o3_pblheight_relhum_temp_u_v.nc',), 'a', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vincent/PycharmProjects/mlair/mlair/workflows/abstract_workflow.py", line 30, in run
    stage(**self._registry_kwargs[pos])
  File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 57, in __init__
    self._run()
  File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 63, in _run
    "preprocessing")  # , store_processed_data=False)
  File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 283, in validate_station
    dh, s = f_proc(data_handler, station, set_name, store_processed_data, **kwargs)
  File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 358, in f_proc
    res = data_handler.build(station, name_affix=name_affix, store_processed_data=store, **kwargs)
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/default_data_handler.py", line 60, in build
    sp = cls.data_handler(station, **sp_keys)
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 97, in __init__
    self.setup_samples()
  File "/home/vincent/PycharmProjects/mlair/mlair/helpers/time_tracking.py", line 40, in __call__
    return self.__wrapped__(*args, **kwargs)
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 220, in setup_samples
    self.start, self.end)
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 275, in load_data
    store_data_locally=store_data_locally, data_origin=data_origin)
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 313, in download_data
    iter_dim=self.iter_dim)
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 307, in download_data_from_join
    xarr.to_netcdf(path=file_name)
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/core/dataarray.py", line 2664, in to_netcdf
    return dataset.to_netcdf(*args, **kwargs)
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/core/dataset.py", line 1654, in to_netcdf
    invalid_netcdf=invalid_netcdf,
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/api.py", line 1094, in to_netcdf
    store = store_open(target, mode, format, group, **kwargs)
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 364, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 314, in __init__
    self.format = self.ds.data_model
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 373, in ds
    return self._acquire()
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 367, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 187, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 205, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
  File "netCDF4/_netCDF4.pyx", line 2357, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1925, in netCDF4._netCDF4._ensure_nc_success
PermissionError: [Errno 13] Permission denied: b'/home/vincent/PycharmProjects/mlair/data/daily/DEBW013_cloudcover_no_no2_o3_pblheight_relhum_temp_u_v.nc'
2021-03-02 16:34:47,642 - INFO: DefaultWorkflow finished after 0:00:28 (hh:mm:ss)  [run_environment.py:__del__:118]
2021-03-02 16:34:50,356 - INFO: Copy tracker file to /home/vincent/PycharmProjects/mlair/testrun_network_daily/logging/tracking_002.json  [run_environment.py:__save_tracking:155]
2021-03-02 16:34:50,358 - INFO: Move log file to /home/vincent/PycharmProjects/mlair/testrun_network_daily/logging/logging_002.log  [run_environment.py:__move_log_file:143]
Traceback (most recent call last):
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 268, in load_data
    self.check_station_meta(meta, station, station_type, network)
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 332, in check_station_meta
    raise FileNotFoundError
FileNotFoundError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 199, in _acquire_with_cache_info
    file = self._cache[self._key]
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/lru_cache.py", line 53, in __getitem__
    value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/home/vincent/PycharmProjects/mlair/data/daily/DEBW013_cloudcover_no_no2_o3_pblheight_relhum_temp_u_v.nc',), 'a', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run.py", line 40, in <module>
    main(args)
  File "run.py", line 32, in main
    workflow.run()
  File "/home/vincent/PycharmProjects/mlair/mlair/workflows/abstract_workflow.py", line 30, in run
    stage(**self._registry_kwargs[pos])
  File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 57, in __init__
    self._run()
  File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 63, in _run
    "preprocessing")  # , store_processed_data=False)
  File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 283, in validate_station
    dh, s = f_proc(data_handler, station, set_name, store_processed_data, **kwargs)
  File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/pre_processing.py", line 358, in f_proc
    res = data_handler.build(station, name_affix=name_affix, store_processed_data=store, **kwargs)
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/default_data_handler.py", line 60, in build
    sp = cls.data_handler(station, **sp_keys)
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 97, in __init__
    self.setup_samples()
  File "/home/vincent/PycharmProjects/mlair/mlair/helpers/time_tracking.py", line 40, in __call__
    return self.__wrapped__(*args, **kwargs)
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 220, in setup_samples
    self.start, self.end)
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 275, in load_data
    store_data_locally=store_data_locally, data_origin=data_origin)
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 313, in download_data
    iter_dim=self.iter_dim)
  File "/home/vincent/PycharmProjects/mlair/mlair/data_handler/data_handler_single_station.py", line 307, in download_data_from_join
    xarr.to_netcdf(path=file_name)
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/core/dataarray.py", line 2664, in to_netcdf
    return dataset.to_netcdf(*args, **kwargs)
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/core/dataset.py", line 1654, in to_netcdf
    invalid_netcdf=invalid_netcdf,
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/api.py", line 1094, in to_netcdf
    store = store_open(target, mode, format, group, **kwargs)
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 364, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 314, in __init__
    self.format = self.ds.data_model
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 373, in ds
    return self._acquire()
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 367, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 187, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
  File "/home/vincent/.local/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 205, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
  File "netCDF4/_netCDF4.pyx", line 2357, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1925, in netCDF4._netCDF4._ensure_nc_success
PermissionError: [Errno 13] Permission denied: b'/home/vincent/PycharmProjects/mlair/data/daily/DEBW013_cloudcover_no_no2_o3_pblheight_relhum_temp_u_v.nc'
Exception ignored in: <bound method RunEnvironment.__del__ of <mlair.run_modules.pre_processing.PreProcessing object at 0x7feb217cfac8>>
Traceback (most recent call last):
  File "/home/vincent/PycharmProjects/mlair/mlair/run_modules/run_environment.py", line 118, in __del__
    logging.info(f"{self._name} finished after {self.time}")
  File "/usr/lib64/python3.6/logging/__init__.py", line 1902, in info
    root.info(msg, *args, **kwargs)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1308, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1444, in _log
    self.handle(record)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1454, in handle
    self.callHandlers(record)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1516, in callHandlers
    hdlr.handle(record)
  File "/usr/lib64/python3.6/logging/__init__.py", line 865, in handle
    self.emit(record)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1071, in emit
    self.stream = self._open()
  File "/usr/lib64/python3.6/logging/__init__.py", line 1061, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
FileNotFoundError: [Errno 2] No such file or directory: '/home/vincent/PycharmProjects/mlair/logging/logging_2021-03-02_16-34-19.log'

First guess on error origin

Error origin

Solution