conflict of lazy preprocessing and overwrite local data
Bug
Error description
There is a logical conflict when using lazy_preprocessing
and overwrite_local_data
together. The value of lazy_preprocessing
is always evaluated first which leads to a neglegtion of overwrite_local_data
if lazy_preprocessing=True
. Therefore, data are never overwritten with lazy_preprocessing=True
, which is not intended by the parameter overwrite_local_data
.
Error message
Logical error without obvious error message.
First guess on error origin
Error origin
Solution
Adjust class method load_lazy
to reload data in case of enabled data overwriting.
def load_lazy(self):
hash = self._get_hash()
filename = os.path.join(self.lazy_path, hash + ".pickle")
try:
if self.overwrite_local_data is True:
raise FileNotFoundError
with open(filename, "rb") as pickle_file:
lazy_data = dill.load(pickle_file)
self._extract_lazy(lazy_data)
logging.debug(f"{self.station[0]}: used lazy data")
except FileNotFoundError:
logging.debug(f"{self.station[0]}: could not use lazy data")
self.make_input_target()