BUG: loading of custom objects not working
Bug
Error description
When continuing a training process (and not starting from epoch 1), there is an error when using a model with custom objects. The error arises on callback loading: self.callbacks.load_callbacks()
will return an error. This error can be triggered using the model MyPaperModel
, create_new_model=False
, train_model=True
and with an interrupted training (so that a model is already stored in callbacks).
Error message
2021-02-16 15:13:22,307 - INFO: Found locally stored model and checkpoints. Training is resumed from the last checkpoint. [training.py:train:145]
2021-02-16 15:13:22,342 - ERROR: Unknown layer: SymmetricPadding2D [run_environment.py:__exit__:137]
Traceback (most recent call last):
File "/home/l.leufen/mlair/mlair/workflows/abstract_workflow.py", line 30, in run
stage(**self._registry_kwargs[pos])
File "/home/l.leufen/mlair/mlair/run_modules/training.py", line 77, in __init__
self._run()
File "/home/l.leufen/mlair/mlair/run_modules/training.py", line 84, in _run
self.train()
File "/home/l.leufen/mlair/mlair/run_modules/training.py", line 146, in train
self.callbacks.load_callbacks()
File "/home/l.leufen/mlair/mlair/model_modules/keras_extensions.py", line 333, in load_callbacks
clb = pickle.load(open(path, "rb"))
File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/engine/network.py", line 1266, in __setstate__
model = saving.unpickle_model(state)
File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/engine/saving.py", line 435, in unpickle_model
return _deserialize_model(f)
File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/engine/saving.py", line 225, in _deserialize_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/engine/saving.py", line 458, in model_from_config
return deserialize(config, custom_objects=custom_objects)
File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
printable_module_name='layer')
File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
list(custom_objects.items())))
File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/engine/network.py", line 1022, in from_config
process_layer(layer_data)
File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/engine/network.py", line 1008, in process_layer
custom_objects=custom_objects)
File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
printable_module_name='layer')
File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/utils/generic_utils.py", line 138, in deserialize_keras_object
': ' + class_name)
ValueError: Unknown layer: SymmetricPadding2D
2021-02-16 15:13:22,366 - INFO: DefaultWorkflow finished after 0:00:19 (hh:mm:ss) [run_environment.py:__del__:118]
First guess on error origin
Something is going wrong with the custom object loading.
Error origin
problems with custom objects when loading callbacks because callbacks contain the model itself (and there is no possibility to parse the custom objects as in the load model method)
Solution
Found this article: https://stackoverflow.com/questions/55753753/in-keras-how-can-you-clone-a-model-with-custom-objects
Just add custom objects to the keras utils and everything works....