Skip to content

BUG: loading of custom objects not working

Bug

Error description

When continuing a training process (and not starting from epoch 1), there is an error when using a model with custom objects. The error arises on callback loading: self.callbacks.load_callbacks() will return an error. This error can be triggered using the model MyPaperModel, create_new_model=False, train_model=True and with an interrupted training (so that a model is already stored in callbacks).

Error message

2021-02-16 15:13:22,307 - INFO: Found locally stored model and checkpoints. Training is resumed from the last checkpoint.  [training.py:train:145]
2021-02-16 15:13:22,342 - ERROR: Unknown layer: SymmetricPadding2D  [run_environment.py:__exit__:137]
Traceback (most recent call last):
  File "/home/l.leufen/mlair/mlair/workflows/abstract_workflow.py", line 30, in run
    stage(**self._registry_kwargs[pos])
  File "/home/l.leufen/mlair/mlair/run_modules/training.py", line 77, in __init__
    self._run()
  File "/home/l.leufen/mlair/mlair/run_modules/training.py", line 84, in _run
    self.train()
  File "/home/l.leufen/mlair/mlair/run_modules/training.py", line 146, in train
    self.callbacks.load_callbacks()
  File "/home/l.leufen/mlair/mlair/model_modules/keras_extensions.py", line 333, in load_callbacks
    clb = pickle.load(open(path, "rb"))
  File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/engine/network.py", line 1266, in __setstate__
    model = saving.unpickle_model(state)
  File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/engine/saving.py", line 435, in unpickle_model
    return _deserialize_model(f)
  File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/engine/saving.py", line 225, in _deserialize_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/engine/saving.py", line 458, in model_from_config
    return deserialize(config, custom_objects=custom_objects)
  File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
    list(custom_objects.items())))
  File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/engine/network.py", line 1022, in from_config
    process_layer(layer_data)
  File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/engine/network.py", line 1008, in process_layer
    custom_objects=custom_objects)
  File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/home/l.leufen/mlair/venv/lib64/python3.6/site-packages/keras/utils/generic_utils.py", line 138, in deserialize_keras_object
    ': ' + class_name)
ValueError: Unknown layer: SymmetricPadding2D
2021-02-16 15:13:22,366 - INFO: DefaultWorkflow finished after 0:00:19 (hh:mm:ss)  [run_environment.py:__del__:118]

First guess on error origin

Something is going wrong with the custom object loading.

Error origin

problems with custom objects when loading callbacks because callbacks contain the model itself (and there is no possibility to parse the custom objects as in the load model method)

Solution

Found this article: https://stackoverflow.com/questions/55753753/in-keras-how-can-you-clone-a-model-with-custom-objects

Just add custom objects to the keras utils and everything works....

Edited by Ghost User