continue training

Create functionality to continue with training of a network. Therefore history and model should be saved (even if the programm crashed during the last training and saving is possible).

• Load model
• continue learning by setting initial_epoch to last tracked epoch
• ...?

UPDATE 2020-01-30 Keras is not able to store the history object by itself. Therefore, continuing to train a loaded model will always start with epoch=0. One could set the inital_epoch to the desired level and all other callbacks, but not the history callback. Of course this could be manually added after training

UPDATE 2020-01-31 Implemented a reload of all used callbacks (that are manually added). Still it is not possible to reload the history itself. But for this issue, there is a new class HistoryAdvanced that can load the history.

changed milestone to %Training

mentioned in issue #17 (closed)

Replace load_best_model() in training.py and model_setup.py by class method

def load_best_model(self):
  logging.debug("load best model")
  try:
    self.model.load_weights(self.model_name)
    logging.info('reload weights...')
  except OSError:
    logging.info('no weights to reload...')

changed the description

added enhancement label

added To Do label

added Doing label and removed To Do label

mentioned in issue #38 (closed)

changed the description

changed the description

assigned to @leufen1

mentioned in commit 9d4776fa

added 4h 32m of time spent at 2020-01-30

added 2h 45m of time spent at 2020-01-31

added merged in Develop label and removed Doing label

mentioned in commit 6db0375f

closed via merge request !37 (merged)