Include advanced data handling in workflow

Include all developments from #140 (closed), #141 (closed), #142 (closed), #143 (closed) into run_modules

• replace distributor by KerasIterator
• replace generator by DataCollection
• replace preparation by StationPrep + DataPrepration
• adjust data transformation
• adjust check stations
• all kwargs of StationPrep need to be implemented explicitly (like start, end)

changed milestone to %Advanced data handling

added Doing enhancement refactoring labels

created merge request !119 (merged) to address this issue

mentioned in merge request !119 (merged)

changed the description

changed the description

Comment on the new design (updated description soon!):

There is a new abstract base class AbstractDataPreparation that has:

• (classmethod) build(): returns a class instance
• (classmethod) requirements(): returns name of all required parameters
• (classmethod) own_args(): returns only parameters of class init
• get_X(), get_Y(): raises not implemented
• __init__(): just pass
• _requirements: set all additional requirements here (e.g. from another used class)

With this base class, it is possible to parse a data preparation class from outside. Instead of calling the init, the class is parsed without initialisation. Inside preprocessing, first the requirements() are requested and then build() will create an instance for each given station (station of interest / id_class).

There are already two inheriting classes: DefaultDataPreparation and DataPreparationNeighbors

DefaultDataPreparation just uses a single central station (based on the StationPreparation). It overwrites the _requirements, build(), requirements(), get_X(), and get_Y() from the mother class for a proper working.

DataPrepration is a special implementation of DefaultDataPreparation using neighbors if given additionally. This class just has to overwrite the build() method, as well as another method (_create_collection) that is implemented in the Default class (not in the abstract one!). This class still doesn't work as requested, because it is not possible to add the neighbors for individual stations. Either think to add all and let the network select the neighbors or think about a solution to specify the neighbors for each station. Maybe this could be implemented in the build method.

marked the checklist item replace generator by DataCollection as completed

marked the checklist item replace preparation by StationPrep + DataPrepration as completed

marked the checklist item adjust data transformation as completed

marked the checklist item adjust check stations as completed

marked the checklist item all kwargs of StationPrep need to be implemented explicitly (like start, end) as completed

added .done label and removed Doing label

added merged in Develop label and removed .done label

mentioned in commit d099bd6d

closed

removed merged in Develop label