CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

v1.1.0 - 2020-11-18 - hourly resolution support and new data handlers

general:

• MLAir can be used with 1H resolution data from JOIN
• new data handlers to use the Kolmogorov-Zurbenko filter and mixed sampling types

new features:

• new data handler DataHandlerKzFilter to use Kolmogorov-Zurbenko filter (kz filter) on inputs (#195 (closed))
• new data handler DataHandlerMixedSampling that can used mixed sampling types for input and target (#197 (closed))
• new data handler DataHandlerMixedSamplingWithFilter that uses kz filter and mixed sampling (#197 (closed))
• new data handler DataHandlerSeparationOfScales to filter-depended time steps sizes on filtered inputs using mixed sampling (#196 (closed))

technical:

• bug fix for very short time series in TimeSeriesPlot (#215 (closed))
• bug fix for variable dictionary when using hourly resolution (#212 (closed))
• variable naming for data from JOIN interface harmonised (#206 (closed))
• transformation setup is now separated for inputs and targets (#202 (closed))
• bug fix in PlotClimatologicalSkillScore if only single station is used (#193 (closed))
• preprocessed data is now stored inside experiment and not in the data folder

v1.0.0 - 2020-10-08 - official release of new version 1.0.0

general:

• This is the first official release of MLAir ready for use
• updated license, installation instruction

technical:

• restructured order of packages in requirements

v0.12.2 - 2020-10-01 - HDFML support

general:

• HDFML support

technical:

• installation script for HDFML adjusted, #183 (closed)

v0.12.1 - 2020-09-28 - examples in notebook

general:

• introduced a notebook documentation for easy starting, #174 (closed)
• updated special installation instructions for the Juelich HPC systems, #172 (closed)

new features:

• names of input and output shape are renamed consistently to: input_shape, and output_shape, #175 (closed)

technical:

• it is possible to assign a custom name to a run module (e.g. used in logging), #173 (closed)

v0.12.0 - 2020-09-21 - Documentation and Bugfixes

general:

• improved documentation include installation instructions and many examples from the paper, #153 (closed)
• bugfixes (see technical)

new features:

• MyLittleModel is now a pure feed-forward network (before it had a CNN part), #168 (closed)

technical:

• new compile options check to ensure its execution, #154 (closed)
• bugfix for key errors in time series plot, #169 (closed)
• bugfix for not used kwargs in DefaultDataHandler, #170 (closed)
• trainable parameter is renamed by train_model to prevent confusion with the tf trainable parameter, #162 (closed)
• fixed HPC installation failure, #159 (closed)

v0.11.0 - 2020-08-24 - Advanced Data Handling for MLAir

general

• Introduce advanced data handling with much more flexibility (independent of TOAR DB, custom data handling is pluggable), #144 (closed)
• default data handler is still using TOAR DB

new features

• default data handler using TOAR DB refactored according to advanced data handling, #140 (closed), #141 (closed), #152 (closed)
• data sets are handled as collections, #142 (closed), and are iterable in a standard way (StandardIterator) and optimised for keras (KerasIterator), #143 (closed)
• automatically moving station map plot, #136 (closed)

technical

• model modules available from package, #139 (closed)
• renaming of parameter time dimension, #151 (closed)
• refactoring of README.md, #138 (closed)

v0.10.0 - 2020-07-15 - MLAir is official name, Workflows, easy Model plug-in

general

• Official project name is released: MLAir (Machine Learning on Air data)
• a model class can now easily be plugged in into MLAir. #121 (closed)
• introduced new concept of workflows, #134 (closed)

new features

• workflows are used to execute a sequence of run modules, #134 (closed)
• default workflows for standard and the Juelich HPC systems are available, custom workflows can be defined, #134 (closed)
• seasonal decomposition is available for conditional quantile plot, #112 (closed)
• map plot is created with coordinates, #108 (closed)
• flatten_tails are now more general and easier to customise, #114 (closed)
• model classes have custom compile options (replaces set_loss), #110 (closed)
• model can be set in ExperimentSetup from outside, #121 (closed)
• default experiment settings can be queried using get_defaults(), #123 (closed)
• training and model settings are reported as MarkDown and Tex tables, #145 (closed)

technical

• Juelich HPC systems are supported and installation scripts are available, #106 (closed)
• data store is tracked, I/O is saved and illustrated in a plot, #116 (closed)
• batch size, epoch parameter have to be defined in ExperimentSetup, #127 (closed), #122 (closed)
• automatic documentation with sphinx, #109 (closed)
• default experiment settings are updated, #123 (closed)
• refactoring of experiment path and its default naming, #124 (closed)
• refactoring of some parameter names, #146 (closed)
• preparation for package distribution with pip, #119 (closed)
• all run scripts are updated to run with workflows, #134 (closed)
• the experiment folder is restructured, #130 (closed)

v0.9.0 - 2020-04-15 - faster bootstraps, extreme value upsamling

general

• improved and faster bootstrap workflow
• new plot PlotAvailability
• extreme values upsampling
• improved runtime environment

new features

• entire bootstrap workflow has been refactored and much faster now, can be skipped with evaluate_bootstraps=False, #60 (closed)
• upsampling of extreme values, set with parameter extreme_values=[your_values_standardised] (e.g. [1, 2]) and extremes_on_right_tail_only=<True/False> if only right tail of distribution is affected or both, #58 (closed), #87 (closed)
• minimal data length property (in total and for all subsets), #76 (closed)
• custom objects in model class to load customised model objects like padding class, loss, #72 (closed)
• new plot for data availability: PlotAvailability, #103 (closed)
• introduced (default) plot_list to specify which plots to draw
• latex and markdown information on sample sizes for each station, #90 (closed)

technical

• implemented tests on gpu and from scratch for develop, release and master branches, #95 (closed)
• usage of tensorflow 1.13.1 (gpu / cpu), separated in 2 different requirements, #81 (closed)
• new abstract plot class to have uniform plot class design
• New time tracking wrapper to use for functions or classes
• improved logger (info on display, debug into file), #73 (closed), #85 (closed), #88 (closed)
• improved run environment, especially for error handling, #86 (closed)
• prefix general in data store scope is now optional and can be skipped. If given scope is not general, it is treated as subscope, #82 (closed)
• all 2D Padding classes are now selected by Padding2D(padding_name=<padding_type>) e.g. Padding2D(padding_name="SymPad2D"), #78 (closed)
• custom learning rate (or lr_decay) is optional now, #71 (closed)