MLAir tags
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags
2023-06-30T11:39:43+02:00
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v2.4.0
v2.4.0
<h3 data-sourcepos="1:1-1:12" dir="auto">
<a id="user-content-general" class="anchor" href="#general" aria-hidden="true"></a>general:</h3>
<ul data-sourcepos="2:1-4:0" dir="auto">
<li data-sourcepos="2:1-2:55">support IFS data (local) and ERA5 data (from toar db)</li>
<li data-sourcepos="3:1-4:0">bias free evaluation</li>
</ul>
<h3 data-sourcepos="5:1-5:17" dir="auto">
<a id="user-content-new-features" class="anchor" href="#new-features" aria-hidden="true"></a>new features:</h3>
<ul data-sourcepos="6:1-12:1" dir="auto">
<li data-sourcepos="6:1-6:50">can load local IFS forecast data as input (<a href="/esde/machine-learning/mlair/-/issues/450" data-reference-type="issue" data-original="#450" data-link="false" data-link-reference="false" data-project="2411" data-issue="17778" data-project-path="esde/machine-learning/mlair" data-iid="450" data-issue-type="issue" data-container="body" data-placement="top" title="load ifs data" class="gfm gfm-issue">#450</a>)</li>
<li data-sourcepos="7:1-7:83">can also load ERA5 data from toar db as alternative to locally stored data (<a href="/esde/machine-learning/mlair/-/issues/449" data-reference-type="issue" data-original="#449" data-link="false" data-link-reference="false" data-project="2411" data-issue="17777" data-project-path="esde/machine-learning/mlair" data-iid="449" data-issue-type="issue" data-container="body" data-placement="top" title="load era5 data from toar db" class="gfm gfm-issue">#449</a>)</li>
<li data-sourcepos="8:1-8:63">new plot to show monthly data distributions in subsets (<a href="/esde/machine-learning/mlair/-/issues/445" data-reference-type="issue" data-original="#445" data-link="false" data-link-reference="false" data-project="2411" data-issue="16976" data-project-path="esde/machine-learning/mlair" data-iid="445" data-issue-type="issue" data-container="body" data-placement="top" title="Data Insight Plot Monthly Distribution" class="gfm gfm-issue">#445</a>)</li>
<li data-sourcepos="9:1-9:47">can load a DL model from external path (<a href="/esde/machine-learning/mlair/-/issues/448" data-reference-type="issue" data-original="#448" data-link="false" data-link-reference="false" data-project="2411" data-issue="17725" data-project-path="esde/machine-learning/mlair" data-iid="448" data-issue-type="issue" data-container="body" data-placement="top" title="load model from path" class="gfm gfm-issue">#448</a>)</li>
<li data-sourcepos="10:1-10:77">introduced option to bias-correct model's and competitors' forecasts (<a href="/esde/machine-learning/mlair/-/issues/442" data-reference-type="issue" data-original="#442" data-link="false" data-link-reference="false" data-project="2411" data-issue="16765" data-project-path="esde/machine-learning/mlair" data-iid="442" data-issue-type="issue" data-container="body" data-placement="top" title="bias free evaluation" class="gfm gfm-issue">#442</a>)</li>
<li data-sourcepos="11:1-12:1">can use different interpolation methods when having CAMS as competitor (<a href="/esde/machine-learning/mlair/-/issues/444" data-reference-type="issue" data-original="#444" data-link="false" data-link-reference="false" data-project="2411" data-issue="16949" data-project-path="esde/machine-learning/mlair" data-iid="444" data-issue-type="issue" data-container="body" data-placement="top" title="choose interp method in CAMS competitor" class="gfm gfm-issue">#444</a>)</li>
</ul>
<h3 data-sourcepos="13:1-13:14" dir="auto">
<a id="user-content-technical" class="anchor" href="#technical" aria-hidden="true"></a>technical:</h3>
<ul data-sourcepos="14:1-18:111" dir="auto">
<li data-sourcepos="14:1-14:53">change toar statistics from api v1 to api v2 (<a href="/esde/machine-learning/mlair/-/issues/454" data-reference-type="issue" data-original="#454" data-link="false" data-link-reference="false" data-project="2411" data-issue="17993" data-project-path="esde/machine-learning/mlair" data-iid="454" data-issue-type="issue" data-container="body" data-placement="top" title="Use Toar statistics api v2" class="gfm gfm-issue">#454</a>)</li>
<li data-sourcepos="15:1-15:96">now able to set configuration paths for local era5 and ifs data as experiment parameter (<a href="/esde/machine-learning/mlair/-/issues/457" data-reference-type="issue" data-original="#457" data-link="false" data-link-reference="false" data-project="2411" data-issue="18083" data-project-path="esde/machine-learning/mlair" data-iid="457" data-issue-type="issue" data-container="body" data-placement="top" title="set config paths as parameter" class="gfm gfm-issue">#457</a>)</li>
<li data-sourcepos="16:1-16:67">improved retry strategy when downloading data from toar db (<a href="/esde/machine-learning/mlair/-/issues/453" data-reference-type="issue" data-original="#453" data-link="false" data-link-reference="false" data-project="2411" data-issue="17946" data-project-path="esde/machine-learning/mlair" data-iid="453" data-issue-type="issue" data-container="body" data-placement="top" title="advanced retry strategy" class="gfm gfm-issue">#453</a>)</li>
<li data-sourcepos="17:1-17:25">updated packages (<a href="/esde/machine-learning/mlair/-/issues/452" data-reference-type="issue" data-original="#452" data-link="false" data-link-reference="false" data-project="2411" data-issue="17908" data-project-path="esde/machine-learning/mlair" data-iid="452" data-issue-type="issue" data-container="body" data-placement="top" title="update proj version" class="gfm gfm-issue">#452</a>)</li>
<li data-sourcepos="18:1-18:111">calculation of filter apriori is more robust now, properties are stored inside experiment folder (<a href="/esde/machine-learning/mlair/-/issues/447" data-reference-type="issue" data-original="#447" data-link="false" data-link-reference="false" data-project="2411" data-issue="17720" data-project-path="esde/machine-learning/mlair" data-iid="447" data-issue-type="issue" data-container="body" data-placement="top" title="store and load local clim apriori data" class="gfm gfm-issue">#447</a>, <a href="/esde/machine-learning/mlair/-/issues/451" data-reference-type="issue" data-original="#451" data-link="false" data-link-reference="false" data-project="2411" data-issue="17838" data-project-path="esde/machine-learning/mlair" data-iid="451" data-issue-type="issue" data-container="body" data-placement="top" title="robust apriori estimate for short timeseries" class="gfm gfm-issue">#451</a>)</li>
</ul>
2023-06-30T11:39:43+02:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v2.3.0
v2.3.0
new models and plots
<h3 data-sourcepos="1:1-1:12" dir="auto">
<a id="user-content-general" class="anchor" href="#general" aria-hidden="true"></a>general:</h3>
<ul data-sourcepos="2:1-4:0" dir="auto">
<li data-sourcepos="2:1-2:40">new model classes for ResNet and U-Net</li>
<li data-sourcepos="3:1-4:0">new plots and variations of existing plots</li>
</ul>
<h3 data-sourcepos="5:1-5:17" dir="auto">
<a id="user-content-new-features" class="anchor" href="#new-features" aria-hidden="true"></a>new features:</h3>
<ul data-sourcepos="6:1-14:0" dir="auto">
<li data-sourcepos="6:1-6:48">new model classes: ResNet (<a href="/esde/machine-learning/mlair/-/issues/419" data-reference-type="issue" data-original="#419" data-link="false" data-link-reference="false" data-project="2411" data-issue="15596" data-project-path="esde/machine-learning/mlair" data-iid="419" data-issue-type="issue" data-container="body" data-placement="top" title="ResNet model class" class="gfm gfm-issue">#419</a>), U-Net (<a href="/esde/machine-learning/mlair/-/issues/423" data-reference-type="issue" data-original="#423" data-link="false" data-link-reference="false" data-project="2411" data-issue="15794" data-project-path="esde/machine-learning/mlair" data-iid="423" data-issue-type="issue" data-container="body" data-placement="top" title="U-Net model class" class="gfm gfm-issue">#423</a>)</li>
<li data-sourcepos="7:1-7:32">seasonal mse stack plot (<a href="/esde/machine-learning/mlair/-/issues/422" data-reference-type="issue" data-original="#422" data-link="false" data-link-reference="false" data-project="2411" data-issue="15756" data-project-path="esde/machine-learning/mlair" data-iid="422" data-issue-type="issue" data-container="body" data-placement="top" title="seasonal mse stack plot" class="gfm gfm-issue">#422</a>)</li>
<li data-sourcepos="8:1-8:70">new aggregated and line versions of Time Evolution Plot (<a href="/esde/machine-learning/mlair/-/issues/424" data-reference-type="issue" data-original="#424" data-link="false" data-link-reference="false" data-project="2411" data-issue="15820" data-project-path="esde/machine-learning/mlair" data-iid="424" data-issue-type="issue" data-container="body" data-placement="top" title="time evolution of aggregated metrics" class="gfm gfm-issue">#424</a>, <a href="/esde/machine-learning/mlair/-/issues/427" data-reference-type="issue" data-original="#427" data-link="false" data-link-reference="false" data-project="2411" data-issue="15939" data-project-path="esde/machine-learning/mlair" data-iid="427" data-issue-type="issue" data-container="body" data-placement="top" title="MSE line plot" class="gfm gfm-issue">#427</a>)</li>
<li data-sourcepos="9:1-9:64">box-and-whisker plots are created for all error metrics (<a href="/esde/machine-learning/mlair/-/issues/431" data-reference-type="issue" data-original="#431" data-link="false" data-link-reference="false" data-project="2411" data-issue="16016" data-project-path="esde/machine-learning/mlair" data-iid="431" data-issue-type="issue" data-container="body" data-placement="top" title="plot error metrics over all stations" class="gfm gfm-issue">#431</a>)</li>
<li data-sourcepos="10:1-10:103">new split and frequency distribution versions of box-and-whisker plots for error metrics (<a href="/esde/machine-learning/mlair/-/issues/425" data-reference-type="issue" data-original="#425" data-link="false" data-link-reference="false" data-project="2411" data-issue="15848" data-project-path="esde/machine-learning/mlair" data-iid="425" data-issue-type="issue" data-container="body" data-placement="top" title="split day ahead in MSE uncertainy plot on distinct plots" class="gfm gfm-issue">#425</a>, <a href="/esde/machine-learning/mlair/-/issues/434" data-reference-type="issue" data-original="#434" data-link="false" data-link-reference="false" data-project="2411" data-issue="16116" data-project-path="esde/machine-learning/mlair" data-iid="434" data-issue-type="issue" data-container="body" data-placement="top" title="Frequency error plots" class="gfm gfm-issue">#434</a>)</li>
<li data-sourcepos="11:1-11:49">new evaluation metric: mean error / bias (<a href="/esde/machine-learning/mlair/-/issues/430" data-reference-type="issue" data-original="#430" data-link="false" data-link-reference="false" data-project="2411" data-issue="16015" data-project-path="esde/machine-learning/mlair" data-iid="430" data-issue-type="issue" data-container="body" data-placement="top" title="new metric bias / mean error" class="gfm gfm-issue">#430</a>)</li>
<li data-sourcepos="12:1-12:72">conditional quantiles are now available for all competitors too (<a href="/esde/machine-learning/mlair/-/issues/435" data-reference-type="issue" data-original="#435" data-link="false" data-link-reference="false" data-project="2411" data-issue="16243" data-project-path="esde/machine-learning/mlair" data-iid="435" data-issue-type="issue" data-container="body" data-placement="top" title="conditional quantiles for competitors" class="gfm gfm-issue">#435</a>)</li>
<li data-sourcepos="13:1-14:0">new map plot showing mse at locations (<a href="/esde/machine-learning/mlair/-/issues/432" data-reference-type="issue" data-original="#432" data-link="false" data-link-reference="false" data-project="2411" data-issue="16047" data-project-path="esde/machine-learning/mlair" data-iid="432" data-issue-type="issue" data-container="body" data-placement="top" title="MSE Map Plot" class="gfm gfm-issue">#432</a>)</li>
</ul>
<h3 data-sourcepos="15:1-15:14" dir="auto">
<a id="user-content-technical" class="anchor" href="#technical" aria-hidden="true"></a>technical:</h3>
<ul data-sourcepos="16:1-21:33" dir="auto">
<li data-sourcepos="16:1-16:32">speed up in model setup (<a href="/esde/machine-learning/mlair/-/issues/421" data-reference-type="issue" data-original="#421" data-link="false" data-link-reference="false" data-project="2411" data-issue="15602" data-project-path="esde/machine-learning/mlair" data-iid="421" data-issue-type="issue" data-container="body" data-placement="top" title="Calc number of samples only if needed by model" class="gfm gfm-issue">#421</a>)</li>
<li data-sourcepos="17:1-17:47">bugfix for boundary trim in FIR filter (<a href="/esde/machine-learning/mlair/-/issues/418" data-reference-type="issue" data-original="#418" data-link="false" data-link-reference="false" data-project="2411" data-issue="15570" data-project-path="esde/machine-learning/mlair" data-iid="418" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: wrong valid range in FIR filter" class="gfm gfm-issue">#418</a>)</li>
<li data-sourcepos="18:1-18:53">persistence is now calculated only on demand (<a href="/esde/machine-learning/mlair/-/issues/426" data-reference-type="issue" data-original="#426" data-link="false" data-link-reference="false" data-project="2411" data-issue="15938" data-project-path="esde/machine-learning/mlair" data-iid="426" data-issue-type="issue" data-container="body" data-placement="top" title="enable persi only if requested" class="gfm gfm-issue">#426</a>)</li>
<li data-sourcepos="19:1-19:47">block mse are stored locally in a file (<a href="/esde/machine-learning/mlair/-/issues/428" data-reference-type="issue" data-original="#428" data-link="false" data-link-reference="false" data-project="2411" data-issue="16004" data-project-path="esde/machine-learning/mlair" data-iid="428" data-issue-type="issue" data-container="body" data-placement="top" title="Store block mse results" class="gfm gfm-issue">#428</a>)</li>
<li data-sourcepos="20:1-20:68">fix issue with boolean variables not recognized by argparse (<a href="/esde/machine-learning/mlair/-/issues/417" data-reference-type="issue" data-original="#417" data-link="false" data-link-reference="false" data-project="2411" data-issue="15564" data-project-path="esde/machine-learning/mlair" data-iid="417" data-issue-type="issue" data-container="body" data-placement="top" title="proper boolean conversion" class="gfm gfm-issue">#417</a>)</li>
<li data-sourcepos="21:1-21:33">renaming of ahead labels (<a href="/esde/machine-learning/mlair/-/issues/436" data-reference-type="issue" data-original="#436" data-link="false" data-link-reference="false" data-project="2411" data-issue="16247" data-project-path="esde/machine-learning/mlair" data-iid="436" data-issue-type="issue" data-container="body" data-placement="top" title="update forecast step labels" class="gfm gfm-issue">#436</a>)</li>
</ul>
2022-12-02T15:27:08+01:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v2.2.0
v2.2.0
new data sources and python3.9
<h3 data-sourcepos="1:1-1:12" dir="auto">
<a id="user-content-general" class="anchor" href="#general" aria-hidden="true"></a>general:</h3>
<ul data-sourcepos="2:1-6:0" dir="auto">
<li data-sourcepos="2:1-2:43">new data sources: era5 data and ToarDB V2</li>
<li data-sourcepos="3:1-3:27">CAMS competitor available</li>
<li data-sourcepos="4:1-4:26">improved execution speed</li>
<li data-sourcepos="5:1-6:0">MLAir is now updated to python3.9</li>
</ul>
<h3 data-sourcepos="7:1-7:17" dir="auto">
<a id="user-content-new-features" class="anchor" href="#new-features" aria-hidden="true"></a>new features:</h3>
<ul data-sourcepos="8:1-14:0" dir="auto">
<li data-sourcepos="8:1-8:69">new data loading method to load era5 data on Jülich systems (<a href="/esde/machine-learning/mlair/-/issues/393" data-reference-type="issue" data-original="#393" data-link="false" data-link-reference="false" data-project="2411" data-issue="15047" data-project-path="esde/machine-learning/mlair" data-iid="393" data-issue-type="issue" data-container="body" data-placement="top" title="era5 data" class="gfm gfm-issue">#393</a>)</li>
<li data-sourcepos="9:1-9:60">new data loading method to load data from ToarDB V2 (<a href="/esde/machine-learning/mlair/-/issues/396" data-reference-type="issue" data-original="#396" data-link="false" data-link-reference="false" data-project="2411" data-issue="15216" data-project-path="esde/machine-learning/mlair" data-iid="396" data-issue-type="issue" data-container="body" data-placement="top" title="Load data from ToarDB V2" class="gfm gfm-issue">#396</a>)</li>
<li data-sourcepos="10:1-10:67">implemented competitor model using CAMS ensemble forecasts (<a href="/esde/machine-learning/mlair/-/issues/394" data-reference-type="issue" data-original="#394" data-link="false" data-link-reference="false" data-project="2411" data-issue="15185" data-project-path="esde/machine-learning/mlair" data-iid="394" data-issue-type="issue" data-container="body" data-placement="top" title="CAMS competitor" class="gfm gfm-issue">#394</a>)</li>
<li data-sourcepos="11:1-11:73">OLS competitor is only calculated if provided in competitor list (<a href="/esde/machine-learning/mlair/-/issues/404" data-reference-type="issue" data-original="#404" data-link="false" data-link-reference="false" data-project="2411" data-issue="15383" data-project-path="esde/machine-learning/mlair" data-iid="404" data-issue-type="issue" data-container="body" data-placement="top" title="Skip OLS on request" class="gfm gfm-issue">#404</a>)</li>
<li data-sourcepos="12:1-12:80">experimental: snapshot creation to skip preprocessing stage (<a href="/esde/machine-learning/mlair/-/issues/346" data-reference-type="issue" data-original="#346" data-link="false" data-link-reference="false" data-project="2411" data-issue="13999" data-project-path="esde/machine-learning/mlair" data-iid="346" data-issue-type="issue" data-container="body" data-placement="top" title="experimental: load preprocessing snapshot" class="gfm gfm-issue">#346</a>, <a href="/esde/machine-learning/mlair/-/issues/405" data-reference-type="issue" data-original="#405" data-link="false" data-link-reference="false" data-project="2411" data-issue="15394" data-project-path="esde/machine-learning/mlair" data-iid="405" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: error calc fails as no obs is avail" class="gfm gfm-issue">#405</a>, <a href="/esde/machine-learning/mlair/-/issues/406" data-reference-type="issue" data-original="#406" data-link="false" data-link-reference="false" data-project="2411" data-issue="15397" data-project-path="esde/machine-learning/mlair" data-iid="406" data-issue-type="issue" data-container="body" data-placement="top" title="REFAC: improve snapshot" class="gfm gfm-issue">#406</a>)</li>
<li data-sourcepos="13:1-14:0">new workflow HyperSearchWorkflow stopping after training stage (<a href="/esde/machine-learning/mlair/-/issues/408" data-reference-type="issue" data-original="#408" data-link="false" data-link-reference="false" data-project="2411" data-issue="15399" data-project-path="esde/machine-learning/mlair" data-iid="408" data-issue-type="issue" data-container="body" data-placement="top" title="hypersearch workflow without Postprocessing" class="gfm gfm-issue">#408</a>)</li>
</ul>
<h3 data-sourcepos="15:1-15:14" dir="auto">
<a id="user-content-technical" class="anchor" href="#technical" aria-hidden="true"></a>technical:</h3>
<ul data-sourcepos="16:1-21:35" dir="auto">
<li data-sourcepos="16:1-16:80">fixed minor issues and improved execution speed in postprocessing (<a href="/esde/machine-learning/mlair/-/issues/401" data-reference-type="issue" data-original="#401" data-link="false" data-link-reference="false" data-project="2411" data-issue="15302" data-project-path="esde/machine-learning/mlair" data-iid="401" data-issue-type="issue" data-container="body" data-placement="top" title="missing datetime import" class="gfm gfm-issue">#401</a>, <a href="/esde/machine-learning/mlair/-/issues/413" data-reference-type="issue" data-original="#413" data-link="false" data-link-reference="false" data-project="2411" data-issue="15507" data-project-path="esde/machine-learning/mlair" data-iid="413" data-issue-type="issue" data-container="body" data-placement="top" title="Remove model prediction from normalized loop" class="gfm gfm-issue">#413</a>)</li>
<li data-sourcepos="17:1-17:50">improved speed in keras iterator creation (<a href="/esde/machine-learning/mlair/-/issues/409" data-reference-type="issue" data-original="#409" data-link="false" data-link-reference="false" data-project="2411" data-issue="15402" data-project-path="esde/machine-learning/mlair" data-iid="409" data-issue-type="issue" data-container="body" data-placement="top" title="improve set keras generator speed" class="gfm gfm-issue">#409</a>)</li>
<li data-sourcepos="18:1-18:56">solved bug for very long competitor time series (<a href="/esde/machine-learning/mlair/-/issues/395" data-reference-type="issue" data-original="#395" data-link="false" data-link-reference="false" data-project="2411" data-issue="15213" data-project-path="esde/machine-learning/mlair" data-iid="395" data-issue-type="issue" data-container="body" data-placement="top" title="Bug for long competitor series" class="gfm gfm-issue">#395</a>)</li>
<li data-sourcepos="19:1-19:65">updated python, HPC and CI environment (<a href="/esde/machine-learning/mlair/-/issues/402" data-reference-type="issue" data-original="#402" data-link="false" data-link-reference="false" data-project="2411" data-issue="15348" data-project-path="esde/machine-learning/mlair" data-iid="402" data-issue-type="issue" data-container="body" data-placement="top" title="update HPC environment" class="gfm gfm-issue">#402</a>, <a href="/esde/machine-learning/mlair/-/issues/403" data-reference-type="issue" data-original="#403" data-link="false" data-link-reference="false" data-project="2411" data-issue="15379" data-project-path="esde/machine-learning/mlair" data-iid="403" data-issue-type="issue" data-container="body" data-placement="top" title="TECH: Docs Badge not created" class="gfm gfm-issue">#403</a>, <a href="/esde/machine-learning/mlair/-/issues/407" data-reference-type="issue" data-original="#407" data-link="false" data-link-reference="false" data-project="2411" data-issue="15398" data-project-path="esde/machine-learning/mlair" data-iid="407" data-issue-type="issue" data-container="body" data-placement="top" title="TECH: update geos install in CI Test (from Scratch)" class="gfm gfm-issue">#407</a>, <a href="/esde/machine-learning/mlair/-/issues/410" data-reference-type="issue" data-original="#410" data-link="false" data-link-reference="false" data-project="2411" data-issue="15415" data-project-path="esde/machine-learning/mlair" data-iid="410" data-issue-type="issue" data-container="body" data-placement="top" title="TECH: reduce CI running time" class="gfm gfm-issue">#410</a>)</li>
<li data-sourcepos="20:1-20:40">fix for climateFIR data handler (<a href="/esde/machine-learning/mlair/-/issues/399" data-reference-type="issue" data-original="#399" data-link="false" data-link-reference="false" data-project="2411" data-issue="15283" data-project-path="esde/machine-learning/mlair" data-iid="399" data-issue-type="issue" data-container="body" data-placement="top" title="climateFIR empty data after sel_opts" class="gfm gfm-issue">#399</a>)</li>
<li data-sourcepos="21:1-21:35">fix for report model error (<a href="/esde/machine-learning/mlair/-/issues/416" data-reference-type="issue" data-original="#416" data-link="false" data-link-reference="false" data-project="2411" data-issue="15557" data-project-path="esde/machine-learning/mlair" data-iid="416" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: report model fails on empty list" class="gfm gfm-issue">#416</a>)</li>
</ul>
2022-08-16T11:22:52+02:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v2.1.0
v2.1.0
new evaluation metrics and improved training
<h3 data-sourcepos="1:1-1:12" dir="auto">
<a id="user-content-general" class="anchor" href="#general" aria-hidden="true"></a>general:</h3>
<ul data-sourcepos="2:1-5:0" dir="auto">
<li data-sourcepos="2:1-2:38">new evaluation metrics, IOA and MNMB</li>
<li data-sourcepos="3:1-3:43">advanced train options for early stopping</li>
<li data-sourcepos="4:1-5:0">reduced execution time by refactoring</li>
</ul>
<h3 data-sourcepos="6:1-6:17" dir="auto">
<a id="user-content-new-features" class="anchor" href="#new-features" aria-hidden="true"></a>new features:</h3>
<ul data-sourcepos="7:1-13:0" dir="auto">
<li data-sourcepos="7:1-7:80">uncertainty estimation of MSE is now applied for each season separately (<a href="/esde/machine-learning/mlair/-/issues/374" data-reference-type="issue" data-original="#374" data-link="false" data-link-reference="false" data-project="2411" data-issue="14596" data-project-path="esde/machine-learning/mlair" data-iid="374" data-issue-type="issue" data-container="body" data-placement="top" title="uncertainty estimate on seasons" class="gfm gfm-issue">#374</a>)</li>
<li data-sourcepos="8:1-8:98">added different configurations of early stopping to use either last trained or best epoch (<a href="/esde/machine-learning/mlair/-/issues/378" data-reference-type="issue" data-original="#378" data-link="false" data-link-reference="false" data-project="2411" data-issue="14717" data-project-path="esde/machine-learning/mlair" data-iid="378" data-issue-type="issue" data-container="body" data-placement="top" title="disable/enable early stopping" class="gfm gfm-issue">#378</a>)</li>
<li data-sourcepos="9:1-9:87">train monitoring plots now add a star for best epoch when using early stopping (<a href="/esde/machine-learning/mlair/-/issues/367" data-reference-type="issue" data-original="#367" data-link="false" data-link-reference="false" data-project="2411" data-issue="14421" data-project-path="esde/machine-learning/mlair" data-iid="367" data-issue-type="issue" data-container="body" data-placement="top" title="loss plot with best result marker" class="gfm gfm-issue">#367</a>)</li>
<li data-sourcepos="10:1-10:54">new evaluation metric index of agreement, IOA (<a href="/esde/machine-learning/mlair/-/issues/376" data-reference-type="issue" data-original="#376" data-link="false" data-link-reference="false" data-project="2411" data-issue="14714" data-project-path="esde/machine-learning/mlair" data-iid="376" data-issue-type="issue" data-container="body" data-placement="top" title="add metric IOA" class="gfm gfm-issue">#376</a>)</li>
<li data-sourcepos="11:1-11:66">new evaluation metric modified normalised mean bias, MNMB (<a href="/esde/machine-learning/mlair/-/issues/380" data-reference-type="issue" data-original="#380" data-link="false" data-link-reference="false" data-project="2411" data-issue="14831" data-project-path="esde/machine-learning/mlair" data-iid="380" data-issue-type="issue" data-container="body" data-placement="top" title="add metric MNMB" class="gfm gfm-issue">#380</a>)</li>
<li data-sourcepos="12:1-13:0">new plot available that shows temporal evolution of MSE for each station (<a href="/esde/machine-learning/mlair/-/issues/381" data-reference-type="issue" data-original="#381" data-link="false" data-link-reference="false" data-project="2411" data-issue="14832" data-project-path="esde/machine-learning/mlair" data-iid="381" data-issue-type="issue" data-container="body" data-placement="top" title="time evolution of metrics" class="gfm gfm-issue">#381</a>)</li>
</ul>
<h3 data-sourcepos="14:1-14:14" dir="auto">
<a id="user-content-technical" class="anchor" href="#technical" aria-hidden="true"></a>technical:</h3>
<ul data-sourcepos="15:1-20:78" dir="auto">
<li data-sourcepos="15:1-15:57">reduced loading of forecast path from data store (<a href="/esde/machine-learning/mlair/-/issues/328" data-reference-type="issue" data-original="#328" data-link="false" data-link-reference="false" data-project="2411" data-issue="13163" data-project-path="esde/machine-learning/mlair" data-iid="328" data-issue-type="issue" data-container="body" data-placement="top" title="REFAC: reduce forecast path loading" class="gfm gfm-issue">#328</a>)</li>
<li data-sourcepos="16:1-16:60">bug fix for not catched error during transformation (<a href="/esde/machine-learning/mlair/-/issues/385" data-reference-type="issue" data-original="#385" data-link="false" data-link-reference="false" data-project="2411" data-issue="14952" data-project-path="esde/machine-learning/mlair" data-iid="385" data-issue-type="issue" data-container="body" data-placement="top" title="transformation fails with IndexError" class="gfm gfm-issue">#385</a>)</li>
<li data-sourcepos="17:1-17:120">bug fix for data handler with climate and fir filter leading to calculate transformation always with fir filter (<a href="/esde/machine-learning/mlair/-/issues/387" data-reference-type="issue" data-original="#387" data-link="false" data-link-reference="false" data-project="2411" data-issue="14986" data-project-path="esde/machine-learning/mlair" data-iid="387" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: on transformation for climate and fir clim mix" class="gfm gfm-issue">#387</a>)</li>
<li data-sourcepos="18:1-18:76">improved duration for latex report creation at end of preprocessing (<a href="/esde/machine-learning/mlair/-/issues/388" data-reference-type="issue" data-original="#388" data-link="false" data-link-reference="false" data-project="2411" data-issue="14988" data-project-path="esde/machine-learning/mlair" data-iid="388" data-issue-type="issue" data-container="body" data-placement="top" title="parallel create latex report" class="gfm gfm-issue">#388</a>)</li>
<li data-sourcepos="19:1-19:61">enhanced speed for make prediction in postprocessing (<a href="/esde/machine-learning/mlair/-/issues/389" data-reference-type="issue" data-original="#389" data-link="false" data-link-reference="false" data-project="2411" data-issue="14989" data-project-path="esde/machine-learning/mlair" data-iid="389" data-issue-type="issue" data-container="body" data-placement="top" title="parallel make_prediction in postprocessing" class="gfm gfm-issue">#389</a>)</li>
<li data-sourcepos="20:1-20:78">fix to always create version badge from version and not from tag name (<a href="/esde/machine-learning/mlair/-/issues/382" data-reference-type="issue" data-original="#382" data-link="false" data-link-reference="false" data-project="2411" data-issue="14833" data-project-path="esde/machine-learning/mlair" data-iid="382" data-issue-type="issue" data-container="body" data-placement="top" title="fix version badge" class="gfm gfm-issue">#382</a>)</li>
</ul>
2022-06-08T12:39:16+02:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/Kleinert_etal_2022_initial_submission
Kleinert_etal_2022_initial_submission
Initial version submitted to GMD
Felix Kleinert
f.kleinert@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v2.0.0
v2.0.0
tf2 usage, new model classes, and improved uncertainty estimate
<h3 data-sourcepos="1:1-1:12" dir="auto">
<a id="user-content-general" class="anchor" href="#general" aria-hidden="true"></a>general:</h3>
<ul data-sourcepos="2:1-5:0" dir="auto">
<li data-sourcepos="2:1-2:30">MLAir now uses tensorflow v2</li>
<li data-sourcepos="3:1-3:48">new customisable model classes for CNN and RNN</li>
<li data-sourcepos="4:1-5:0">improved uncertainty estimate</li>
</ul>
<h3 data-sourcepos="6:1-6:17" dir="auto">
<a id="user-content-new-features" class="anchor" href="#new-features" aria-hidden="true"></a>new features:</h3>
<ul data-sourcepos="7:1-18:0" dir="auto">
<li data-sourcepos="7:1-7:43">MLAir depends now on tensorflow v2 (<a href="/esde/machine-learning/mlair/-/issues/331" data-reference-type="issue" data-original="#331" data-link="false" data-link-reference="false" data-project="2411" data-issue="13541" data-project-path="esde/machine-learning/mlair" data-iid="331" data-issue-type="issue" data-container="body" data-placement="top" title="upgrade code to TensorFlow V2" class="gfm gfm-issue">#331</a>)</li>
<li data-sourcepos="8:1-8:56">new CNN class that can be configured layer-wise (<a href="/esde/machine-learning/mlair/-/issues/368" data-reference-type="issue" data-original="#368" data-link="false" data-link-reference="false" data-project="2411" data-issue="14428" data-project-path="esde/machine-learning/mlair" data-iid="368" data-issue-type="issue" data-container="body" data-placement="top" title="prepare CNN class for filter benchmarking" class="gfm gfm-issue">#368</a>)</li>
<li data-sourcepos="9:1-9:60">new RNN class that can be configured in more detail (<a href="/esde/machine-learning/mlair/-/issues/361" data-reference-type="issue" data-original="#361" data-link="false" data-link-reference="false" data-project="2411" data-issue="14342" data-project-path="esde/machine-learning/mlair" data-iid="361" data-issue-type="issue" data-container="body" data-placement="top" title="custom dense layers in rnn" class="gfm gfm-issue">#361</a>)</li>
<li data-sourcepos="10:1-10:37">new branched-input CNN class (<a href="/esde/machine-learning/mlair/-/issues/368" data-reference-type="issue" data-original="#368" data-link="false" data-link-reference="false" data-project="2411" data-issue="14428" data-project-path="esde/machine-learning/mlair" data-iid="368" data-issue-type="issue" data-container="body" data-placement="top" title="prepare CNN class for filter benchmarking" class="gfm gfm-issue">#368</a>)</li>
<li data-sourcepos="11:1-11:37">new branched-input RNN class (<a href="/esde/machine-learning/mlair/-/issues/362" data-reference-type="issue" data-original="#362" data-link="false" data-link-reference="false" data-project="2411" data-issue="14378" data-project-path="esde/machine-learning/mlair" data-iid="362" data-issue-type="issue" data-container="body" data-placement="top" title="branched rnn model class" class="gfm gfm-issue">#362</a>)</li>
<li data-sourcepos="12:1-12:60">set custom model display name that is used in plots (<a href="/esde/machine-learning/mlair/-/issues/341" data-reference-type="issue" data-original="#341" data-link="false" data-link-reference="false" data-project="2411" data-issue="13846" data-project-path="esde/machine-learning/mlair" data-iid="341" data-issue-type="issue" data-container="body" data-placement="top" title="define custom model name for plots and skip batch creation if no train" class="gfm gfm-issue">#341</a>)</li>
<li data-sourcepos="13:1-13:75">specify names of input branches to use in feature importance plots (<a href="/esde/machine-learning/mlair/-/issues/356" data-reference-type="issue" data-original="#356" data-link="false" data-link-reference="false" data-project="2411" data-issue="14202" data-project-path="esde/machine-learning/mlair" data-iid="356" data-issue-type="issue" data-container="body" data-placement="top" title="can define names of branches to use in feature importance" class="gfm gfm-issue">#356</a>)</li>
<li data-sourcepos="14:1-14:98">uncertainty estimate of model error is now calculated for each forecast step additionally (<a href="/esde/machine-learning/mlair/-/issues/359" data-reference-type="issue" data-original="#359" data-link="false" data-link-reference="false" data-project="2411" data-issue="14237" data-project-path="esde/machine-learning/mlair" data-iid="359" data-issue-type="issue" data-container="body" data-placement="top" title="separate uncertainty estimate for each forecast step" class="gfm gfm-issue">#359</a>)</li>
<li data-sourcepos="15:1-15:99">data transformation properties are stored locally and can be loaded into an experiment run (<a href="/esde/machine-learning/mlair/-/issues/345" data-reference-type="issue" data-original="#345" data-link="false" data-link-reference="false" data-project="2411" data-issue="13976" data-project-path="esde/machine-learning/mlair" data-iid="345" data-issue-type="issue" data-container="body" data-placement="top" title="store transformation locally as file" class="gfm gfm-issue">#345</a>)</li>
<li data-sourcepos="16:1-16:69">uncertainty estimate includes now a Mann-Whitney U rank test (<a href="/esde/machine-learning/mlair/-/issues/355" data-reference-type="issue" data-original="#355" data-link="false" data-link-reference="false" data-project="2411" data-issue="14193" data-project-path="esde/machine-learning/mlair" data-iid="355" data-issue-type="issue" data-container="body" data-placement="top" title="Include Mann-Whitney U rank test" class="gfm gfm-issue">#355</a>)</li>
<li data-sourcepos="17:1-18:0">data handlers can now have access to "future" data specified by new parameter extend_length_opts (<a href="/esde/machine-learning/mlair/-/issues/339" data-reference-type="issue" data-original="#339" data-link="false" data-link-reference="false" data-project="2411" data-issue="13807" data-project-path="esde/machine-learning/mlair" data-iid="339" data-issue-type="issue" data-container="body" data-placement="top" title="filter with future mix" class="gfm gfm-issue">#339</a>)</li>
</ul>
<h3 data-sourcepos="19:1-19:14" dir="auto">
<a id="user-content-technical" class="anchor" href="#technical" aria-hidden="true"></a>technical:</h3>
<ul data-sourcepos="20:1-38:88" dir="auto">
<li data-sourcepos="20:1-20:56">MLAir now uses python3.8 on Jülich HPC systems (<a href="/esde/machine-learning/mlair/-/issues/375" data-reference-type="issue" data-original="#375" data-link="false" data-link-reference="false" data-project="2411" data-issue="14640" data-project-path="esde/machine-learning/mlair" data-iid="375" data-issue-type="issue" data-container="body" data-placement="top" title="update python version in setup venv" class="gfm gfm-issue">#375</a>)</li>
<li data-sourcepos="21:1-21:69">no support of MLAir for tensorflow v1.X, replaced by tf v2.X (<a href="/esde/machine-learning/mlair/-/issues/331" data-reference-type="issue" data-original="#331" data-link="false" data-link-reference="false" data-project="2411" data-issue="13541" data-project-path="esde/machine-learning/mlair" data-iid="331" data-issue-type="issue" data-container="body" data-placement="top" title="upgrade code to TensorFlow V2" class="gfm gfm-issue">#331</a>)</li>
<li data-sourcepos="22:1-22:67">all data handlers with filters can return data as branches (<a href="/esde/machine-learning/mlair/-/issues/370" data-reference-type="issue" data-original="#370" data-link="false" data-link-reference="false" data-project="2411" data-issue="14467" data-project-path="esde/machine-learning/mlair" data-iid="370" data-issue-type="issue" data-container="body" data-placement="top" title="make use filter branches available for all dh with filters" class="gfm gfm-issue">#370</a>)</li>
<li data-sourcepos="23:1-23:76">bug fix to force model name and competitor names to be unique (<a href="/esde/machine-learning/mlair/-/issues/366" data-reference-type="issue" data-original="#366" data-link="false" data-link-reference="false" data-project="2411" data-issue="14417" data-project-path="esde/machine-learning/mlair" data-iid="366" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: workflow failure on HPC during clim skill score calcuation" class="gfm gfm-issue">#366</a>, <a href="/esde/machine-learning/mlair/-/issues/369" data-reference-type="issue" data-original="#369" data-link="false" data-link-reference="false" data-project="2411" data-issue="14466" data-project-path="esde/machine-learning/mlair" data-iid="369" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: crash when no competitor is provided" class="gfm gfm-issue">#369</a>)</li>
<li data-sourcepos="24:1-24:47">fix to use only a single forecast step (<a href="/esde/machine-learning/mlair/-/issues/315" data-reference-type="issue" data-original="#315" data-link="false" data-link-reference="false" data-project="2411" data-issue="12339" data-project-path="esde/machine-learning/mlair" data-iid="315" data-issue-type="issue" data-container="body" data-placement="top" title="EnableWindowLeadTime_1" class="gfm gfm-issue">#315</a>)</li>
<li data-sourcepos="25:1-25:38">CI pipeline adjustments (<a href="/esde/machine-learning/mlair/-/issues/340" data-reference-type="issue" data-original="#340" data-link="false" data-link-reference="false" data-project="2411" data-issue="13815" data-project-path="esde/machine-learning/mlair" data-iid="340" data-issue-type="issue" data-container="body" data-placement="top" title="error when calling tests from scratch" class="gfm gfm-issue">#340</a>, <a href="/esde/machine-learning/mlair/-/issues/365" data-reference-type="issue" data-original="#365" data-link="false" data-link-reference="false" data-project="2411" data-issue="14415" data-project-path="esde/machine-learning/mlair" data-iid="365" data-issue-type="issue" data-container="body" data-placement="top" title="TECH: set CI/CD artifact expire time" class="gfm gfm-issue">#365</a>)</li>
<li data-sourcepos="26:1-26:57">new option to set the level of the print logging (<a href="/esde/machine-learning/mlair/-/issues/364" data-reference-type="issue" data-original="#364" data-link="false" data-link-reference="false" data-project="2411" data-issue="14389" data-project-path="esde/machine-learning/mlair" data-iid="364" data-issue-type="issue" data-container="body" data-placement="top" title="set logging print mode" class="gfm gfm-issue">#364</a>)</li>
<li data-sourcepos="27:1-27:77">advanced logging for batch data creation and in postprocessing (<a href="/esde/machine-learning/mlair/-/issues/350" data-reference-type="issue" data-original="#350" data-link="false" data-link-reference="false" data-project="2411" data-issue="14046" data-project-path="esde/machine-learning/mlair" data-iid="350" data-issue-type="issue" data-container="body" data-placement="top" title="add log if batch creation fails" class="gfm gfm-issue">#350</a>, <a href="/esde/machine-learning/mlair/-/issues/360" data-reference-type="issue" data-original="#360" data-link="false" data-link-reference="false" data-project="2411" data-issue="14326" data-project-path="esde/machine-learning/mlair" data-iid="360" data-issue-type="issue" data-container="body" data-placement="top" title="add more logging to postprocessing" class="gfm gfm-issue">#360</a>)</li>
<li data-sourcepos="28:1-28:60">batch data creation is skipped on disabled training (<a href="/esde/machine-learning/mlair/-/issues/341" data-reference-type="issue" data-original="#341" data-link="false" data-link-reference="false" data-project="2411" data-issue="13846" data-project-path="esde/machine-learning/mlair" data-iid="341" data-issue-type="issue" data-container="body" data-placement="top" title="define custom model name for plots and skip batch creation if no train" class="gfm gfm-issue">#341</a>)</li>
<li data-sourcepos="29:1-29:54">multiprocessing pools are now closed properly (<a href="/esde/machine-learning/mlair/-/issues/342" data-reference-type="issue" data-original="#342" data-link="false" data-link-reference="false" data-project="2411" data-issue="13859" data-project-path="esde/machine-learning/mlair" data-iid="342" data-issue-type="issue" data-container="body" data-placement="top" title="free the workers" class="gfm gfm-issue">#342</a>)</li>
<li data-sourcepos="30:1-30:51">bug fix if no competitor data is available (<a href="/esde/machine-learning/mlair/-/issues/343" data-reference-type="issue" data-original="#343" data-link="false" data-link-reference="false" data-project="2411" data-issue="13898" data-project-path="esde/machine-learning/mlair" data-iid="343" data-issue-type="issue" data-container="body" data-placement="top" title="fix bugs caused by model name refac and tf update" class="gfm gfm-issue">#343</a>)</li>
<li data-sourcepos="31:1-31:34">bug fix for model loading (<a href="/esde/machine-learning/mlair/-/issues/343" data-reference-type="issue" data-original="#343" data-link="false" data-link-reference="false" data-project="2411" data-issue="13898" data-project-path="esde/machine-learning/mlair" data-iid="343" data-issue-type="issue" data-container="body" data-placement="top" title="fix bugs caused by model name refac and tf update" class="gfm gfm-issue">#343</a>)</li>
<li data-sourcepos="32:1-32:92">models plotted by PlotSampleUncertaintyFromBootstrap are now ordered by mean error (<a href="/esde/machine-learning/mlair/-/issues/344" data-reference-type="issue" data-original="#344" data-link="false" data-link-reference="false" data-project="2411" data-issue="13926" data-project-path="esde/machine-learning/mlair" data-iid="344" data-issue-type="issue" data-container="body" data-placement="top" title="sorted uncertainty estimate plot" class="gfm gfm-issue">#344</a>)</li>
<li data-sourcepos="33:1-33:71">fix for usage of lazy data caused unintended reloading of data (<a href="/esde/machine-learning/mlair/-/issues/347" data-reference-type="issue" data-original="#347" data-link="false" data-link-reference="false" data-project="2411" data-issue="14004" data-project-path="esde/machine-learning/mlair" data-iid="347" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: wrong time extend when using lazy preprocessing" class="gfm gfm-issue">#347</a>)</li>
<li data-sourcepos="34:1-34:70">fix for latex reports no showing all stations and competitors (<a href="/esde/machine-learning/mlair/-/issues/349" data-reference-type="issue" data-original="#349" data-link="false" data-link-reference="false" data-project="2411" data-issue="14035" data-project-path="esde/machine-learning/mlair" data-iid="349" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: competitors not appearing in latex reports" class="gfm gfm-issue">#349</a>)</li>
<li data-sourcepos="35:1-35:78">refactoring of hard coded dimension names in skill scores calculation (<a href="/esde/machine-learning/mlair/-/issues/357" data-reference-type="issue" data-original="#357" data-link="false" data-link-reference="false" data-project="2411" data-issue="14224" data-project-path="esde/machine-learning/mlair" data-iid="357" data-issue-type="issue" data-container="body" data-placement="top" title="REFAC: remove hard coded type dim from skill score" class="gfm gfm-issue">#357</a>)</li>
<li data-sourcepos="36:1-36:93">bug fix of order of bootstrap method in feature importance calculation causes errors (<a href="/esde/machine-learning/mlair/-/issues/358" data-reference-type="issue" data-original="#358" data-link="false" data-link-reference="false" data-project="2411" data-issue="14233" data-project-path="esde/machine-learning/mlair" data-iid="358" data-issue-type="issue" data-container="body" data-placement="top" title="Bug: order in feature_importance_bootstrap_method causes crash" class="gfm gfm-issue">#358</a>)</li>
<li data-sourcepos="37:1-38:88">distinguish now between window_history_offset (pos of last time step), window_history_size (total length of input
sample), and extend_length_opts ("future" data that is available at given time) (<a href="/esde/machine-learning/mlair/-/issues/353" data-reference-type="issue" data-original="#353" data-link="false" data-link-reference="false" data-project="2411" data-issue="14095" data-project-path="esde/machine-learning/mlair" data-iid="353" data-issue-type="issue" data-container="body" data-placement="top" title="filter dh: differentiate between history offset and information offset" class="gfm gfm-issue">#353</a>)</li>
</ul>
2022-04-11T12:11:35+02:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v1.5.0
v1.5.0
new uncertainty estimation
<h3 data-sourcepos="1:1-1:12" dir="auto">
<a id="user-content-general" class="anchor" href="#general" aria-hidden="true"></a>general:</h3>
<ul data-sourcepos="2:1-5:0" dir="auto">
<li data-sourcepos="2:1-2:50">introduces method to estimate sample uncertainty</li>
<li data-sourcepos="3:1-3:26">improved multiprocessing</li>
<li data-sourcepos="4:1-5:0">last release with tensorflow v1 support</li>
</ul>
<h3 data-sourcepos="6:1-6:17" dir="auto">
<a id="user-content-new-features" class="anchor" href="#new-features" aria-hidden="true"></a>new features:</h3>
<ul data-sourcepos="7:1-9:0" dir="auto">
<li data-sourcepos="7:1-7:68">test set sample uncertainty estmation during postprocessing (<a href="/esde/machine-learning/mlair/-/issues/333" data-reference-type="issue" data-original="#333" data-link="false" data-link-reference="false" data-project="2411" data-issue="13583" data-project-path="esde/machine-learning/mlair" data-iid="333" data-issue-type="issue" data-container="body" data-placement="top" title="Test Set Sample Uncertainty in PostProcessing" class="gfm gfm-issue">#333</a>)</li>
<li data-sourcepos="8:1-9:0">support of Kolmogorov Zurbenko filter for data handlers with filters (<a href="/esde/machine-learning/mlair/-/issues/334" data-reference-type="issue" data-original="#334" data-link="false" data-link-reference="false" data-project="2411" data-issue="13616" data-project-path="esde/machine-learning/mlair" data-iid="334" data-issue-type="issue" data-container="body" data-placement="top" title="make kzf filter running for climate fir filterc data handler" class="gfm gfm-issue">#334</a>)</li>
</ul>
<h3 data-sourcepos="10:1-10:14" dir="auto">
<a id="user-content-technical" class="anchor" href="#technical" aria-hidden="true"></a>technical:</h3>
<ul data-sourcepos="11:1-15:70" dir="auto">
<li data-sourcepos="11:1-11:59">new communication scheme for multiprocessing (<a href="/esde/machine-learning/mlair/-/issues/321" data-reference-type="issue" data-original="#321" data-link="false" data-link-reference="false" data-project="2411" data-issue="13045" data-project-path="esde/machine-learning/mlair" data-iid="321" data-issue-type="issue" data-container="body" data-placement="top" title="too big data for memory" class="gfm gfm-issue">#321</a>, <a href="/esde/machine-learning/mlair/-/issues/322" data-reference-type="issue" data-original="#322" data-link="false" data-link-reference="false" data-project="2411" data-issue="13048" data-project-path="esde/machine-learning/mlair" data-iid="322" data-issue-type="issue" data-container="body" data-placement="top" title="REFAC: reduce communication in multiprocessing" class="gfm gfm-issue">#322</a>)</li>
<li data-sourcepos="12:1-12:33">improved error reporting (<a href="/esde/machine-learning/mlair/-/issues/323" data-reference-type="issue" data-original="#323" data-link="false" data-link-reference="false" data-project="2411" data-issue="13053" data-project-path="esde/machine-learning/mlair" data-iid="323" data-issue-type="issue" data-container="body" data-placement="top" title="raise ValueError in remove_nan(), if data is NaN only" class="gfm gfm-issue">#323</a>)</li>
<li data-sourcepos="13:1-13:60">feature importance returns now unaggregated results (<a href="/esde/machine-learning/mlair/-/issues/335" data-reference-type="issue" data-original="#335" data-link="false" data-link-reference="false" data-project="2411" data-issue="13644" data-project-path="esde/machine-learning/mlair" data-iid="335" data-issue-type="issue" data-container="body" data-placement="top" title="Feature Importance use unaggregated results" class="gfm gfm-issue">#335</a>)</li>
<li data-sourcepos="14:1-14:55">error metrics are reported for all competitors (<a href="/esde/machine-learning/mlair/-/issues/332" data-reference-type="issue" data-original="#332" data-link="false" data-link-reference="false" data-project="2411" data-issue="13557" data-project-path="esde/machine-learning/mlair" data-iid="332" data-issue-type="issue" data-container="body" data-placement="top" title="report error metrics for all competitors" class="gfm gfm-issue">#332</a>)</li>
<li data-sourcepos="15:1-15:70">minor bugfixes and refacs (<a href="/esde/machine-learning/mlair/-/issues/330" data-reference-type="issue" data-original="#330" data-link="false" data-link-reference="false" data-project="2411" data-issue="13459" data-project-path="esde/machine-learning/mlair" data-iid="330" data-issue-type="issue" data-container="body" data-placement="top" title="BUGFIX: calculate bootstraps" class="gfm gfm-issue">#330</a>, <a href="/esde/machine-learning/mlair/-/issues/326" data-reference-type="issue" data-original="#326" data-link="false" data-link-reference="false" data-project="2411" data-issue="13124" data-project-path="esde/machine-learning/mlair" data-iid="326" data-issue-type="issue" data-container="body" data-placement="top" title="REFAC: do not stop if filter plots fail" class="gfm gfm-issue">#326</a>, <a href="/esde/machine-learning/mlair/-/issues/329" data-reference-type="issue" data-original="#329" data-link="false" data-link-reference="false" data-project="2411" data-issue="13170" data-project-path="esde/machine-learning/mlair" data-iid="329" data-issue-type="issue" data-container="body" data-placement="top" title="REFAC: individual trafo for unfiltered data" class="gfm gfm-issue">#329</a>, <a href="/esde/machine-learning/mlair/-/issues/325" data-reference-type="issue" data-original="#325" data-link="false" data-link-reference="false" data-project="2411" data-issue="13116" data-project-path="esde/machine-learning/mlair" data-iid="325" data-issue-type="issue" data-container="body" data-placement="top" title="REFAC: split overwriting of lazy data and raw data" class="gfm gfm-issue">#325</a>, <a href="/esde/machine-learning/mlair/-/issues/324" data-reference-type="issue" data-original="#324" data-link="false" data-link-reference="false" data-project="2411" data-issue="13085" data-project-path="esde/machine-learning/mlair" data-iid="324" data-issue-type="issue" data-container="body" data-placement="top" title="Fix problem in multiply extremes" class="gfm gfm-issue">#324</a>, <a href="/esde/machine-learning/mlair/-/issues/320" data-reference-type="issue" data-original="#320" data-link="false" data-link-reference="false" data-project="2411" data-issue="13027" data-project-path="esde/machine-learning/mlair" data-iid="320" data-issue-type="issue" data-container="body" data-placement="top" title="conflict of lazy preprocessing and overwrite local data" class="gfm gfm-issue">#320</a>, <a href="/esde/machine-learning/mlair/-/issues/337" data-reference-type="issue" data-original="#337" data-link="false" data-link-reference="false" data-project="2411" data-issue="13691" data-project-path="esde/machine-learning/mlair" data-iid="337" data-issue-type="issue" data-container="body" data-placement="top" title="CI pipeline fails for docs" class="gfm gfm-issue">#337</a>)</li>
</ul>
2022-04-11T12:09:16+02:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v1.4.0
v1.4.0
new model classes and data handlers, improved usability and transparency
<h3 data-sourcepos="1:1-1:12" dir="auto">
<a id="user-content-general" class="anchor" href="#general" aria-hidden="true"></a>general:</h3>
<ul data-sourcepos="2:1-5:0" dir="auto">
<li data-sourcepos="2:1-2:75">many technical adjustments to improve usability and transparency of MLAir</li>
<li data-sourcepos="3:1-3:52">new FCN and CNN classes for easy NN model creation</li>
<li data-sourcepos="4:1-5:0">new plots</li>
</ul>
<h3 data-sourcepos="6:1-6:17" dir="auto">
<a id="user-content-new-features" class="anchor" href="#new-features" aria-hidden="true"></a>new features:</h3>
<ul data-sourcepos="7:1-14:0" dir="auto">
<li data-sourcepos="7:1-7:58">new FCN class that can be customized in many ways (<a href="/esde/machine-learning/mlair/-/issues/284" data-reference-type="issue" data-original="#284" data-link="false" data-link-reference="false" data-project="2411" data-issue="10819" data-project-path="esde/machine-learning/mlair" data-iid="284" data-issue-type="issue" data-container="body" data-placement="top" title="create FCN model class" class="gfm gfm-issue">#284</a>)</li>
<li data-sourcepos="8:1-8:27">also new CNN class (<a href="/esde/machine-learning/mlair/-/issues/289" data-reference-type="issue" data-original="#289" data-link="false" data-link-reference="false" data-project="2411" data-issue="10979" data-project-path="esde/machine-learning/mlair" data-iid="289" data-issue-type="issue" data-container="body" data-placement="top" title="create CNN model class" class="gfm gfm-issue">#289</a>)</li>
<li data-sourcepos="9:1-9:64">added new bootstrap analysis method: mean bootstrapping (<a href="/esde/machine-learning/mlair/-/issues/300" data-reference-type="issue" data-original="#300" data-link="false" data-link-reference="false" data-project="2411" data-issue="11302" data-project-path="esde/machine-learning/mlair" data-iid="300" data-issue-type="issue" data-container="body" data-placement="top" title="mean bootstrapping" class="gfm gfm-issue">#300</a>)</li>
<li data-sourcepos="10:1-10:43">new data handler using FIR filters (<a href="/esde/machine-learning/mlair/-/issues/306" data-reference-type="issue" data-original="#306" data-link="false" data-link-reference="false" data-project="2411" data-issue="11492" data-project-path="esde/machine-learning/mlair" data-iid="306" data-issue-type="issue" data-container="body" data-placement="top" title="data handler FIR filter" class="gfm gfm-issue">#306</a>)</li>
<li data-sourcepos="11:1-11:59">performance measures are now stored in local files (<a href="/esde/machine-learning/mlair/-/issues/286" data-reference-type="issue" data-original="#286" data-link="false" data-link-reference="false" data-project="2411" data-issue="10873" data-project-path="esde/machine-learning/mlair" data-iid="286" data-issue-type="issue" data-container="body" data-placement="top" title="store performance measures" class="gfm gfm-issue">#286</a>)</li>
<li data-sourcepos="12:1-12:47">histogram plots for inputs and targets (<a href="/esde/machine-learning/mlair/-/issues/299" data-reference-type="issue" data-original="#299" data-link="false" data-link-reference="false" data-project="2411" data-issue="11301" data-project-path="esde/machine-learning/mlair" data-iid="299" data-issue-type="issue" data-container="body" data-placement="top" title="histogram of inputs and targets" class="gfm gfm-issue">#299</a>)</li>
<li data-sourcepos="13:1-14:0">periodogram plots for filtered data (<a href="/esde/machine-learning/mlair/-/issues/298" data-reference-type="issue" data-original="#298" data-link="false" data-link-reference="false" data-project="2411" data-issue="11300" data-project-path="esde/machine-learning/mlair" data-iid="298" data-issue-type="issue" data-container="body" data-placement="top" title="periodogram for kzf" class="gfm gfm-issue">#298</a>)</li>
</ul>
<h3 data-sourcepos="15:1-15:14" dir="auto">
<a id="user-content-technical" class="anchor" href="#technical" aria-hidden="true"></a>technical:</h3>
<ul data-sourcepos="16:1-31:53" dir="auto">
<li data-sourcepos="16:1-16:117">a calling run script can be stored inside experiment folder if reference to this script is parsed as argument (<a href="/esde/machine-learning/mlair/-/issues/99" data-reference-type="issue" data-original="#99" data-link="false" data-link-reference="false" data-project="2411" data-issue="8327" data-project-path="esde/machine-learning/mlair" data-iid="99" data-issue-type="issue" data-container="body" data-placement="top" title="Store run script in experiment dir" class="gfm gfm-issue">#99</a>)</li>
<li data-sourcepos="17:1-17:44">new callback to track epoch-runtime (<a href="/esde/machine-learning/mlair/-/issues/312" data-reference-type="issue" data-original="#312" data-link="false" data-link-reference="false" data-project="2411" data-issue="12204" data-project-path="esde/machine-learning/mlair" data-iid="312" data-issue-type="issue" data-container="body" data-placement="top" title="Implement Callback to track epoch-runtime" class="gfm gfm-issue">#312</a>)</li>
<li data-sourcepos="18:1-18:44">added switch to use multiprocessing (<a href="/esde/machine-learning/mlair/-/issues/297" data-reference-type="issue" data-original="#297" data-link="false" data-link-reference="false" data-project="2411" data-issue="11227" data-project-path="esde/machine-learning/mlair" data-iid="297" data-issue-type="issue" data-container="body" data-placement="top" title="add switch to use multiprocessing" class="gfm gfm-issue">#297</a>)</li>
<li data-sourcepos="19:1-19:55">customize maximum number of parallel processes (<a href="/esde/machine-learning/mlair/-/issues/308" data-reference-type="issue" data-original="#308" data-link="false" data-link-reference="false" data-project="2411" data-issue="11882" data-project-path="esde/machine-learning/mlair" data-iid="308" data-issue-type="issue" data-container="body" data-placement="top" title="TECH: max number of parallel processes" class="gfm gfm-issue">#308</a>)</li>
<li data-sourcepos="20:1-20:48">support non-monotonic window lead times (<a href="/esde/machine-learning/mlair/-/issues/313" data-reference-type="issue" data-original="#313" data-link="false" data-link-reference="false" data-project="2411" data-issue="12244" data-project-path="esde/machine-learning/mlair" data-iid="313" data-issue-type="issue" data-container="body" data-placement="top" title="allow non-monotonic window lead times in helpers/statistics.py ahead_names definition" class="gfm gfm-issue">#313</a>)</li>
<li data-sourcepos="21:1-21:42">resolved bug with FileExistsError (<a href="/esde/machine-learning/mlair/-/issues/311" data-reference-type="issue" data-original="#311" data-link="false" data-link-reference="false" data-project="2411" data-issue="12191" data-project-path="esde/machine-learning/mlair" data-iid="311" data-issue-type="issue" data-container="body" data-placement="top" title="FileExistsError in Cleanup / DataHandler Build" class="gfm gfm-issue">#311</a>)</li>
<li data-sourcepos="22:1-22:51">resolved bug if no chemical is used at all (<a href="/esde/machine-learning/mlair/-/issues/307" data-reference-type="issue" data-original="#307" data-link="false" data-link-reference="false" data-project="2411" data-issue="11508" data-project-path="esde/machine-learning/mlair" data-iid="307" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: data handler fails if no chem variable is used" class="gfm gfm-issue">#307</a>)</li>
<li data-sourcepos="23:1-23:51">min/max scaler now scales between -1 and 1 (<a href="/esde/machine-learning/mlair/-/issues/302" data-reference-type="issue" data-original="#302" data-link="false" data-link-reference="false" data-project="2411" data-issue="11363" data-project-path="esde/machine-learning/mlair" data-iid="302" data-issue-type="issue" data-container="body" data-placement="top" title="adjust min/max scaler" class="gfm gfm-issue">#302</a>)</li>
<li data-sourcepos="24:1-24:62">added missing offset parameter to some data handlers (<a href="/esde/machine-learning/mlair/-/issues/305" data-reference-type="issue" data-original="#305" data-link="false" data-link-reference="false" data-project="2411" data-issue="11398" data-project-path="esde/machine-learning/mlair" data-iid="305" data-issue-type="issue" data-container="body" data-placement="top" title="add offset to Seperation of Scales data handler" class="gfm gfm-issue">#305</a>)</li>
<li data-sourcepos="25:1-25:36">improved data store logging (<a href="/esde/machine-learning/mlair/-/issues/304" data-reference-type="issue" data-original="#304" data-link="false" data-link-reference="false" data-project="2411" data-issue="11381" data-project-path="esde/machine-learning/mlair" data-iid="304" data-issue-type="issue" data-container="body" data-placement="top" title="improve datastore logging" class="gfm gfm-issue">#304</a>)</li>
<li data-sourcepos="26:1-26:69">improved logging message on station removal in preprocessing (<a href="/esde/machine-learning/mlair/-/issues/294" data-reference-type="issue" data-original="#294" data-link="false" data-link-reference="false" data-project="2411" data-issue="11091" data-project-path="esde/machine-learning/mlair" data-iid="294" data-issue-type="issue" data-container="body" data-placement="top" title="better understanding why station is removed in preprocessing" class="gfm gfm-issue">#294</a>)</li>
<li data-sourcepos="27:1-27:49">limited number of retries in JOIN module (<a href="/esde/machine-learning/mlair/-/issues/296" data-reference-type="issue" data-original="#296" data-link="false" data-link-reference="false" data-project="2411" data-issue="11218" data-project-path="esde/machine-learning/mlair" data-iid="296" data-issue-type="issue" data-container="body" data-placement="top" title="limit retries in join module" class="gfm gfm-issue">#296</a>)</li>
<li data-sourcepos="28:1-28:44">adjusted competing skill score plot (<a href="/esde/machine-learning/mlair/-/issues/301" data-reference-type="issue" data-original="#301" data-link="false" data-link-reference="false" data-project="2411" data-issue="11307" data-project-path="esde/machine-learning/mlair" data-iid="301" data-issue-type="issue" data-container="body" data-placement="top" title="refac: limit axis in competing skill score plot" class="gfm gfm-issue">#301</a>)</li>
<li data-sourcepos="29:1-29:39">transformation parameter check (<a href="/esde/machine-learning/mlair/-/issues/295" data-reference-type="issue" data-original="#295" data-link="false" data-link-reference="false" data-project="2411" data-issue="11140" data-project-path="esde/machine-learning/mlair" data-iid="295" data-issue-type="issue" data-container="body" data-placement="top" title="Transformation parameters in data_handler_single_station must be xarray" class="gfm gfm-issue">#295</a>)</li>
<li data-sourcepos="30:1-30:71">implemented lazy data preprocessing for selected data handlers (<a href="/esde/machine-learning/mlair/-/issues/292" data-reference-type="issue" data-original="#292" data-link="false" data-link-reference="false" data-project="2411" data-issue="11059" data-project-path="esde/machine-learning/mlair" data-iid="292" data-issue-type="issue" data-container="body" data-placement="top" title="implement lazy data preprocessing" class="gfm gfm-issue">#292</a>)</li>
<li data-sourcepos="31:1-31:53">fix bug in separation of scales data handler (<a href="/esde/machine-learning/mlair/-/issues/290" data-reference-type="issue" data-original="#290" data-link="false" data-link-reference="false" data-project="2411" data-issue="11027" data-project-path="esde/machine-learning/mlair" data-iid="290" data-issue-type="issue" data-container="body" data-placement="top" title="fix separation of scales data handler" class="gfm gfm-issue">#290</a>)</li>
</ul>
2022-04-11T12:09:46+02:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v1.3.0
v1.3.0
<h3 data-sourcepos="1:1-1:12" dir="auto">
<a id="user-content-general" class="anchor" href="#general" aria-hidden="true"></a>general:</h3>
<ul data-sourcepos="2:1-5:0" dir="auto">
<li data-sourcepos="2:1-2:39">release of official MLAir logo (<a href="/esde/machine-learning/mlair/-/issues/274" data-reference-type="issue" data-original="#274" data-link="false" data-link-reference="false" data-project="2411" data-issue="10588" data-project-path="esde/machine-learning/mlair" data-iid="274" data-issue-type="issue" data-container="body" data-placement="top" title="release MLAir Logo" class="gfm gfm-issue">#274</a>)</li>
<li data-sourcepos="3:1-3:84">new transformation schema for better independence of MLAir and data handler (<a href="/esde/machine-learning/mlair/-/issues/272" data-reference-type="issue" data-original="#272" data-link="false" data-link-reference="false" data-project="2411" data-issue="10572" data-project-path="esde/machine-learning/mlair" data-iid="272" data-issue-type="issue" data-container="body" data-placement="top" title="individual transformation" class="gfm gfm-issue">#272</a>)</li>
<li data-sourcepos="4:1-5:0">competing models can be included in postprocessing for direct comparison (<a href="/esde/machine-learning/mlair/-/issues/198" data-reference-type="issue" data-original="#198" data-link="false" data-link-reference="false" data-project="2411" data-issue="9666" data-project-path="esde/machine-learning/mlair" data-iid="198" data-issue-type="issue" data-container="body" data-placement="top" title="Competitor Models" class="gfm gfm-issue">#198</a>)</li>
</ul>
<h3 data-sourcepos="6:1-6:17" dir="auto">
<a id="user-content-new-features" class="anchor" href="#new-features" aria-hidden="true"></a>new features:</h3>
<ul data-sourcepos="7:1-10:0" dir="auto">
<li data-sourcepos="7:1-7:51">new helper functions for geographic issues (<a href="/esde/machine-learning/mlair/-/issues/280" data-reference-type="issue" data-original="#280" data-link="false" data-link-reference="false" data-project="2411" data-issue="10725" data-project-path="esde/machine-learning/mlair" data-iid="280" data-issue-type="issue" data-container="body" data-placement="top" title="Refac: Make geo-helpers available" class="gfm gfm-issue">#280</a>)</li>
<li data-sourcepos="8:1-8:91">default data handler and inheritances can use min/max and log transformation (<a href="/esde/machine-learning/mlair/-/issues/276" data-reference-type="issue" data-original="#276" data-link="false" data-link-reference="false" data-project="2411" data-issue="10603" data-project-path="esde/machine-learning/mlair" data-iid="276" data-issue-type="issue" data-container="body" data-placement="top" title="add min max scaling for default data handler and statistics" class="gfm gfm-issue">#276</a>, <a href="/esde/machine-learning/mlair/-/issues/275" data-reference-type="issue" data-original="#275" data-link="false" data-link-reference="false" data-project="2411" data-issue="10602" data-project-path="esde/machine-learning/mlair" data-iid="275" data-issue-type="issue" data-container="body" data-placement="top" title="add box-cox transformation to default data handler and statistics" class="gfm gfm-issue">#275</a>)</li>
<li data-sourcepos="9:1-10:0">include IntelliO3-ts model as reference via automatic download (<a href="/esde/machine-learning/mlair/-/issues/131" data-reference-type="issue" data-original="#131" data-link="false" data-link-reference="false" data-project="2411" data-issue="8918" data-project-path="esde/machine-learning/mlair" data-iid="131" data-issue-type="issue" data-container="body" data-placement="top" title="Make IntelliO3-ts v1.0 available as reference" class="gfm gfm-issue">#131</a>)</li>
</ul>
<h3 data-sourcepos="11:1-11:14" dir="auto">
<a id="user-content-technical" class="anchor" href="#technical" aria-hidden="true"></a>technical:</h3>
<ul data-sourcepos="12:1-18:79" dir="auto">
<li data-sourcepos="12:1-12:65">experiment name now always includes target sampling type (<a href="/esde/machine-learning/mlair/-/issues/263" data-reference-type="issue" data-original="#263" data-link="false" data-link-reference="false" data-project="2411" data-issue="10473" data-project-path="esde/machine-learning/mlair" data-iid="263" data-issue-type="issue" data-container="body" data-placement="top" title="experiment name with mixed sampling" class="gfm gfm-issue">#263</a>)</li>
<li data-sourcepos="13:1-13:51">competitive skill score plot is refactored (<a href="/esde/machine-learning/mlair/-/issues/260" data-reference-type="issue" data-original="#260" data-link="false" data-link-reference="false" data-project="2411" data-issue="10407" data-project-path="esde/machine-learning/mlair" data-iid="260" data-issue-type="issue" data-container="body" data-placement="top" title="rotate labels in competitive skill score plot" class="gfm gfm-issue">#260</a>)</li>
<li data-sourcepos="14:1-14:48">bug fix for climatological skill scores (<a href="/esde/machine-learning/mlair/-/issues/259" data-reference-type="issue" data-original="#259" data-link="false" data-link-reference="false" data-project="2411" data-issue="10317" data-project-path="esde/machine-learning/mlair" data-iid="259" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: change of internal and external climatology" class="gfm gfm-issue">#259</a>)</li>
<li data-sourcepos="15:1-15:44">bug fix for custom objects handling (<a href="/esde/machine-learning/mlair/-/issues/277" data-reference-type="issue" data-original="#277" data-link="false" data-link-reference="false" data-project="2411" data-issue="10644" data-project-path="esde/machine-learning/mlair" data-iid="277" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: loading of custom objects not working" class="gfm gfm-issue">#277</a>)</li>
<li data-sourcepos="16:1-16:76">bug fix for monitoring plots when multiple output branches are used (<a href="/esde/machine-learning/mlair/-/issues/278" data-reference-type="issue" data-original="#278" data-link="false" data-link-reference="false" data-project="2411" data-issue="10676" data-project-path="esde/machine-learning/mlair" data-iid="278" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: mse monitoring plot not working with multiple branches" class="gfm gfm-issue">#278</a>)</li>
<li data-sourcepos="17:1-17:68">update requirements to newer version and dependencies (<a href="/esde/machine-learning/mlair/-/issues/262" data-reference-type="issue" data-original="#262" data-link="false" data-link-reference="false" data-project="2411" data-issue="10469" data-project-path="esde/machine-learning/mlair" data-iid="262" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: update reqirements" class="gfm gfm-issue">#262</a>, <a href="/esde/machine-learning/mlair/-/issues/273" data-reference-type="issue" data-original="#273" data-link="false" data-link-reference="false" data-project="2411" data-issue="10586" data-project-path="esde/machine-learning/mlair" data-iid="273" data-issue-type="issue" data-container="body" data-placement="top" title="new dependencies occured" class="gfm gfm-issue">#273</a>)</li>
<li data-sourcepos="18:1-18:79">HPC scripts are updated to work properly with parallel data processing (<a href="/esde/machine-learning/mlair/-/issues/281" data-reference-type="issue" data-original="#281" data-link="false" data-link-reference="false" data-project="2411" data-issue="10794" data-project-path="esde/machine-learning/mlair" data-iid="281" data-issue-type="issue" data-container="body" data-placement="top" title="Changes required from HPC tests" class="gfm gfm-issue">#281</a>)</li>
</ul>
2021-02-24T18:15:03+01:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v1.2.1
v1.2.1
<h3 data-sourcepos="1:1-1:12" dir="auto">
<a id="user-content-general" class="anchor" href="#general" aria-hidden="true"></a>general:</h3>
<ul data-sourcepos="3:1-4:0" dir="auto">
<li data-sourcepos="3:1-4:0">applied bug fix</li>
</ul>
<h3 data-sourcepos="5:1-5:14" dir="auto">
<a id="user-content-technical" class="anchor" href="#technical" aria-hidden="true"></a>technical:</h3>
<ul data-sourcepos="7:1-7:43" dir="auto">
<li data-sourcepos="7:1-7:43">bug fix for recursive import error (<a href="/esde/machine-learning/mlair/-/issues/269" data-reference-type="issue" data-original="#269" data-link="false" data-link-reference="false" data-project="2411" data-issue="10549" data-project-path="esde/machine-learning/mlair" data-iid="269" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: recursion errors" class="gfm gfm-issue">#269</a>)</li>
</ul>
2021-02-08T17:51:52+01:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v1.2.0
v1.2.0
<h3 data-sourcepos="1:1-1:12" dir="auto">
<a id="user-content-general" class="anchor" href="#general" aria-hidden="true"></a>general:</h3>
<ul data-sourcepos="3:2-7:1" dir="auto">
<li data-sourcepos="3:2-3:12">new plots</li>
<li data-sourcepos="4:2-4:39">parallelism for faster preprocessing</li>
<li data-sourcepos="5:2-5:50">improved data handler with mixed sampling types</li>
<li data-sourcepos="6:2-7:1">enhanced test coverage</li>
</ul>
<h3 data-sourcepos="8:1-8:17" dir="auto">
<a id="user-content-new-features" class="anchor" href="#new-features" aria-hidden="true"></a>new features:</h3>
<ul data-sourcepos="10:1-15:0" dir="auto">
<li data-sourcepos="10:1-10:113">station map plot highlights now subsets on the map and displays number of stations for each subset (<a href="/esde/machine-learning/mlair/-/issues/227" data-reference-type="issue" data-original="#227" data-link="false" data-link-reference="false" data-project="2411" data-issue="10143" data-project-path="esde/machine-learning/mlair" data-iid="227" data-issue-type="issue" data-container="body" data-placement="top" title="REFAC: Map Plot for more subsets" class="gfm gfm-issue">#227</a>, <a href="/esde/machine-learning/mlair/-/issues/231" data-reference-type="issue" data-original="#231" data-link="false" data-link-reference="false" data-project="2411" data-issue="10209" data-project-path="esde/machine-learning/mlair" data-iid="231" data-issue-type="issue" data-container="body" data-placement="top" title="Add number of stations to Station Map legend" class="gfm gfm-issue">#231</a>)</li>
<li data-sourcepos="11:1-11:80">two new data availability plots <code data-sourcepos="11:36-11:60">PlotAvailabilityHistogram</code> (<a href="/esde/machine-learning/mlair/-/issues/191" data-reference-type="issue" data-original="#191" data-link="false" data-link-reference="false" data-project="2411" data-issue="9592" data-project-path="esde/machine-learning/mlair" data-iid="191" data-issue-type="issue" data-container="body" data-placement="top" title="Add additional data plot (histogram)" class="gfm gfm-issue">#191</a>, <a href="/esde/machine-learning/mlair/-/issues/192" data-reference-type="issue" data-original="#192" data-link="false" data-link-reference="false" data-project="2411" data-issue="9593" data-project-path="esde/machine-learning/mlair" data-iid="192" data-issue-type="issue" data-container="body" data-placement="top" title="Add additional data plot (cummulative)" class="gfm gfm-issue">#192</a>, <a href="/esde/machine-learning/mlair/-/issues/223" data-reference-type="issue" data-original="#223" data-link="false" data-link-reference="false" data-project="2411" data-issue="10049" data-project-path="esde/machine-learning/mlair" data-iid="223" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: PlotAvailabilityHistogram expects 2D data" class="gfm gfm-issue">#223</a>)</li>
<li data-sourcepos="12:1-12:93">introduced parallel code in preprocessing if system supports parallelism (<a href="/esde/machine-learning/mlair/-/issues/164" data-reference-type="issue" data-original="#164" data-link="false" data-link-reference="false" data-project="2411" data-issue="9421" data-project-path="esde/machine-learning/mlair" data-iid="164" data-issue-type="issue" data-container="body" data-placement="top" title="Parallel station check" class="gfm gfm-issue">#164</a>, <a href="/esde/machine-learning/mlair/-/issues/224" data-reference-type="issue" data-original="#224" data-link="false" data-link-reference="false" data-project="2411" data-issue="10056" data-project-path="esde/machine-learning/mlair" data-iid="224" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: add ValueError to f_proc" class="gfm gfm-issue">#224</a>, <a href="/esde/machine-learning/mlair/-/issues/225" data-reference-type="issue" data-original="#225" data-link="false" data-link-reference="false" data-project="2411" data-issue="10057" data-project-path="esde/machine-learning/mlair" data-iid="225" data-issue-type="issue" data-container="body" data-placement="top" title="parallel transformation" class="gfm gfm-issue">#225</a>)</li>
<li data-sourcepos="13:1-13:144">data handler <code data-sourcepos="13:17-13:40">DataHandlerMixedSampling</code> (and inheritances) supports an offset parameter to end inputs at a different time than 00 hours (<a href="/esde/machine-learning/mlair/-/issues/220" data-reference-type="issue" data-original="#220" data-link="false" data-link-reference="false" data-project="2411" data-issue="9999" data-project-path="esde/machine-learning/mlair" data-iid="220" data-issue-type="issue" data-container="body" data-placement="top" title="REFAC: history for mixed sampling data handler" class="gfm gfm-issue">#220</a>)</li>
<li data-sourcepos="14:1-15:0">args for data handler <code data-sourcepos="14:26-14:49">DataHandlerMixedSampling</code> (and inheritances) that differ for input and target can now be parsed as tuple (<a href="/esde/machine-learning/mlair/-/issues/229" data-reference-type="issue" data-original="#229" data-link="false" data-link-reference="false" data-project="2411" data-issue="10171" data-project-path="esde/machine-learning/mlair" data-iid="229" data-issue-type="issue" data-container="body" data-placement="top" title="mixed sampling decouple interpolation" class="gfm gfm-issue">#229</a>)</li>
</ul>
<h3 data-sourcepos="16:1-16:14" dir="auto">
<a id="user-content-technical" class="anchor" href="#technical" aria-hidden="true"></a>technical:</h3>
<ul data-sourcepos="18:1-29:79" dir="auto">
<li data-sourcepos="18:1-18:51">added templates for release and bug issues (<a href="/esde/machine-learning/mlair/-/issues/189" data-reference-type="issue" data-original="#189" data-link="false" data-link-reference="false" data-project="2411" data-issue="9559" data-project-path="esde/machine-learning/mlair" data-iid="189" data-issue-type="issue" data-container="body" data-placement="top" title="issue templates" class="gfm gfm-issue">#189</a>)</li>
<li data-sourcepos="19:1-19:79">improved test coverage (<a href="/esde/machine-learning/mlair/-/issues/236" data-reference-type="issue" data-original="#236" data-link="false" data-link-reference="false" data-project="2411" data-issue="10228" data-project-path="esde/machine-learning/mlair" data-iid="236" data-issue-type="issue" data-container="body" data-placement="top" title="new tests for __init__" class="gfm gfm-issue">#236</a>, <a href="/esde/machine-learning/mlair/-/issues/238" data-reference-type="issue" data-original="#238" data-link="false" data-link-reference="false" data-project="2411" data-issue="10230" data-project-path="esde/machine-learning/mlair" data-iid="238" data-issue-type="issue" data-container="body" data-placement="top" title="new tests for iterator" class="gfm gfm-issue">#238</a>, <a href="/esde/machine-learning/mlair/-/issues/239" data-reference-type="issue" data-original="#239" data-link="false" data-link-reference="false" data-project="2411" data-issue="10231" data-project-path="esde/machine-learning/mlair" data-iid="239" data-issue-type="issue" data-container="body" data-placement="top" title="new tests for advanced data handler" class="gfm gfm-issue">#239</a>, <a href="/esde/machine-learning/mlair/-/issues/240" data-reference-type="issue" data-original="#240" data-link="false" data-link-reference="false" data-project="2411" data-issue="10232" data-project-path="esde/machine-learning/mlair" data-iid="240" data-issue-type="issue" data-container="body" data-placement="top" title="new tests for datastore" class="gfm gfm-issue">#240</a>, <a href="/esde/machine-learning/mlair/-/issues/241" data-reference-type="issue" data-original="#241" data-link="false" data-link-reference="false" data-project="2411" data-issue="10233" data-project-path="esde/machine-learning/mlair" data-iid="241" data-issue-type="issue" data-container="body" data-placement="top" title="new tests for helpers" class="gfm gfm-issue">#241</a>, <a href="/esde/machine-learning/mlair/-/issues/242" data-reference-type="issue" data-original="#242" data-link="false" data-link-reference="false" data-project="2411" data-issue="10234" data-project-path="esde/machine-learning/mlair" data-iid="242" data-issue-type="issue" data-container="body" data-placement="top" title="new tests for helper testing" class="gfm gfm-issue">#242</a>, <a href="/esde/machine-learning/mlair/-/issues/243" data-reference-type="issue" data-original="#243" data-link="false" data-link-reference="false" data-project="2411" data-issue="10235" data-project-path="esde/machine-learning/mlair" data-iid="243" data-issue-type="issue" data-container="body" data-placement="top" title="new tests for tracker plot" class="gfm gfm-issue">#243</a>, <a href="/esde/machine-learning/mlair/-/issues/244" data-reference-type="issue" data-original="#244" data-link="false" data-link-reference="false" data-project="2411" data-issue="10237" data-project-path="esde/machine-learning/mlair" data-iid="244" data-issue-type="issue" data-container="body" data-placement="top" title="new test for abstract workflow" class="gfm gfm-issue">#244</a>, <a href="/esde/machine-learning/mlair/-/issues/245" data-reference-type="issue" data-original="#245" data-link="false" data-link-reference="false" data-project="2411" data-issue="10238" data-project-path="esde/machine-learning/mlair" data-iid="245" data-issue-type="issue" data-container="body" data-placement="top" title="new tests for default workflow" class="gfm gfm-issue">#245</a>)</li>
<li data-sourcepos="20:1-20:73">station map plot includes now number of stations for each subset (<a href="/esde/machine-learning/mlair/-/issues/231" data-reference-type="issue" data-original="#231" data-link="false" data-link-reference="false" data-project="2411" data-issue="10209" data-project-path="esde/machine-learning/mlair" data-iid="231" data-issue-type="issue" data-container="body" data-placement="top" title="Add number of stations to Station Map legend" class="gfm gfm-issue">#231</a>)</li>
<li data-sourcepos="21:1-21:71">postprocessing plots are encapsulated in try except statements (<a href="/esde/machine-learning/mlair/-/issues/107" data-reference-type="issue" data-original="#107" data-link="false" data-link-reference="false" data-project="2411" data-issue="8386" data-project-path="esde/machine-learning/mlair" data-iid="107" data-issue-type="issue" data-container="body" data-placement="top" title="Use try except statements for all plots" class="gfm gfm-issue">#107</a>)</li>
<li data-sourcepos="22:1-22:29">updated git settings (<a href="/esde/machine-learning/mlair/-/issues/213" data-reference-type="issue" data-original="#213" data-link="false" data-link-reference="false" data-project="2411" data-issue="9911" data-project-path="esde/machine-learning/mlair" data-iid="213" data-issue-type="issue" data-container="body" data-placement="top" title="add entries to gitignore" class="gfm gfm-issue">#213</a>)</li>
<li data-sourcepos="23:1-23:33">bug fix for data handler (<a href="/esde/machine-learning/mlair/-/issues/235" data-reference-type="issue" data-original="#235" data-link="false" data-link-reference="false" data-project="2411" data-issue="10222" data-project-path="esde/machine-learning/mlair" data-iid="235" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: missing indexing in DataHandlerMixedSamplingWithFilter" class="gfm gfm-issue">#235</a>)</li>
<li data-sourcepos="24:1-24:65">reordering and bug fix for preprocessing reporting (<a href="/esde/machine-learning/mlair/-/issues/207" data-reference-type="issue" data-original="#207" data-link="false" data-link-reference="false" data-project="2411" data-issue="9852" data-project-path="esde/machine-learning/mlair" data-iid="207" data-issue-type="issue" data-container="body" data-placement="top" title="Reoder data summary table" class="gfm gfm-issue">#207</a>, <a href="/esde/machine-learning/mlair/-/issues/232" data-reference-type="issue" data-original="#232" data-link="false" data-link-reference="false" data-project="2411" data-issue="10210" data-project-path="esde/machine-learning/mlair" data-iid="232" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: wrong count of stations in preprocessing reporting" class="gfm gfm-issue">#232</a>)</li>
<li data-sourcepos="25:1-25:47">bug fix for outdated system path style (<a href="/esde/machine-learning/mlair/-/issues/226" data-reference-type="issue" data-original="#226" data-link="false" data-link-reference="false" data-project="2411" data-issue="10061" data-project-path="esde/machine-learning/mlair" data-iid="226" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: NaN error during PostProcessing" class="gfm gfm-issue">#226</a>)</li>
<li data-sourcepos="26:1-26:52">new plots are included in default plot list (<a href="/esde/machine-learning/mlair/-/issues/211" data-reference-type="issue" data-original="#211" data-link="false" data-link-reference="false" data-project="2411" data-issue="9877" data-project-path="esde/machine-learning/mlair" data-iid="211" data-issue-type="issue" data-container="body" data-placement="top" title="Inclue new IntelliO3 plots in MLAir" class="gfm gfm-issue">#211</a>)</li>
<li data-sourcepos="27:1-27:125">
<code data-sourcepos="27:4-27:15">helpers/join</code> connection to ToarDB (e.g. used by DefaultDataHandler) reports now which variable could not be loaded (<a href="/esde/machine-learning/mlair/-/issues/222" data-reference-type="issue" data-original="#222" data-link="false" data-link-reference="false" data-project="2411" data-issue="10046" data-project-path="esde/machine-learning/mlair" data-iid="222" data-issue-type="issue" data-container="body" data-placement="top" title='REFAC: "bad" stations do not report which variable is missing' class="gfm gfm-issue">#222</a>)</li>
<li data-sourcepos="28:1-28:135">plot <code data-sourcepos="28:9-28:31">PlotBootstrapSkillScore</code> can now additionally highlight specific variables, but not included in postprocessing up to now (<a href="/esde/machine-learning/mlair/-/issues/201" data-reference-type="issue" data-original="#201" data-link="false" data-link-reference="false" data-project="2411" data-issue="9813" data-project-path="esde/machine-learning/mlair" data-iid="201" data-issue-type="issue" data-container="body" data-placement="top" title="Influence of input variables" class="gfm gfm-issue">#201</a>)</li>
<li data-sourcepos="29:1-29:79">data handler <code data-sourcepos="29:17-29:40">DataHandlerMixedSampling</code> has now a reduced data loading (<a href="/esde/machine-learning/mlair/-/issues/221" data-reference-type="issue" data-original="#221" data-link="false" data-link-reference="false" data-project="2411" data-issue="10043" data-project-path="esde/machine-learning/mlair" data-iid="221" data-issue-type="issue" data-container="body" data-placement="top" title="REFAC: mixed sampling data handler loads to much data" class="gfm gfm-issue">#221</a>)</li>
</ul>
2020-12-18T12:54:32+01:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v1.1.0
v1.1.0
hourly resolution support and new data handlers
<ul data-sourcepos="1:1-15:80" dir="auto">
<li data-sourcepos="1:1-3:84">general:
<ul data-sourcepos="2:3-3:84">
<li data-sourcepos="2:3-2:55">MLAir can be used with 1H resolution data from JOIN</li>
<li data-sourcepos="3:3-3:84">new data handlers to use the Kolmogorov-Zurbenko filter and mixed sampling types</li>
</ul>
</li>
<li data-sourcepos="4:1-8:135">new features:
<ul data-sourcepos="5:3-8:135">
<li data-sourcepos="5:3-5:105">new data handler <code data-sourcepos="5:23-5:41">DataHandlerKzFilter</code> to use Kolmogorov-Zurbenko filter (kz filter) on inputs (<a href="/esde/machine-learning/mlair/-/issues/195" data-reference-type="issue" data-original="#195" data-link="false" data-link-reference="false" data-project="2411" data-issue="9659" data-project-path="esde/machine-learning/mlair" data-iid="195" data-issue-type="issue" data-container="body" data-placement="top" title="KZ Filter creating additional dimension" class="gfm gfm-issue">#195</a>)</li>
<li data-sourcepos="6:3-6:110">new data handler <code data-sourcepos="6:23-6:46">DataHandlerMixedSampling</code> that can used mixed sampling types for input and target (<a href="/esde/machine-learning/mlair/-/issues/197" data-reference-type="issue" data-original="#197" data-link="false" data-link-reference="false" data-project="2411" data-issue="9661" data-project-path="esde/machine-learning/mlair" data-iid="197" data-issue-type="issue" data-container="body" data-placement="top" title="Mixed sampling types" class="gfm gfm-issue">#197</a>)</li>
<li data-sourcepos="7:3-7:103">new data handler <code data-sourcepos="7:23-7:56">DataHandlerMixedSamplingWithFilter</code> that uses kz filter and mixed sampling (<a href="/esde/machine-learning/mlair/-/issues/197" data-reference-type="issue" data-original="#197" data-link="false" data-link-reference="false" data-project="2411" data-issue="9661" data-project-path="esde/machine-learning/mlair" data-iid="197" data-issue-type="issue" data-container="body" data-placement="top" title="Mixed sampling types" class="gfm gfm-issue">#197</a>)</li>
<li data-sourcepos="8:3-8:135">new data handler <code data-sourcepos="8:23-8:51">DataHandlerSeparationOfScales</code> to filter-depended time steps sizes on filtered inputs using mixed sampling (<a href="/esde/machine-learning/mlair/-/issues/196" data-reference-type="issue" data-original="#196" data-link="false" data-link-reference="false" data-project="2411" data-issue="9660" data-project-path="esde/machine-learning/mlair" data-iid="196" data-issue-type="issue" data-container="body" data-placement="top" title="Separation of Scales" class="gfm gfm-issue">#196</a>)</li>
</ul>
</li>
<li data-sourcepos="9:1-15:80">technical:
<ul data-sourcepos="10:3-15:80">
<li data-sourcepos="10:3-10:63">bug fix for very short time series in TimeSeriesPlot (<a href="/esde/machine-learning/mlair/-/issues/215" data-reference-type="issue" data-original="#215" data-link="false" data-link-reference="false" data-project="2411" data-issue="9915" data-project-path="esde/machine-learning/mlair" data-iid="215" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: time series plot on HDFML" class="gfm gfm-issue">#215</a>)</li>
<li data-sourcepos="11:3-11:71">bug fix for variable dictionary when using hourly resolution (<a href="/esde/machine-learning/mlair/-/issues/212" data-reference-type="issue" data-original="#212" data-link="false" data-link-reference="false" data-project="2411" data-issue="9883" data-project-path="esde/machine-learning/mlair" data-iid="212" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: mixed sampling causing errors on data loading" class="gfm gfm-issue">#212</a>)</li>
<li data-sourcepos="12:3-12:66">variable naming for data from JOIN interface harmonised (<a href="/esde/machine-learning/mlair/-/issues/206" data-reference-type="issue" data-original="#206" data-link="false" data-link-reference="false" data-project="2411" data-issue="9831" data-project-path="esde/machine-learning/mlair" data-iid="206" data-issue-type="issue" data-container="body" data-placement="top" title="handle inconsistent naming in data source" class="gfm gfm-issue">#206</a>)</li>
<li data-sourcepos="13:3-13:71">transformation setup is now separated for inputs and targets (<a href="/esde/machine-learning/mlair/-/issues/202" data-reference-type="issue" data-original="#202" data-link="false" data-link-reference="false" data-project="2411" data-issue="9817" data-project-path="esde/machine-learning/mlair" data-iid="202" data-issue-type="issue" data-container="body" data-placement="top" title="REFAC: transformation setup" class="gfm gfm-issue">#202</a>)</li>
<li data-sourcepos="14:3-14:81">bug fix in PlotClimatologicalSkillScore if only single station is used (<a href="/esde/machine-learning/mlair/-/issues/193" data-reference-type="issue" data-original="#193" data-link="false" data-link-reference="false" data-project="2411" data-issue="9600" data-project-path="esde/machine-learning/mlair" data-iid="193" data-issue-type="issue" data-container="body" data-placement="top" title="bug: too many indices in PlotClimatologicalSkillScore" class="gfm gfm-issue">#193</a>)</li>
<li data-sourcepos="15:3-15:80">preprocessed data is now stored inside experiment and not in the data folder</li>
</ul>
</li>
</ul>
2020-11-18T15:02:52+01:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/IntelliO3-ts-v1.0_R1-submit
IntelliO3-ts-v1.0_R1-submit
This version was used for R1 of https://gmd.copernicus.org/preprints/gmd-2020-169/
<p data-sourcepos="1:1-1:84" dir="auto">This version was used for R1 of <code data-sourcepos="1:34-1:83">https://gmd.copernicus.org/preprints/gmd-2020-169/</code></p>
2020-11-09T11:35:25+01:00
felix kleinert
f.kleinert@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v1.0.0
v1.0.0
official release of new version 1.0.0
<ul data-sourcepos="1:1-5:50" dir="auto">
<li data-sourcepos="1:1-3:45">general:
<ul data-sourcepos="2:3-3:45">
<li data-sourcepos="2:3-2:61">This is the first official release of MLAir ready for use</li>
<li data-sourcepos="3:3-3:45">updated license, installation instruction</li>
</ul>
</li>
<li data-sourcepos="4:1-5:50">technical:
<ul data-sourcepos="5:3-5:50">
<li data-sourcepos="5:3-5:50">restructured order of packages in requirements</li>
</ul>
</li>
</ul>
2020-10-08T16:30:27+02:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v0.12.2
v0.12.2
HDFML support
<ul data-sourcepos="1:1-4:48" dir="auto">
<li data-sourcepos="1:1-2:17">general:
<ul data-sourcepos="2:3-2:17">
<li data-sourcepos="2:3-2:17">HDFML support</li>
</ul>
</li>
<li data-sourcepos="3:1-4:48">technical:
<ul data-sourcepos="4:3-4:48">
<li data-sourcepos="4:3-4:48">installation script for HDFML adjusted, <a href="/esde/machine-learning/mlair/-/issues/183" data-reference-type="issue" data-original="#183" data-link="false" data-link-reference="false" data-project="2411" data-issue="9505" data-project-path="esde/machine-learning/mlair" data-iid="183" data-issue-type="issue" data-container="body" data-placement="top" title="Check installation script for HDFML" class="gfm gfm-issue">#183</a>
</li>
</ul>
</li>
</ul>
2020-10-01T12:50:18+02:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v0.12.1
v0.12.1
<ul data-sourcepos="1:1-7:87" dir="auto">
<li data-sourcepos="1:1-3:79">general:
<ul data-sourcepos="2:3-3:79">
<li data-sourcepos="2:3-2:63">introduced a notebook documentation for easy starting, <a href="/esde/machine-learning/mlair/-/issues/174" data-reference-type="issue" data-original="#174" data-link="false" data-link-reference="false" data-project="2411" data-issue="9456" data-project-path="esde/machine-learning/mlair" data-iid="174" data-issue-type="issue" data-container="body" data-placement="top" title="Create Quick-Start Notebook" class="gfm gfm-issue">#174</a>
</li>
<li data-sourcepos="3:3-3:79">updated special installation instructions for the Juelich HPC systems, <a href="/esde/machine-learning/mlair/-/issues/172" data-reference-type="issue" data-original="#172" data-link="false" data-link-reference="false" data-project="2411" data-issue="9454" data-project-path="esde/machine-learning/mlair" data-iid="172" data-issue-type="issue" data-container="body" data-placement="top" title="Update README HPC setup" class="gfm gfm-issue">#172</a>
</li>
</ul>
</li>
<li data-sourcepos="4:1-5:100">new features:
<ul data-sourcepos="5:3-5:100">
<li data-sourcepos="5:3-5:100">names of input and output shape are renamed consistently to: input_shape, and output_shape, <a href="/esde/machine-learning/mlair/-/issues/175" data-reference-type="issue" data-original="#175" data-link="false" data-link-reference="false" data-project="2411" data-issue="9457" data-project-path="esde/machine-learning/mlair" data-iid="175" data-issue-type="issue" data-container="body" data-placement="top" title="REFAC: rename shapes of input and output" class="gfm gfm-issue">#175</a>
</li>
</ul>
</li>
<li data-sourcepos="6:1-7:87">technical:
<ul data-sourcepos="7:3-7:87">
<li data-sourcepos="7:3-7:87">it is possible to assign a custom name to a run module (e.g. used in logging), <a href="/esde/machine-learning/mlair/-/issues/173" data-reference-type="issue" data-original="#173" data-link="false" data-link-reference="false" data-project="2411" data-issue="9455" data-project-path="esde/machine-learning/mlair" data-iid="173" data-issue-type="issue" data-container="body" data-placement="top" title="mlair started not appearing" class="gfm gfm-issue">#173</a>
</li>
</ul>
</li>
</ul>
2020-09-28T13:24:30+02:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v0.12.0
v0.12.0
Documentation and Bugfixes
<ul data-sourcepos="1:1-11:40" dir="auto">
<li data-sourcepos="1:1-3:28">general:
<ul data-sourcepos="2:3-3:28">
<li data-sourcepos="2:3-2:100">improved documentation include installation instructions and many examples from the paper, <a href="/esde/machine-learning/mlair/-/issues/153" data-reference-type="issue" data-original="#153" data-link="false" data-link-reference="false" data-project="2411" data-issue="9283" data-project-path="esde/machine-learning/mlair" data-iid="153" data-issue-type="issue" data-container="body" data-placement="top" title="Advanced Documentation" class="gfm gfm-issue">#153</a>
</li>
<li data-sourcepos="3:3-3:28">bugfixes (see technical)</li>
</ul>
</li>
<li data-sourcepos="4:1-5:87">new features:
<ul data-sourcepos="5:3-5:87">
<li data-sourcepos="5:3-5:87">
<code data-sourcepos="5:6-5:18">MyLittleModel</code> is now a pure feed-forward network (before it had a CNN part), <a href="/esde/machine-learning/mlair/-/issues/168" data-reference-type="issue" data-original="#168" data-link="false" data-link-reference="false" data-project="2411" data-issue="9435" data-project-path="esde/machine-learning/mlair" data-iid="168" data-issue-type="issue" data-container="body" data-placement="top" title="simplify MyLittleModel" class="gfm gfm-issue">#168</a>
</li>
</ul>
</li>
<li data-sourcepos="6:1-11:40">technical:
<ul data-sourcepos="7:3-11:40">
<li data-sourcepos="7:3-7:60">new compile options check to ensure its execution, <a href="/esde/machine-learning/mlair/-/issues/154" data-reference-type="issue" data-original="#154" data-link="false" data-link-reference="false" data-project="2411" data-issue="9286" data-project-path="esde/machine-learning/mlair" data-iid="154" data-issue-type="issue" data-container="body" data-placement="top" title="missing compile option check" class="gfm gfm-issue">#154</a>
</li>
<li data-sourcepos="8:3-8:52">bugfix for key errors in time series plot, <a href="/esde/machine-learning/mlair/-/issues/169" data-reference-type="issue" data-original="#169" data-link="false" data-link-reference="false" data-project="2411" data-issue="9441" data-project-path="esde/machine-learning/mlair" data-iid="169" data-issue-type="issue" data-container="body" data-placement="top" title="BUG: key error in plot time series" class="gfm gfm-issue">#169</a>
</li>
<li data-sourcepos="9:3-9:61">bugfix for not used kwargs in <code data-sourcepos="9:36-9:53">DefaultDataHandler</code>, <a href="/esde/machine-learning/mlair/-/issues/170" data-reference-type="issue" data-original="#170" data-link="false" data-link-reference="false" data-project="2411" data-issue="9449" data-project-path="esde/machine-learning/mlair" data-iid="170" data-issue-type="issue" data-container="body" data-placement="top" title="bug: default data handler" class="gfm gfm-issue">#170</a>
</li>
<li data-sourcepos="10:3-10:113">
<code data-sourcepos="10:6-10:14">trainable</code> parameter is renamed by <code data-sourcepos="10:42-10:52">train_model</code> to prevent confusion with the tf trainable parameter, <a href="/esde/machine-learning/mlair/-/issues/162" data-reference-type="issue" data-original="#162" data-link="false" data-link-reference="false" data-project="2411" data-issue="9379" data-project-path="esde/machine-learning/mlair" data-iid="162" data-issue-type="issue" data-container="body" data-placement="top" title="rename parameter trainable" class="gfm gfm-issue">#162</a>
</li>
<li data-sourcepos="11:3-11:40">fixed HPC installation failure, <a href="/esde/machine-learning/mlair/-/issues/159" data-reference-type="issue" data-original="#159" data-link="false" data-link-reference="false" data-project="2411" data-issue="9305" data-project-path="esde/machine-learning/mlair" data-iid="159" data-issue-type="issue" data-container="body" data-placement="top" title="Failure on HPC installation" class="gfm gfm-issue">#159</a>
</li>
</ul>
</li>
</ul>
2020-09-21T15:20:49+02:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v0.11.0
v0.11.0
v0.11.0
<ul data-sourcepos="1:1-11:34" dir="auto">
<li data-sourcepos="1:1-3:49">general:
<ul data-sourcepos="2:3-3:49">
<li data-sourcepos="2:3-2:131">Introduce advanced data handling with much more flexibility (independent of <em data-sourcepos="2:81-2:89">TOAR DB</em>, custom data handling is pluggable), <a href="/esde/machine-learning/mlair/-/issues/144" data-reference-type="issue" data-original="#144" data-link="false" data-link-reference="false" data-project="2411" data-issue="9033" data-project-path="esde/machine-learning/mlair" data-iid="144" data-issue-type="issue" data-container="body" data-placement="top" title="Include advanced data handling in workflow" class="gfm gfm-issue">#144</a>
</li>
<li data-sourcepos="3:3-3:49">default data handler is still using <em data-sourcepos="3:41-3:49">TOAR DB</em>
</li>
</ul>
</li>
<li data-sourcepos="4:1-7:47">new features:
<ul data-sourcepos="5:3-7:47">
<li data-sourcepos="5:3-5:105">default data handler using <em data-sourcepos="5:32-5:40">TOAR DB</em> refactored according to advanced data handling, <a href="/esde/machine-learning/mlair/-/issues/140" data-reference-type="issue" data-original="#140" data-link="false" data-link-reference="false" data-project="2411" data-issue="9012" data-project-path="esde/machine-learning/mlair" data-iid="140" data-issue-type="issue" data-container="body" data-placement="top" title="Implement Station Preparation" class="gfm gfm-issue">#140</a>, <a href="/esde/machine-learning/mlair/-/issues/141" data-reference-type="issue" data-original="#141" data-link="false" data-link-reference="false" data-project="2411" data-issue="9013" data-project-path="esde/machine-learning/mlair" data-iid="141" data-issue-type="issue" data-container="body" data-placement="top" title="Implement Data Preparation" class="gfm gfm-issue">#141</a>, <a href="/esde/machine-learning/mlair/-/issues/152" data-reference-type="issue" data-original="#152" data-link="false" data-link-reference="false" data-project="2411" data-issue="9164" data-project-path="esde/machine-learning/mlair" data-iid="152" data-issue-type="issue" data-container="body" data-placement="top" title="reimplement interpolation in station class" class="gfm gfm-issue">#152</a>
</li>
<li data-sourcepos="6:3-6:149">data sets are handled as collections, <a href="/esde/machine-learning/mlair/-/issues/142" data-reference-type="issue" data-original="#142" data-link="false" data-link-reference="false" data-project="2411" data-issue="9014" data-project-path="esde/machine-learning/mlair" data-iid="142" data-issue-type="issue" data-container="body" data-placement="top" title="Implement Data Collection" class="gfm gfm-issue">#142</a>, and are itable in a standard way (<code data-sourcepos="6:84-6:99">StandardIterator</code>) and optimised for keras (<code data-sourcepos="6:129-6:141">KerasIterator</code>), <a href="/esde/machine-learning/mlair/-/issues/143" data-reference-type="issue" data-original="#143" data-link="false" data-link-reference="false" data-project="2411" data-issue="9015" data-project-path="esde/machine-learning/mlair" data-iid="143" data-issue-type="issue" data-container="body" data-placement="top" title="Implement Iterator" class="gfm gfm-issue">#143</a>
</li>
<li data-sourcepos="7:3-7:47">automatically moving station map plot, <a href="/esde/machine-learning/mlair/-/issues/136" data-reference-type="issue" data-original="#136" data-link="false" data-link-reference="false" data-project="2411" data-issue="8957" data-project-path="esde/machine-learning/mlair" data-iid="136" data-issue-type="issue" data-container="body" data-placement="top" title="moving station map" class="gfm gfm-issue">#136</a>
</li>
</ul>
</li>
<li data-sourcepos="8:1-11:34">technical:
<ul data-sourcepos="9:3-11:34">
<li data-sourcepos="9:3-9:47">model modules available from package, <a href="/esde/machine-learning/mlair/-/issues/139" data-reference-type="issue" data-original="#139" data-link="false" data-link-reference="false" data-project="2411" data-issue="9011" data-project-path="esde/machine-learning/mlair" data-iid="139" data-issue-type="issue" data-container="body" data-placement="top" title="model modules are not available from mlair package" class="gfm gfm-issue">#139</a>
</li>
<li data-sourcepos="10:3-10:46">renaming of parameter time dimension, <a href="/esde/machine-learning/mlair/-/issues/151" data-reference-type="issue" data-original="#151" data-link="false" data-link-reference="false" data-project="2411" data-issue="9163" data-project-path="esde/machine-learning/mlair" data-iid="151" data-issue-type="issue" data-container="body" data-placement="top" title="rename interpolation dimension by time dimension" class="gfm gfm-issue">#151</a>
</li>
<li data-sourcepos="11:3-11:34">refactoring of README.md, <a href="/esde/machine-learning/mlair/-/issues/138" data-reference-type="issue" data-original="#138" data-link="false" data-link-reference="false" data-project="2411" data-issue="8967" data-project-path="esde/machine-learning/mlair" data-iid="138" data-issue-type="issue" data-container="body" data-placement="top" title="refactor readme files" class="gfm gfm-issue">#138</a>
</li>
</ul>
</li>
</ul>
2020-08-24T14:36:55+02:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/v0.10.0
v0.10.0
official name released MLAir, new Workflows, easy Model plug-in possible
<ul data-sourcepos="1:1-25:47" dir="auto">
<li data-sourcepos="1:1-4:45">general:
<ul data-sourcepos="2:3-4:45">
<li data-sourcepos="2:3-2:75">Official project name is released: MLAir (Machine Learning on Air data)</li>
<li data-sourcepos="3:3-3:63">a model class can now easily be plugged in into MLAir. <a href="/esde/machine-learning/mlair/-/issues/121" data-reference-type="issue" data-original="#121" data-link="false" data-link-reference="false" data-project="2411" data-issue="8766" data-project-path="esde/machine-learning/mlair" data-iid="121" data-issue-type="issue" data-container="body" data-placement="top" title="model link in experiment setup" class="gfm gfm-issue">#121</a>
</li>
<li data-sourcepos="4:3-4:45">introduced new concept of workflows, <a href="/esde/machine-learning/mlair/-/issues/134" data-reference-type="issue" data-original="#134" data-link="false" data-link-reference="false" data-project="2411" data-issue="8926" data-project-path="esde/machine-learning/mlair" data-iid="134" data-issue-type="issue" data-container="body" data-placement="top" title="run module registry" class="gfm gfm-issue">#134</a>
</li>
</ul>
</li>
<li data-sourcepos="5:1-14:77">new features:
<ul data-sourcepos="6:3-14:77">
<li data-sourcepos="6:3-6:65">workflows are used to execute a sequence of run modules, <a href="/esde/machine-learning/mlair/-/issues/134" data-reference-type="issue" data-original="#134" data-link="false" data-link-reference="false" data-project="2411" data-issue="8926" data-project-path="esde/machine-learning/mlair" data-iid="134" data-issue-type="issue" data-container="body" data-placement="top" title="run module registry" class="gfm gfm-issue">#134</a>
</li>
<li data-sourcepos="7:3-7:115">default workflows for standard and the Juelich HPC systems are available, custom workflows can be defined, <a href="/esde/machine-learning/mlair/-/issues/134" data-reference-type="issue" data-original="#134" data-link="false" data-link-reference="false" data-project="2411" data-issue="8926" data-project-path="esde/machine-learning/mlair" data-iid="134" data-issue-type="issue" data-container="body" data-placement="top" title="run module registry" class="gfm gfm-issue">#134</a>
</li>
<li data-sourcepos="8:3-8:75">seasonal decomposition is available for conditional quantile plot, <a href="/esde/machine-learning/mlair/-/issues/112" data-reference-type="issue" data-original="#112" data-link="false" data-link-reference="false" data-project="2411" data-issue="8452" data-project-path="esde/machine-learning/mlair" data-iid="112" data-issue-type="issue" data-container="body" data-placement="top" title="Seasonal decomposition of cond. quantile plots" class="gfm gfm-issue">#112</a>
</li>
<li data-sourcepos="9:3-9:46">map plot is created with coordinates, <a href="/esde/machine-learning/mlair/-/issues/108" data-reference-type="issue" data-original="#108" data-link="false" data-link-reference="false" data-project="2411" data-issue="8398" data-project-path="esde/machine-learning/mlair" data-iid="108" data-issue-type="issue" data-container="body" data-placement="top" title="map plot with coordinates" class="gfm gfm-issue">#108</a>
</li>
<li data-sourcepos="10:3-10:70">
<code data-sourcepos="10:6-10:18">flatten_tails</code> are now more general and easier to customise, <a href="/esde/machine-learning/mlair/-/issues/114" data-reference-type="issue" data-original="#114" data-link="false" data-link-reference="false" data-project="2411" data-issue="8502" data-project-path="esde/machine-learning/mlair" data-iid="114" data-issue-type="issue" data-container="body" data-placement="top" title="Customise flatten_tails" class="gfm gfm-issue">#114</a>
</li>
<li data-sourcepos="11:3-11:73">model classes have custom compile options (replaces <code data-sourcepos="11:58-11:65">set_loss</code>), <a href="/esde/machine-learning/mlair/-/issues/110" data-reference-type="issue" data-original="#110" data-link="false" data-link-reference="false" data-project="2411" data-issue="8433" data-project-path="esde/machine-learning/mlair" data-iid="110" data-issue-type="issue" data-container="body" data-placement="top" title="Add custom compile options" class="gfm gfm-issue">#110</a>
</li>
<li data-sourcepos="12:3-12:58">model can be set in ExperimentSetup from outside, <a href="/esde/machine-learning/mlair/-/issues/121" data-reference-type="issue" data-original="#121" data-link="false" data-link-reference="false" data-project="2411" data-issue="8766" data-project-path="esde/machine-learning/mlair" data-iid="121" data-issue-type="issue" data-container="body" data-placement="top" title="model link in experiment setup" class="gfm gfm-issue">#121</a>
</li>
<li data-sourcepos="13:3-13:75">default experiment settings can be queried using <code data-sourcepos="13:55-13:68">get_defaults()</code>, <a href="/esde/machine-learning/mlair/-/issues/123" data-reference-type="issue" data-original="#123" data-link="false" data-link-reference="false" data-project="2411" data-issue="8769" data-project-path="esde/machine-learning/mlair" data-iid="123" data-issue-type="issue" data-container="body" data-placement="top" title="redefine default experiment settings" class="gfm gfm-issue">#123</a>
</li>
<li data-sourcepos="14:3-14:77">training and model settings are reported as MarkDown and Tex tables, <a href="/esde/machine-learning/mlair/-/issues/145" data-reference-type="issue" data-original="#145" data-link="false" data-link-reference="false" data-project="2411" data-issue="9054" data-project-path="esde/machine-learning/mlair" data-iid="145" data-issue-type="issue" data-container="body" data-placement="top" title="ML param reporting" class="gfm gfm-issue">#145</a>
</li>
</ul>
</li>
<li data-sourcepos="15:1-25:47">technical
<ul data-sourcepos="16:3-25:47">
<li data-sourcepos="16:3-16:82">Juelich HPC systems are supported and installation scripts are available, <a href="/esde/machine-learning/mlair/-/issues/106" data-reference-type="issue" data-original="#106" data-link="false" data-link-reference="false" data-project="2411" data-issue="8367" data-project-path="esde/machine-learning/mlair" data-iid="106" data-issue-type="issue" data-container="body" data-placement="top" title="Setup for JUWELS" class="gfm gfm-issue">#106</a>
</li>
<li data-sourcepos="17:3-17:71">data store is tracked, I/O is saved and illustrated in a plot, <a href="/esde/machine-learning/mlair/-/issues/116" data-reference-type="issue" data-original="#116" data-link="false" data-link-reference="false" data-project="2411" data-issue="8516" data-project-path="esde/machine-learning/mlair" data-iid="116" data-issue-type="issue" data-container="body" data-placement="top" title="track data store use of parameters" class="gfm gfm-issue">#116</a>
</li>
<li data-sourcepos="18:3-18:81">batch size, epoch parameter have to be defined in ExperimentSetup, <a href="/esde/machine-learning/mlair/-/issues/127" data-reference-type="issue" data-original="#127" data-link="false" data-link-reference="false" data-project="2411" data-issue="8817" data-project-path="esde/machine-learning/mlair" data-iid="127" data-issue-type="issue" data-container="body" data-placement="top" title="batch size definition in exp setup" class="gfm gfm-issue">#127</a>, <a href="/esde/machine-learning/mlair/-/issues/122" data-reference-type="issue" data-original="#122" data-link="false" data-link-reference="false" data-project="2411" data-issue="8767" data-project-path="esde/machine-learning/mlair" data-iid="122" data-issue-type="issue" data-container="body" data-placement="top" title="Discussion: right location of epoch parameter" class="gfm gfm-issue">#122</a>
</li>
<li data-sourcepos="19:3-19:45">automatic documentation with sphinx, <a href="/esde/machine-learning/mlair/-/issues/109" data-reference-type="issue" data-original="#109" data-link="false" data-link-reference="false" data-project="2411" data-issue="8399" data-project-path="esde/machine-learning/mlair" data-iid="109" data-issue-type="issue" data-container="body" data-placement="top" title="create sphinx docu" class="gfm gfm-issue">#109</a>
</li>
<li data-sourcepos="20:3-20:49">default experiment settings are updated, <a href="/esde/machine-learning/mlair/-/issues/123" data-reference-type="issue" data-original="#123" data-link="false" data-link-reference="false" data-project="2411" data-issue="8769" data-project-path="esde/machine-learning/mlair" data-iid="123" data-issue-type="issue" data-container="body" data-placement="top" title="redefine default experiment settings" class="gfm gfm-issue">#123</a>
</li>
<li data-sourcepos="21:3-21:63">refactoring of experiment path and its default naming, <a href="/esde/machine-learning/mlair/-/issues/124" data-reference-type="issue" data-original="#124" data-link="false" data-link-reference="false" data-project="2411" data-issue="8801" data-project-path="esde/machine-learning/mlair" data-iid="124" data-issue-type="issue" data-container="body" data-placement="top" title="rename exp path and name" class="gfm gfm-issue">#124</a>
</li>
<li data-sourcepos="22:3-22:45">refactoring of some parameter names, <a href="/esde/machine-learning/mlair/-/issues/146" data-reference-type="issue" data-original="#146" data-link="false" data-link-reference="false" data-project="2411" data-issue="9058" data-project-path="esde/machine-learning/mlair" data-iid="146" data-issue-type="issue" data-container="body" data-placement="top" title="REFAC: rename parameters interpolate_... with interpolation_...." class="gfm gfm-issue">#146</a>
</li>
<li data-sourcepos="23:3-23:55">preparation for package distribution with pip, <a href="/esde/machine-learning/mlair/-/issues/119" data-reference-type="issue" data-original="#119" data-link="false" data-link-reference="false" data-project="2411" data-issue="8658" data-project-path="esde/machine-learning/mlair" data-iid="119" data-issue-type="issue" data-container="body" data-placement="top" title="package distribution" class="gfm gfm-issue">#119</a>
</li>
<li data-sourcepos="24:3-24:59">all run scripts are updated to run with workflows, <a href="/esde/machine-learning/mlair/-/issues/134" data-reference-type="issue" data-original="#134" data-link="false" data-link-reference="false" data-project="2411" data-issue="8926" data-project-path="esde/machine-learning/mlair" data-iid="134" data-issue-type="issue" data-container="body" data-placement="top" title="run module registry" class="gfm gfm-issue">#134</a>
</li>
<li data-sourcepos="25:3-25:47">the experiment folder is restructured, <a href="/esde/machine-learning/mlair/-/issues/130" data-reference-type="issue" data-original="#130" data-link="false" data-link-reference="false" data-project="2411" data-issue="8875" data-project-path="esde/machine-learning/mlair" data-iid="130" data-issue-type="issue" data-container="body" data-placement="top" title="model folder in experiment" class="gfm gfm-issue">#130</a>
</li>
</ul>
</li>
</ul>
2020-07-15T15:51:58+02:00
lukas leufen
l.leufen@fz-juelich.de
https://gitlab.jsc.fz-juelich.de/esde/machine-learning/mlair/-/tags/IntelliO3-ts-v1.0_initial-submit
IntelliO3-ts-v1.0_initial-submit
IntelliO3-ts version1.0;
<p data-sourcepos="1:1-1:216" dir="auto">This version is used for <strong data-sourcepos="1:26-1:129">IntelliO3-ts v1.0: A neural network approach to predict near-surface ozone concentrations in Germany</strong>" by F. Kleinert, L. H. Leufen and M. G. Schultz (2020, submitted to GMD, gmd-2020-169)</p>
2020-05-27T15:58:31+02:00
felix kleinert
f.kleinert@fz-juelich.de