* A Simple Framework for Contrastive Learning of Visual Representations
* A Simple Framework for Contrastive Learning of Visual Representations <br>
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, 2020
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, 2020<br>
http://arxiv.org/abs/2002.05709
http://arxiv.org/abs/2002.05709<br>
15 pages, incl appendix
15 pages, incl appendix
* Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey <br>
* Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
L. Jing and Y. Tian, CVPR2019<br>
L. Jing and Y. Tian, CVPR2019
http://arxiv.org/abs/1902.06162<br>
http://arxiv.org/abs/1902.06162
21 pages
21 pages
The first paper presents a state-of-the-art approach for self-supervised learning of strong visual features based on contrastive learning. Random data augmentation is applied to images from the ImageNet dataset and a model is trained to match augmented and original images. The second paper revisits several self-supervised training techniques for visual representation learning and offers a nice overview over different approaches.
The first paper presents a state-of-the-art approach for self-supervised learning of strong visual features based on contrastive learning. Random data augmentation is applied to images from the ImageNet dataset and a model is trained to match augmented and original images. The second paper revisits several self-supervised training techniques for visual representation learning and offers a nice overview over different approaches.
...
@@ -56,45 +54,45 @@ The first paper presents a state-of-the-art approach for self-supervised learnin
...
@@ -56,45 +54,45 @@ The first paper presents a state-of-the-art approach for self-supervised learnin
### 15 June 2020 - Generative Adversarial Networks (and VAE)
### 15 June 2020 - Generative Adversarial Networks (and VAE)
Venue: JSC meetingroom 2, building 16.3; room 315 <br>
### Monday 20 April - Learning discrete representations from data
### Monday 20 April - Learning discrete representations from data
* A. van den Oord, O. Vinyals, K. Kavukcuoglu, Neural Discrete Representation Learning, NeurIPS 2017<br>
* A. van den Oord, O. Vinyals, K. Kavukcuoglu, Neural Discrete Representation Learning, NeurIPS 2017
https://arxiv.org/abs/1711.00937
https://arxiv.org/abs/1711.00937
* A. Razavi, A. van den Oord, O. Vinyals, Generating Diverse High-Fidelity Images with VQ-VAE-2, NeurIPS 2019<br>
* A. Razavi, A. van den Oord, O. Vinyals, Generating Diverse High-Fidelity Images with VQ-VAE-2, NeurIPS 2019
https://arxiv.org/abs/1906.00446
https://arxiv.org/abs/1906.00446
Those two papers are about learning discrete representations from data by taking inspiration from vector quantization. Learning discrete representations using neural networks is challenging and can helpful for tasks such as compression, planning, reasoning, and can be potentially more interpretable than continuous ones. In the two papers use those learned discrete representations to build autoregressive generative models on image, sound, and video. The second paper (Generating Diverse High-Fidelity Images with VQ-VAE-2) is basically a sequel of the first (Neural Discrete Representation Learning) where they scale the models to bigger datasets and images (up to 1024x1024 resolution).
Those two papers are about learning discrete representations from data by taking inspiration from vector quantization. Learning discrete representations using neural networks is challenging and can helpful for tasks such as compression, planning, reasoning, and can be potentially more interpretable than continuous ones. In the two papers use those learned discrete representations to build autoregressive generative models on image, sound, and video. The second paper (Generating Diverse High-Fidelity Images with VQ-VAE-2) is basically a sequel of the first (Neural Discrete Representation Learning) where they scale the models to bigger datasets and images (up to 1024x1024 resolution).
...
@@ -104,12 +102,12 @@ Those two papers are about learning discrete representations from data by taking
...
@@ -104,12 +102,12 @@ Those two papers are about learning discrete representations from data by taking
**Replacement date for canceled meeting at Monday 16 March**
**Replacement date for canceled meeting at Monday 16 March**
* Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes (CVPR'17 Oral)<br>
* Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes (CVPR'17 Oral)
T. Pohlen, A. Hermans, M. Mathias, and B. Leibe<br>
T. Pohlen, A. Hermans, M. Mathias, and B. Leibe
Paper: http://arxiv.org/abs/1611.08323 <br>
Paper: http://arxiv.org/abs/1611.08323
Code: https://github.com/TobyPDE/FRRN
Code: https://github.com/TobyPDE/FRRN
* Deep High-Resolution Representation Learning for Visual Recognition, Wang et al. 2019<br>
* Deep High-Resolution Representation Learning for Visual Recognition, Wang et al. 2019
Most common architectures for semantic segmentation consisting of an encoder and decoder part (e.g. U-Net) heavily reduce the spatial dimension of input images and may loose important details or fail to localize precisely. The proposed papers present full-resolution networks, which try to preserve high-resolution features throughout the network and improve localization accuracy.
Most common architectures for semantic segmentation consisting of an encoder and decoder part (e.g. U-Net) heavily reduce the spatial dimension of input images and may loose important details or fail to localize precisely. The proposed papers present full-resolution networks, which try to preserve high-resolution features throughout the network and improve localization accuracy.
...
@@ -117,27 +115,27 @@ Most common architectures for semantic segmentation consisting of an encoder and
...
@@ -117,27 +115,27 @@ Most common architectures for semantic segmentation consisting of an encoder and
### Monday 17 February - Speech Recognition
### Monday 17 February - Speech Recognition
* Deep Speech 2: End-to-End Speech Recognition in English and Mandarin, Amodei et al., 2015<br>
* Deep Speech 2: End-to-End Speech Recognition in English and Mandarin, Amodei et al., 2015
https://arxiv.org/abs/1512.02595<br>
https://arxiv.org/abs/1512.02595
Not the most recent paper about speech recognition, but a breakthrough for language modelling, thus worth reading. It discusses some topics, which are relevant for timeseries analysis, and also makes reference to good use of HPC. Following this topic, we could continue with latest papers on that at next Journal Club.
Not the most recent paper about speech recognition, but a breakthrough for language modelling, thus worth reading. It discusses some topics, which are relevant for timeseries analysis, and also makes reference to good use of HPC. Following this topic, we could continue with latest papers on that at next Journal Club.
* and, if needed, as background paper about deep recurrent networks:<br>
* and, if needed, as background paper about deep recurrent networks:
Speech Recognition with Deep Recurrent Neural Networks<br>
Speech Recognition with Deep Recurrent Neural Networks
Alex Graves, Abdel-rahman Mohamed, Geoffrey Hinton, 2013<br>
Alex Graves, Abdel-rahman Mohamed, Geoffrey Hinton, 2013
https://arxiv.org/abs/1303.5778
https://arxiv.org/abs/1303.5778
### Monday 20 January
### Monday 20 January
* Multi-Context Recurrent Neural Networks for Time Series Applications <br>
* Multi-Context Recurrent Neural Networks for Time Series Applications
https://publications.waset.org/3524/pdf
https://publications.waset.org/3524/pdf
* Global Sparse Momentum SGD for Pruning Very Deep Neural Networks <br>
* Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
https://arxiv.org/pdf/1909.12778v3.pdf
https://arxiv.org/pdf/1909.12778v3.pdf
### Monday 16 December
### Monday 16 December
Our first Journal Club will cover two papers from ICCV 2019 about GANs.
Our first Journal Club will cover two papers from ICCV 2019 about GANs.
* SinGAN: Learning a Generative Model from a Single Natural Image (Best Paper Award) <br> http://openaccess.thecvf.com/content_ICCV_2019/papers/Shaham_SinGAN_Learning_a_Generative_Model_From_a_Single_Natural_Image_ICCV_2019_paper.pdf
* SinGAN: Learning a Generative Model from a Single Natural Image (Best Paper Award) http://openaccess.thecvf.com/content_ICCV_2019/papers/Shaham_SinGAN_Learning_a_Generative_Model_From_a_Single_Natural_Image_ICCV_2019_paper.pdf
* COCO-GAN: Generation by Parts via Conditional Coordinating <br>
* COCO-GAN: Generation by Parts via Conditional Coordinating
full paper and supplementary material available at https://hubert0527.github.io/COCO-GAN/
full paper and supplementary material available at https://hubert0527.github.io/COCO-GAN/
* Intro Slides by Mickael Cormier available [[here](https://gitlab.version.fz-juelich.de/MLDL_FZJ/General_Wiki/blob/master/files/JournalClub/20191216_gan.pptx)]
* Intro Slides by Mickael Cormier available [[here](https://gitlab.version.fz-juelich.de/MLDL_FZJ/General_Wiki/blob/master/files/JournalClub/20191216_gan.pptx)]
...
@@ -145,11 +143,11 @@ full paper and supplementary material available at https://hubert0527.github.io/
...
@@ -145,11 +143,11 @@ full paper and supplementary material available at https://hubert0527.github.io/
## Archive paper proposals, not read yet
## Archive paper proposals, not read yet
* 17.12.2019 Hanno Scharr
* 17.12.2019 Hanno Scharr
* A General and Adaptive Robust Loss Function<br>
* A General and Adaptive Robust Loss Function
Jonathan T. Barron, Google Research <br> http://openaccess.thecvf.com/content_CVPR_2019/papers/Barron_A_General_and_Adaptive_Robust_Loss_Function_CVPR_2019_paper.pdf<br>
Jonathan T. Barron, Google Research http://openaccess.thecvf.com/content_CVPR_2019/papers/Barron_A_General_and_Adaptive_Robust_Loss_Function_CVPR_2019_paper.pdf
Simple but effective for improving accuracy in regression tasks
Simple but effective for improving accuracy in regression tasks
* RePr: Improved Training of Convolutional Filters<br>
* RePr: Improved Training of Convolutional Filters
Aaditya Prakash, James Storer, Dinei Florencio, Cha Zhang, Brandeis and Microsoft <br> https://arxiv.org/abs/1811.07275<br>
Aaditya Prakash, James Storer, Dinei Florencio, Cha Zhang, Brandeis and Microsoft https://arxiv.org/abs/1811.07275
A training schedule using filter pruning and orthogonal reinitialization
A training schedule using filter pruning and orthogonal reinitialization
