| ... | ... | @@ -17,17 +17,16 @@ If you’re interested in more details about the Journal Club, please subscribe |
|
|
|
|
|
|
|
### 17 August 2020 - Attention Networks
|
|
|
|
|
|
|
|
Venue: if possible f2f INM seminar room, bldg. 15.9, room 4001b <br>
|
|
|
|
Venue: if possible f2f INM seminar room, bldg. 15.9, room 4001b
|
|
|
|
alternatively online: https://webconf.fz-juelich.de/b/wen-mym-pj7
|
|
|
|
|
|
|
|
|
|
|
|
* Attention Is All You Need<br>
|
|
|
|
Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin, NIPS 2017 <br>
|
|
|
|
https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf <br>
|
|
|
|
* Attention Is All You Need
|
|
|
|
Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin, NIPS 2017
|
|
|
|
https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf
|
|
|
|
9 pages
|
|
|
|
* Self-Attention Generative Adversarial Networks<br>
|
|
|
|
Zhang, Goodfellow, Metaxas, Odena, ICML 2019<br>
|
|
|
|
https://arxiv.org/abs/1805.08318<br>
|
|
|
|
* Self-Attention Generative Adversarial Networks
|
|
|
|
Zhang, Goodfellow, Metaxas, Odena, ICML 2019
|
|
|
|
https://arxiv.org/abs/1805.08318
|
|
|
|
8 pages
|
|
|
|
|
|
|
|
|
| ... | ... | @@ -36,17 +35,16 @@ https://arxiv.org/abs/1805.08318<br> |
|
|
|
|
|
|
|
### 21 September 2020 Self-Supervised Visual Representation Learning
|
|
|
|
|
|
|
|
Venue: if possible f2f INM seminar room, bldg. 15.9, room 4001b <br>
|
|
|
|
Venue: if possible f2f INM seminar room, bldg. 15.9, room 4001b
|
|
|
|
alternatively online: https://webconf.fz-juelich.de/b/wen-mym-pj7
|
|
|
|
|
|
|
|
|
|
|
|
* A Simple Framework for Contrastive Learning of Visual Representations <br>
|
|
|
|
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, 2020<br>
|
|
|
|
http://arxiv.org/abs/2002.05709<br>
|
|
|
|
* A Simple Framework for Contrastive Learning of Visual Representations
|
|
|
|
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, 2020
|
|
|
|
http://arxiv.org/abs/2002.05709
|
|
|
|
15 pages, incl appendix
|
|
|
|
* Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey <br>
|
|
|
|
L. Jing and Y. Tian, CVPR2019<br>
|
|
|
|
http://arxiv.org/abs/1902.06162<br>
|
|
|
|
* Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
|
|
|
|
L. Jing and Y. Tian, CVPR2019
|
|
|
|
http://arxiv.org/abs/1902.06162
|
|
|
|
21 pages
|
|
|
|
|
|
|
|
The first paper presents a state-of-the-art approach for self-supervised learning of strong visual features based on contrastive learning. Random data augmentation is applied to images from the ImageNet dataset and a model is trained to match augmented and original images. The second paper revisits several self-supervised training techniques for visual representation learning and offers a nice overview over different approaches.
|
| ... | ... | @@ -56,45 +54,45 @@ The first paper presents a state-of-the-art approach for self-supervised learnin |
|
|
|
|
|
|
|
### 15 June 2020 - Generative Adversarial Networks (and VAE)
|
|
|
|
|
|
|
|
Venue: JSC meetingroom 2, building 16.3; room 315 <br>
|
|
|
|
Venue: JSC meetingroom 2, building 16.3; room 315
|
|
|
|
online: https://webconf.fz-juelich.de/b/wen-mym-pj7
|
|
|
|
|
|
|
|
* Original GAN paper:<br>
|
|
|
|
Generative Adversarial Networks<br>
|
|
|
|
Goodfellow, Bengio et al, NIPS 2014<br>
|
|
|
|
https://arxiv.org/abs/1406.2661 <br>
|
|
|
|
https://papers.nips.cc/paper/5423-generative-adversarial-nets.html<br>
|
|
|
|
* Original GAN paper:
|
|
|
|
Generative Adversarial Networks
|
|
|
|
Goodfellow, Bengio et al, NIPS 2014
|
|
|
|
https://arxiv.org/abs/1406.2661
|
|
|
|
https://papers.nips.cc/paper/5423-generative-adversarial-nets.html
|
|
|
|
8 pages
|
|
|
|
* On relation GAN, VAE:<br>
|
|
|
|
On Unifying Deep Generative Models<br>
|
|
|
|
Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, Eric P. Xing, ICLR 2017<br>
|
|
|
|
https://arxiv.org/abs/1706.00550 <br>
|
|
|
|
https://openreview.net/forum?id=rylSzl-R- <br>
|
|
|
|
* On relation GAN, VAE:
|
|
|
|
On Unifying Deep Generative Models
|
|
|
|
Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, Eric P. Xing, ICLR 2017
|
|
|
|
https://arxiv.org/abs/1706.00550
|
|
|
|
https://openreview.net/forum?id=rylSzl-R-
|
|
|
|
16 pages, incl appendix
|
|
|
|
|
|
|
|
### Monday 18 May Variational Autoencoders
|
|
|
|
|
|
|
|
[Minutes](200518JournalClubMinutes)
|
|
|
|
|
|
|
|
* Original VAE paper<br>
|
|
|
|
Auto-Encoding Variational Bayes<br>
|
|
|
|
Diederik P Kingma, Max Welling, ICLR 2014<br>
|
|
|
|
https://arxiv.org/abs/1312.6114 <br>
|
|
|
|
https://openreview.net/forum?id=33X9fd2-9FyZd <br>
|
|
|
|
* Original VAE paper
|
|
|
|
Auto-Encoding Variational Bayes
|
|
|
|
Diederik P Kingma, Max Welling, ICLR 2014
|
|
|
|
https://arxiv.org/abs/1312.6114
|
|
|
|
https://openreview.net/forum?id=33X9fd2-9FyZd
|
|
|
|
14 pages, incl appendix
|
|
|
|
* Recent VAE review / tutorial<br>
|
|
|
|
An Introduction to Variational Autoencoders<br>
|
|
|
|
Diederik P. Kingma, Max Welling (2019) Foundations and Trends in Machine Learning. 12. 307-392. 10.1561/2200000056. <br>
|
|
|
|
https://arxiv.org/abs/1906.02691<br>
|
|
|
|
86 pages <br>
|
|
|
|
* Recent VAE review / tutorial
|
|
|
|
An Introduction to Variational Autoencoders
|
|
|
|
Diederik P. Kingma, Max Welling (2019) Foundations and Trends in Machine Learning. 12. 307-392. 10.1561/2200000056.
|
|
|
|
https://arxiv.org/abs/1906.02691
|
|
|
|
86 pages
|
|
|
|
|
|
|
|
Discussion notes: <https://gitlab.version.fz-juelich.de/codiMD/9pd1RHfHTTqAB7XqgX-O2A>
|
|
|
|
|
|
|
|
### Monday 20 April - Learning discrete representations from data
|
|
|
|
|
|
|
|
* A. van den Oord, O. Vinyals, K. Kavukcuoglu, Neural Discrete Representation Learning, NeurIPS 2017<br>
|
|
|
|
* A. van den Oord, O. Vinyals, K. Kavukcuoglu, Neural Discrete Representation Learning, NeurIPS 2017
|
|
|
|
https://arxiv.org/abs/1711.00937
|
|
|
|
* A. Razavi, A. van den Oord, O. Vinyals, Generating Diverse High-Fidelity Images with VQ-VAE-2, NeurIPS 2019<br>
|
|
|
|
* A. Razavi, A. van den Oord, O. Vinyals, Generating Diverse High-Fidelity Images with VQ-VAE-2, NeurIPS 2019
|
|
|
|
https://arxiv.org/abs/1906.00446
|
|
|
|
|
|
|
|
Those two papers are about learning discrete representations from data by taking inspiration from vector quantization. Learning discrete representations using neural networks is challenging and can helpful for tasks such as compression, planning, reasoning, and can be potentially more interpretable than continuous ones. In the two papers use those learned discrete representations to build autoregressive generative models on image, sound, and video. The second paper (Generating Diverse High-Fidelity Images with VQ-VAE-2) is basically a sequel of the first (Neural Discrete Representation Learning) where they scale the models to bigger datasets and images (up to 1024x1024 resolution).
|
| ... | ... | @@ -104,12 +102,12 @@ Those two papers are about learning discrete representations from data by taking |
|
|
|
|
|
|
|
**Replacement date for canceled meeting at Monday 16 March**
|
|
|
|
|
|
|
|
* Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes (CVPR'17 Oral)<br>
|
|
|
|
T. Pohlen, A. Hermans, M. Mathias, and B. Leibe<br>
|
|
|
|
Paper: http://arxiv.org/abs/1611.08323 <br>
|
|
|
|
* Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes (CVPR'17 Oral)
|
|
|
|
T. Pohlen, A. Hermans, M. Mathias, and B. Leibe
|
|
|
|
Paper: http://arxiv.org/abs/1611.08323
|
|
|
|
Code: https://github.com/TobyPDE/FRRN
|
|
|
|
* Deep High-Resolution Representation Learning for Visual Recognition, Wang et al. 2019<br>
|
|
|
|
Paper: https://arxiv.org/abs/1908.07919<br>
|
|
|
|
* Deep High-Resolution Representation Learning for Visual Recognition, Wang et al. 2019
|
|
|
|
Paper: https://arxiv.org/abs/1908.07919
|
|
|
|
Code: https://paperswithcode.com/paper/deep-high-resolution-representation-learning-2
|
|
|
|
|
|
|
|
Most common architectures for semantic segmentation consisting of an encoder and decoder part (e.g. U-Net) heavily reduce the spatial dimension of input images and may loose important details or fail to localize precisely. The proposed papers present full-resolution networks, which try to preserve high-resolution features throughout the network and improve localization accuracy.
|
| ... | ... | @@ -117,27 +115,27 @@ Most common architectures for semantic segmentation consisting of an encoder and |
|
|
|
|
|
|
|
### Monday 17 February - Speech Recognition
|
|
|
|
|
|
|
|
* Deep Speech 2: End-to-End Speech Recognition in English and Mandarin, Amodei et al., 2015<br>
|
|
|
|
https://arxiv.org/abs/1512.02595<br>
|
|
|
|
* Deep Speech 2: End-to-End Speech Recognition in English and Mandarin, Amodei et al., 2015
|
|
|
|
https://arxiv.org/abs/1512.02595
|
|
|
|
Not the most recent paper about speech recognition, but a breakthrough for language modelling, thus worth reading. It discusses some topics, which are relevant for timeseries analysis, and also makes reference to good use of HPC. Following this topic, we could continue with latest papers on that at next Journal Club.
|
|
|
|
* and, if needed, as background paper about deep recurrent networks:<br>
|
|
|
|
Speech Recognition with Deep Recurrent Neural Networks<br>
|
|
|
|
Alex Graves, Abdel-rahman Mohamed, Geoffrey Hinton, 2013<br>
|
|
|
|
* and, if needed, as background paper about deep recurrent networks:
|
|
|
|
Speech Recognition with Deep Recurrent Neural Networks
|
|
|
|
Alex Graves, Abdel-rahman Mohamed, Geoffrey Hinton, 2013
|
|
|
|
https://arxiv.org/abs/1303.5778
|
|
|
|
|
|
|
|
### Monday 20 January
|
|
|
|
|
|
|
|
* Multi-Context Recurrent Neural Networks for Time Series Applications <br>
|
|
|
|
* Multi-Context Recurrent Neural Networks for Time Series Applications
|
|
|
|
https://publications.waset.org/3524/pdf
|
|
|
|
* Global Sparse Momentum SGD for Pruning Very Deep Neural Networks <br>
|
|
|
|
* Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
|
|
|
|
https://arxiv.org/pdf/1909.12778v3.pdf
|
|
|
|
|
|
|
|
|
|
|
|
### Monday 16 December
|
|
|
|
|
|
|
|
Our first Journal Club will cover two papers from ICCV 2019 about GANs.
|
|
|
|
* SinGAN: Learning a Generative Model from a Single Natural Image (Best Paper Award) <br> http://openaccess.thecvf.com/content_ICCV_2019/papers/Shaham_SinGAN_Learning_a_Generative_Model_From_a_Single_Natural_Image_ICCV_2019_paper.pdf
|
|
|
|
* COCO-GAN: Generation by Parts via Conditional Coordinating <br>
|
|
|
|
* SinGAN: Learning a Generative Model from a Single Natural Image (Best Paper Award) http://openaccess.thecvf.com/content_ICCV_2019/papers/Shaham_SinGAN_Learning_a_Generative_Model_From_a_Single_Natural_Image_ICCV_2019_paper.pdf
|
|
|
|
* COCO-GAN: Generation by Parts via Conditional Coordinating
|
|
|
|
full paper and supplementary material available at https://hubert0527.github.io/COCO-GAN/
|
|
|
|
* Intro Slides by Mickael Cormier available [[here](https://gitlab.version.fz-juelich.de/MLDL_FZJ/General_Wiki/blob/master/files/JournalClub/20191216_gan.pptx)]
|
|
|
|
|
| ... | ... | @@ -145,11 +143,11 @@ full paper and supplementary material available at https://hubert0527.github.io/ |
|
|
|
## Archive paper proposals, not read yet
|
|
|
|
|
|
|
|
* 17.12.2019 Hanno Scharr
|
|
|
|
* A General and Adaptive Robust Loss Function<br>
|
|
|
|
Jonathan T. Barron, Google Research <br> http://openaccess.thecvf.com/content_CVPR_2019/papers/Barron_A_General_and_Adaptive_Robust_Loss_Function_CVPR_2019_paper.pdf<br>
|
|
|
|
* A General and Adaptive Robust Loss Function
|
|
|
|
Jonathan T. Barron, Google Research http://openaccess.thecvf.com/content_CVPR_2019/papers/Barron_A_General_and_Adaptive_Robust_Loss_Function_CVPR_2019_paper.pdf
|
|
|
|
Simple but effective for improving accuracy in regression tasks
|
|
|
|
* RePr: Improved Training of Convolutional Filters<br>
|
|
|
|
Aaditya Prakash, James Storer, Dinei Florencio, Cha Zhang, Brandeis and Microsoft <br> https://arxiv.org/abs/1811.07275<br>
|
|
|
|
* RePr: Improved Training of Convolutional Filters
|
|
|
|
Aaditya Prakash, James Storer, Dinei Florencio, Cha Zhang, Brandeis and Microsoft https://arxiv.org/abs/1811.07275
|
|
|
|
A training schedule using filter pruning and orthogonal reinitialization
|
|
|
|
|
|
|
|
|
| ... | ... | |
