Deep Learning Seminar, Summer 2019/20
In recent years, deep neural networks have been used to solve complex machine-learning problems and have achieved significant state-of-the-art results in many areas. The whole field of deep learning has been developing rapidly, with new methods and techniques emerging steadily.
The goal of the seminar is to follow the newest advancements in the deep learning field. The course takes form of a reading group – each lecture a paper is presented by one of the students. The paper is announced in advance, hence all participants can read it beforehand and can take part in the discussion of the paper.
If you want to receive announcements about chosen paper, sign up to our mailing list ufal-rg@googlegroups.com.
About
SIS code: NPFL117
Semester: winter + summer
E-credits: 3
Examination: 0/2 C
Guarantor: Milan Straka
Timespace Coordinates
The Deep Learning Seminar takes place on Monday at 14:00 in S11. We will first meet on Monday Feb 24.
Requirements
To pass the course, you need to present a research paper and sufficiently attend the presentations.
License
Unless otherwise stated, teaching materials for this course are available under CC BY-SA 4.0.
If you want to receive announcements about chosen paper, sign up to our mailing list ufal-rg@googlegroups.com.
To add your name to a paper the table below, edit the source code on GitHub and send a PR.
| Date | Who | Topic | Paper(s) |
|---|---|---|---|
| 24 Feb | Milan Straka | CNNs, AutoML | Mingxing Tan, Quoc V. Le: EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks Mingxing Tan, Ruoming Pang, Quoc V. Le: EfficientDet: Scalable and Efficient Object Detection |
| 02 Mar | Jana Rezabkova | Networks with External Memory | Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap: One-shot Learning with Memory-Augmented Neural Networks |
| 09 Mar | Jonáš Kulhánek | DL training, Symbolic DL, SRN | Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever: Deep Double Descent: Where Bigger Models and More Data Hurt Guillaume Lample, François Charton: Deep Learning for Symbolic Mathematics Vincent Sitzmann, Michael Zollhöfer, Gordon Wetzstein: Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations |
| 16 Mar | Ondřej Měkota Paper Summary |
Transformer, Chatbot | David R. So, Chen Liang, Quoc V. Le: The Evolved Transformer (blog) Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, Quoc V. Le: Towards a Human-like Open-Domain Chatbot (blog) |
| 23 Mar | Tomáš Kremel Paper Summary |
AutoML | Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, Jeff Dean: Efficient Neural Architecture Search via Parameter Sharing Hanxiao Liu, Karen Simonyan, Yiming Yang: DARTS: Differentiable Architecture Search |
| 30 Mar | Vastl Martin Paper Summary |
Transformers, BERT | Anna Rogers, Olga Kovaleva, Anna Rumshisky A Primer in BERTology: What we know about how BERT works |
| 06 Apr | Štěpán Procházka | Conditional Random Fields (papers TBA) | |
| 13 Apr | No DL Seminar | Easter Monday | |
| 20 Apr | Kačka Macková Paper Summary |
Q&A | Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang. REALM: Retrieval-Augmented Language Model Pre-Training |
| 27 Apr | Jan Waltl | Not sure yet. | |
| 04 May | Ladislav Malecek | TBA | |
| 11 May | Marek Dobransky Paper Summary |
GAN | Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio: Generative Adversarial Networks Martin Arjovsky, Soumith Chintala, Léon Bottou: Wasserstein GAN Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, Aaron Courville: Improved Training of Wasserstein GANs |
| 18 May | |||
| 22 Sep | Sourabrata Mukherjee Paper Summary |
StyleTransfer | Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang. Style Transfer Through Back-Translation |
You can choose any paper you find interesting, but if you would like some inspiration, you can look at the following list. The papers are grouped, each group is expected to be presented on one seminar.
Natural Language Processing
-
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov: RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning: ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
-
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov: Unsupervised Cross-lingual Representation Learning at Scale
Generative Modeling
-
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio: Generative Adversarial NetworksMartin Arjovsky, Soumith Chintala, Léon Bottou: Wasserstein GAN
-
- Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen: Progressive Growing of GANs for Improved Quality, Stability, and Variation
- Andrew Brock, Jeff Donahue, Karen Simonyan: Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Tero Karras, Samuli Laine, Timo Aila: A Style-Based Generator Architecture for Generative Adversarial Networks
Neural Architecture Search (AutoML)
-
- Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le: Learning Transferable Architectures for Scalable Image Recognition
- Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V Le: Regularized Evolution for Image Classifier Architecture Search
- Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, Kevin Murphy: Progressive Neural Architecture Search
-
Mingxing Tan, Quoc V. Le: EfficientNet: Rethinking Model Scaling for Convolutional Neural NetworksMingxing Tan, Ruoming Pang, Quoc V. Le: EfficientDet: Scalable and Efficient Object Detection
Networks with External Memory
-
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap: One-shot Learning with Memory-Augmented Neural Networks -
Mark Collier, Joeran Beel: Memory-Augmented Neural Networks for Machine Translation
