Deep Learning Seminar, Winter 2018/19
In recent years, deep neural networks have been used to solve complex machine-learning problems and have achieved significant state-of-the-art results in many areas. The whole field of deep learning has been developing rapidly, with new methods and techniques emerging steadily.
The goal of the seminar is to follow the newest advancements in the deep learning field. The course takes form of a reading group – each lecture a paper is presented by one of the students. The paper is announced in advance, hence all participants can read it beforehand and can take part in the discussion of the paper.
If you want to receive announcements about chosen paper, sign up to our mailing list ufal-rg@googlegroups.com.
About
SIS code: NPFL117
Semester: winter + summer
E-credits: 3
Examination: 0/2 C
Guarantor: Milan Straka
Timespace Coordinates
The Deep Learning Seminar takes place on Tuesday at 14:00 in S8. We will first meet on Tuesday Oct 09.
Requirements
To pass the course, you need to present a research paper and sufficiently attend the presentations.
License
Unless otherwise stated, teaching materials for this course are available under CC BY-SA 4.0.
If you want to receive announcements about chosen paper, sign up to our mailing list ufal-rg@googlegroups.com.
To add your name and paper to the table below, edit the source code on GitHub and send a PR.
| Date | Who | Paper(s) |
|---|---|---|
| 09 Oct 2018 | Milan Straka | Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, Armand Joulin: Advances in Pre-Training Distributed Word Representations Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, Tomas Mikolov: Learning Word Vectors for 157 Languages Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, Russell Power: Semi-supervised sequence tagging with bidirectional language models Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer: Deep contextualized word representations Alan Akbik, Duncan Blythe, Roland Vollgraf: Contextual String Embeddings for Sequence Labeling Samuel L. Smith, David H. P. Turban, Steven Hamblin, Nils Y. Hammerla: Offline bilingual word vectors, orthogonal transformations and the inverted softmax Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, Hervé Jégou: Word Translation Without Parallel Data Anders Søgaard, Sebastian Ruder, Ivan Vulić: On the Limitations of Unsupervised Bilingual Dictionary Induction Mareike Hartmann, Yova Kementchedjhieva, Anders Søgaard: Why is unsupervised alignment of English embeddings from different algorithms so hard? Mikel Artetxe, Gorka Labaka, Eneko Agirre: A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings |
| 16 Oct 2018 | Tomas Soucek | Martin Arjovsky, Soumith Chintala, Léon Bottou: Wasserstein GAN Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, Aaron Courville: Improved Training of Wasserstein GANs Zhiming Zhou, Yuxuan Song, Lantao Yu, Hongwei Wang, Zhihua Zhang, Weinan Zhang, Yong Yu: Understanding the Effectiveness of Lipschitz-Continuity in Generative Adversarial Nets |
| 23 Oct 2018 | Marek Černý | Nikolaus Mayer, Eddy Ilg, Philip Häusser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, Thomas Brox: A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation Clément Godard, Oisin Mac Aodha, Gabriel J. Brostow: Unsupervised Monocular Depth Estimation with Left-Right Consistency Tinghui Zhou, Matthew Brown, Noah Snavely, David G. Lowe:Unsupervised Learning of Depth and Ego-Motion from Video Reza Mahjourian, Martin Wicke, Anelia Angelova: Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints Andrea Pilzer, Dan Xu, Mihai Marian Puscas, Elisa Ricci, Nicu Sebe: Unsupervised Adversarial Depth Estimation using Cycled Generative Networks Sudeep Pillai, Rares Ambrus, Adrien Gaidon: SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation Richard Chen, Faisal Mahmood, Alan Yuille, Nicholas J. Durr: Rethinking Monocular Depth Estimation with Adversarial Training |
| 30 Oct 2018 | No seminar | |
| 06 Nov 2018 | Dean's Sport Day | |
| 13 Nov 2018 | Petr Laitoch | Timothy Dozat, Christopher D. Manning: Deep Biaffine Attention for Neural Dependency Parsing Michael Ringgaard, Rahul Gupta, Fernando C. N. Pereira: SLING: A framework for frame semantic parsing |
| 20 Nov 2018 | Eric Lief | Dušan Variš, Natalia Klyueva: Improving a Neural-based Tagger for Multiword Expression Identification Regina Stodden, Behrang QasemiZadeh, Laura Kallmeyer: TRAPACC and TRAPACC_S at PARSEME Shared Task 2018: Neural Transition Tagging of Verbal Multiword Expressions |
| 27 Nov 2018 | Ondřej Měkota | Thomas SCHLEGL, Philipp SEEBÖCK, Sebastian M. WALDSTEIN, Ursula SCHMIDT-ERFURTH and Georg LANGS: Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery |
| 04 Dec 2018 | Martin Víta | Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, Antoine Bordes: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Qianlong Du, Chengqing Zong, Keh-Yih Su: Adopting the Word-Pair-Dependency-Triplets with Individual Comparison for Natural Language Inference |
| 11 Dec 2018 | Miroslav Krabec | Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space Hang Su, Subhransu Maji, Evangelos Kalogerakis, Erik Learned-Miller:Multi-view Convolutional Neural Networks for 3D Shape Recognition Asako Kanezaki, Yasuyuki Matsushita, Yoshifumi Nishida: RotationNet: Joint Object Categorization and Pose Estimation Using Multiviews from Unsupervised Viewpoints Andrew Brock, Theodore Lim, J.M. Ritchie, Nick Weston: Generative and Discriminative Voxel Modeling with Convolutional Neural Networks |
| 18 Dec 2018 | Karry Hořeňovská | Robin Jia, Percy Liang: Adversarial Examples for Evaluating Reading Comprehension Systems |
| 25 Dec 2018 | Christmas Holiday | |
| 01 Jan 2019 | New Year's Day | |
| 08 Jan 2019 | Petr Houška | Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, William T Freeman, Joshua B Tenenbaum: MarrNet: 3D Shape Reconstruction via 2.5D Sketches Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Joshua B. Tenenbaum, William T. Freeman, Jiajun Wu: Learning to Reconstruct Shapes from Unseen Classes |
You can choose any paper you find interesting, but if you would like some inspiration, you can look at the following list.
Current Deep Learning Papers
Parsing
Timothy Dozat, Christopher D. Manning: Deep Biaffine Attention for Neural Dependency ParsingMichael Ringgaard, Rahul Gupta, Fernando C. N. Pereira: SLING: A framework for frame semantic parsing
Neural Machine Translation
- Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, Richard Socher: Non-Autoregressive Neural Machine Translation
- Peter Shaw, Jakob Uszkoreit, Ashish Vaswani: Self-Attention with Relative Position Representations
- Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, Marc'Aurelio Ranzato: Phrase-Based & Neural Unsupervised Machine Translation
Language Modelling
- Gábor Melis, Chris Dyer, Phil Blunsom: On the State of the Art of Evaluation in Neural Language Models
- Anirudh Goyal, Nan Rosemary Ke, Alex Lamb, R Devon Hjelm, Chris Pal, Joelle Pineau, Yoshua Bengio: ACtuAL: Actor-Critic Under Adversarial Learning
Natural Language Generation
- Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, Eric P. Xing: Toward Controlled Generation of Text
- Sai Rajeswar, Sandeep Subramanian, Francis Dutil, Christopher Pal, Aaron Courville: Adversarial Generation of Natural Language
Speech Synthesis
- Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, Demis Hassabis: Parallel WaveNet: Fast High-Fidelity Speech Synthesis
- Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerry-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, Yonghui Wu: Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
Image Recognition
- Yuntian Deng, Anssi Kanervisto, Alexander M. Rush: What You Get Is What You See: A Visual Markup Decompiler
Image Enhancement
- Ryan Dahl, Mohammad Norouzi, Jonathon Shlens: Pixel Recursive Super Resolution
Image 3D Reconstruction
- Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, William T Freeman, Joshua B Tenenbaum: MarrNet: 3D Shape Reconstruction via 2.5D Sketches
Training Methods
- James Martens, Roger Grosse: Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Roger Grosse, James Martens: A Kronecker-factored approximate Fisher matrix for convolution layers
- James Martens, Jimmy Ba, Matt Johnson: Kronecker-factored Curvature Approximations for Recurrent Neural Networks
- Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, Kaiming He: Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Noam Shazeer, Mitchell Stern: Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
Activation Functions
- Prajit Ramachandran, Barret Zoph, Quoc V. Le: Searching for Activation Functions
Regularization
- Sergey Ioffe: Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz: mixup: Beyond Empirical Risk Minimization
Network Interpretation
- Hao Li, Zheng Xu, Gavin Taylor, Tom Goldstein: Visualizing the Loss Landscape of Neural Nets
- Leila Arras, Grégoire Montavon, Klaus-Robert Müller, Wojciech Samek: Explaining Recurrent Neural Network Predictions in Sentiment Analysis
Reinforcement Learning
- Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, Raia Hadsell: Learning to Navigate in Complex Environments
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov: Proximal Policy Optimization Algorithms
- Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver: Rainbow: Combining Improvements in Deep Reinforcement Learning
- Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine: Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
Explicit Memory
- Caglar Gulcehre, Sarath Chandar, Yoshua Bengio: Memory Augmented Neural Networks with Wormhole Connections
Hyperparameter Optimization
- Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Elliot Karro, D. Sculley: Google Vizier: A Service for Black-Box Optimization
Generative Adversarial Networks
- Lantao Yu, Weinan Zhang, Jun Wang, Yong Yu: SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
- Leon Sixt, Benjamin Wild, Tim Landgraf: RenderGAN: Generating Realistic Labeled Data
- Ilya Tolstikhin, Sylvain Gelly, Olivier Bousquet, Carl-Johann Simon-Gabriel, Bernhard Schölkopf: AdaGAN: Boosting Generative Models
- Martin Arjovsky, Soumith Chintala, Léon Bottou: Wasserstein GAN
Adversarial Text
Robin Jia, Percy Liang: Adversarial Examples for Evaluating Reading Comprehension Systems- Zhengli Zhao, Dheeru Dua, Sameer Singh: Generating Natural Adversarial Examples
- Yonatan Belinkov, Yonatan Bisk: Synthetic and Natural Noise Both Break Neural Machine Translation
- Javid Ebrahimi, Anyi Rao, Daniel Lowd, Dejing Dou: HotFlip: White-Box Adversarial Examples for Text Classification
- Ji Gao, Jack Lanchantin, Mary Lou Soffa, Yanjun Qi: Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers
- Mohit Iyyer, John Wieting, Kevin Gimpel, Luke Zettlemoyer: Adversarial Example Generation with Syntactically Controlled Paraphrase Networks
Adversarial Speech
- Nicholas Carlini, David Wagner: Audio Adversarial Examples: Targeted Attacks on Speech-to-Text
- Xuejing Yuan, Yuxuan Chen, Yue Zhao, Yunhui Long, Xiaokang Liu, Kai Chen, Shengzhi Zhang, Heqing Huang, Xiaofeng Wang, Carl A. Gunter: CommanderSong: A Systematic Approach for Practical Adversarial Voice Recognition
Artificial Intelligence
- Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, Igor Mordatch: Emergent Complexity via Multi-Agent Competition
- David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, Demis Hassabis: Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
