#044 - Data-efficient Image Transformers (Hugo Touvron)

Today we are going to talk about the *Data-efficient image Transformers paper or (DeiT) which Hugo is the primary author of. One of the recipes of success for vision models since the DL revolution began has been the availability of large training sets. CNNs have been optimized for almost a decade now, including through extensive architecture search which is prone to overfitting. Motivated by the success of transformers-based models in Natural Language Processing there has been increasing attention in applying these approaches to vision models. Hugo and his collaborators used a different training strategy and a new distillation token to get a massive increase in sample efficiency with image transformers.  00:00:00 Introduction 00:06:33 Data augmentation is all you need 00:09:53 Now the image patches are the convolutions though? 00:12:16 Where are those inductive biases hiding? 00:15:46 Distillation token 00:21:01 Why different resolutions on training 00:24:14 How data efficient can we get? 00:26:47 Out of domain generalisation 00:28:22 Why are transformers data efficient at all? Learning invariances 00:32:04 Is data augmentation cheating? 00:33:25 Distillation strategies - matching the intermediatae teacher representation as well as output 00:35:49 Do ML models learn the same thing for a problem? 00:39:01 How is it like at Facebook AI? 00:41:17 How long is the PhD programme? 00:42:03 Other interests outside of transformers? 00:43:18 Transformers for Vision and Language 00:47:40 Could we improve transformers models? (Hybrid models) 00:49:03 Biggest challenges in AI? 00:50:52 How far can we go with data driven approach?

Om Podcasten

Welcome! We engage in fascinating discussions with pre-eminent figures in the AI field. Our flagship show covers current affairs in AI, cognitive science, neuroscience and philosophy of mind with in-depth analysis. Our approach is unrivalled in terms of scope and rigour – we believe in intellectual diversity in AI, and we touch on all of the main ideas in the field with the hype surgically removed. MLST is run by Tim Scarfe, Ph.D (https://www.linkedin.com/in/ecsquizor/) and features regular appearances from MIT Doctor of Philosophy Keith Duggar (https://www.linkedin.com/in/dr-keith-duggar/).