Multimodal Agents [Video]

Antonio Torralba, Delta electronics Professor and head of the AI+D faculty at the Department of Electrical Engineering and Computer Science (EECS) at the Massachusetts Institute of Technology (MIT), hosted a Distinguished Lecturer seminar on Wednesday, Feb. 28, 2024, presenting his talk, “Multimodal Agents” at Northeastern University and online.

Abstract

In the last few years, large pretrained models have shown impressive performance in a diverse set of tasks. These models have to be trained with large datasets and are, in most cases, opaque on how they process information internally. In this talk Antonio Torralba will focus on two questions: how can we build tools to understand the inner workings of existing pretrained models, and how can we use generative models to reduce the need of large databases to train ML systems. Antonio will start describing our recent work on MAIA: A Multimodal Automated Interpretability Agent. This agent performs iterative experimentation on subcomponents of other models to explain their behavior. He will then talk about his work on reducing the amount of data needed to train large pretrained models. Finally, he’ll go a step further and ask if we can do away with real image datasets entirely when building a computer vision system, instead learning from noise processes.

Biography

Antonio Torralba is the Delta electronics Professor and head of the AI+D faculty at the Department of Electrical Engineering and Computer Science (EECS) at the Massachusetts Institute of Technology (MIT). He received the degree in telecommunications engineering from Telecom BCN, Spain, in 1994 and the Ph.D. degree in signal, image, and speech processing from the Institut National Polytechnique de Grenoble, France, in 2000. He received the 2008 National Science Foundation (NSF) Career award, the best student paper award at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in 2009, the 2010 J. K. Aggarwal Prize from the International Association for Pattern Recognition (IAPR), the 2017 Frank Quick Faculty Research Innovation Fellowship, the Louis D. Smullin (’39) Award for Teaching Excellence, the 2020 PAMI Mark Everingham Prize, and was named 2021 AAAI fellow. In 2021, he was awarded the Inaugural Thomas Huang Memorial Prize by the PAMITC. In 2022, he was invested Honoris Causa doctor by the Universitat Politècnica de Catalunya – BarcelonaTech (UPC).