↓ Skip to main content

Team seminars

What do multimodal LLMs learn?

Speaker
Benoit Favre (Aix-Marseille Université)
Date
Room
Salle 306, bâtiment IMAG

Instruct-tuned language models have changed the way we interact with computers through language. A lot of work is now devoted to outlining the envelope of their capabilities and their limitations. In this talk, I will describe my tentative to tackle a subset of this problem in the context of multimodal learning. The first line of work I will present is devoted to understanding how vision-language models (VLM) represent fine-grained relationships between objects. Previously, we had shown that VLMs trained on image-text matching fail to extract reliable relationships between objects. I will report on an updated set of experiments to see how those capabilities have evolved in recent instruction-tuned multimodal models. The second line of work aims at better understanding the space of tasks that one may ask from a LLM, in order to select better training mixes and adequately evaluate them. I will evoke past work where we tried to decompose the task space into unitary skills from an introspective and benchmark availability perspective, and more recent work which tries to address this problem from a statistical point of view, allowing to build empirically a hierarchy of task solving capabilities. Finally, I will describe my ongoing project on building foundation models of multimodal conversation. Drawing from video generation techniques, this project aims at assessing the representation capabilities of models able to continue a human-human conversation, both in the audio and video modality. I will describe a framework for studying those capabilities through probing underlying representations according to various tasks. This project has applications in studying human conversation behavior, as well as simulating this behavior in the context of robotic interactions.