Using Whisper for translation and transcription: an XAI roadmap for audio multilingual LLMs
- Speaker
- Nicolas Ballier (Université Paris Cité)
- Date
- Room
- Salle 306, bâtiment IMAG + visio
Whisper (Radford, 2023) is a multilingual audio Large Language Model (LLM) developed by OpenAI and trained with 680,000 hours of labeled audio data to do transcription and translation into English (as to the free models released on HuggingFace) for 96 languages. This talk will present results and current research based on a customised version of Whisper’s C++ implementation (Gerganov, 2022) developed at Université Paris Cité (Younès 2023). We were able to probe the LLM outputs for French, English and Persian and investigate Whisper’s dictionary of subtokens.
First, reporting on a paper presented at SummitMT, I will present the results of a comparison of the six models designed to translate from French and Persian into English (Ballier et al, 2023).
Second I will show that the transcriptions in English produced by the multilingual medium model are more robust to non-native speech variability than the tiny model transcriptions. In turn, the tiny model transcriptions provide plausible correlates of non-native pronunciation errors (Ballier et al., submitted). The audio LLM might be used as a plausible simulation of misunderstanding scenarios with native speakers. Using the probability of the LLM prediction, we can try to score non-native productions (Ballier et al., in prep).
Then, presenting ongoing experiments with the translation and transcription of Persian data, I will showcase some architectural biases of the multilingual model: hallucinations can be observed in the transcriptions of Persian that can be explained by the absence of some graphemes of Persian in the vocabulary of subtokens. I will also present some of the Americancentered biases of the Whisper models for English and discuss some of the consequences for educational uses of these models).
More generally, I will discuss the reverse engineering strategy that we have adopted. With great help from data scientists and phoneticians, we have created scripts to map back the LLM predictions onto the speech signal. We aim to contribute to XAI (Explainable Artificial Intelligence) by trying to match the out of vocabulary subtokens produced by Whisper to discrete prosodic events. We also try to model how some of the LLM transcriptions may or may not correlate with human perception tests.
