Adaptation of Transformer-based models to low-resourced languages via linguistic proximity and of specific domains
- Speaker
- Miguel López Otal (Universidad de Zaragoza, GETALP)
- Date
- Room
- Room 306, IMAG building
Transformer-based models have seen growing popularity in modern society, academia and industry, with ever increasing adoption. However, current models are hindered by their lack of support for Low Resourced Languages (LRLs), roughly corresponding to minority languages in a linguistic context. Their lack of text corpora means that they have been excluded from model pre-training —or its impact on it has been negligible. In my PhD, I research ways via which LRLs that are typologically related to other high-resourced languages –widely supported in LLMs– can serve as a scaffold to enhance NLU in these models. My thesis deals specifically with Aragonese, a Romance language that is extremely similar to Spanish, with almost identical syntax and close morphology, but differing lexical roots that render the language closer to Catalan and French.
I first delve into my attempts at linguistic interpretability (the “reverse engineering” of linguistic abilities in LLMs), as an initial attempt to discover potential modifiable mechanisms of LLMs for precise linguistic intervention, followed by experiments on knowledge injection of structured linguistic knowledge as well as precise, carefully balanced continuous training of models with small amounts of text. I also present my attempts at bringing Aragonese closer to the digital world, via the creation of evaluation benchmarks for this language, as well as my participation in the CLASiK project, in which GETALP is involved, and from which my research stay at Grenoble is steered at, via the research on methods for model domain adaptation that could benefit language adaptation as well.
