Active Learning for Billion Scale Data

Speaker: Dr. Talfan Evans
Time: 17:00-18:00
Levett Room, Wolfson College & Online (Teams link).

Biography:

Talfan Evans is a research scientist at Google Deepmind working. His work is focused on developing scalable data curation strategies for compute efficient large-scale pretraining. He has a MEng from Keble College and did his PhD at UCL in Cognitive Neuroscience, where he worked on adapting message-passing algorithms from the autonomous driving literature to explain neural activity during spatial exploration. As a postdoc with Andrew Davison at Imperial, he worked on real-time computer vision systems before moving to Deepmind.

Abstract:

Large foundation model scaling laws tell us that to continue to make additive improvements to performance, we should expect to need to pay orders of magnitude more in compute costs and data. In this talk, I'll present work that paints a more optimistic picture - actively choosing which data to train on can shift these curves in our favour, producing significantly more performant models for the same compute budget.

What have Large Language Models actually learned about language(s)? Lessons from linguistics and mechanistic interpretability

Speaker: Dominik Lukeš
Time: 17:30-18:30
Levett Room, Wolfson College & Online (Teams link).

Biography:

Dominik Lukeš is a Lead Business Technologist at the AI and ML Competency Centre with a focus on digital scholarship and academic practice. Prior to joining the Centre he started the Reading and Writing Innovation Lab where he focused on technologies supporting reading and writing in academic contexts.

Dominik's research focus is in linguistics and language pedagogy. He has previously run workshops at Oxford on using corpus analysis tools for humanities research. Dominik's core area of expertise is an intersection of conceptual metaphor theory and discourse analysis with a particular focus on construction grammar. He was the founding member of the journal Cognitive Approaches to Critical Discourse Analysis (CADAAD) and co-edited with Chris Hart the 2007 volume Cognitive Linguistics in Critical Discourse Analysis. He also translated George Lakoff's Women, Fire and Dangerous Things into Czech. He is the author of Czech Navigator, a grammar of Czech for non-native speakers.

He blogs on MetaphorHacker.net and maintains a site focusing on exploring Large Language Models as Semantic Machines and publishes an occasional newsletter on AI in Academic Practice.

Abstract:

This talk will explore what exactly large language models (LLMs) have learned about language, and why it matters. It will attempt an outline of an answer to how these models understand the structure of language, and how this compares to how humans think about language. To answer the question, it will look at how LLMs perform across a range of different languages, even those with smaller digital footprints, and what this implies about how they represent language. It will contrast the results from this investigation with the latest findings from the field of "mechanistic interpretability." These findings offer insights into the inner workings of LLMs, offering clues about the fundamental differences between how we understand language and how language is represented inside the models. Finally, it will suggest a need a new approach to LLM research that brings together a richer understanding of language and a systematic investigation into how LLMs perform across a variety of languages.