Mathematics of Transformers

Friday 26 Sept 2025, 08:30 → 18:00 Europe/Berlin

Building 1b, Seminar Room 4ab (DESY)

 

This workshop will focus on the transformer architecture and its underlying (self-)attention mechanisms that gained substantial interest in recent years. Despite their empirical success and groundbreaking advances in natural language processing, computer vision, and scientific computing, the mathematical understanding of transformers is still in its infancy, with many fundamental questions only starting to be posed and addressed. 

We aim to bring together researchers with backgrounds in multi-agent dynamics, optimal transport, and PDEs, to initiate discussions on a variety of aspects connected to the theoretical principles governing transformers. By fostering discussions, we seek to advance this young and rapidly evolving research field, uncovering new mathematical perspectives on transformer models.

 

Confirmed speakers

• Giuseppe Bruno (University of Bern)
• Valérie Castin (ENS Paris)
• Subhabrata Dutta (TU Darmstadt)
• Borjan Geshkovski (Inria Paris)
• Michaël E. Sander (Google DeepMind)

 

→ The timetable can be found here.

This is a satellite event to the Conference on Mathematics of Machine Learning 2025 that takes place at TUHH from September 22nd-25th 2025.

We gratefully acknowledge support by the DFG funded priority programme Theoretical Foundations of Deep Learning and Helmholtz Imaging.

08:30 Reception & Coffee

09:00 Welcome & Introduction

1 Dynamic metastability in self-attention dynamics

Speaker: Borjan Geshkovski

2 A multiscale analysis of mean-field transformers in the moderate interaction regime

In this talk, we study the evolution of tokens across the depth of encoder-only transformer models at inference time, modeling them as a system of interacting particles in the infinite-depth limit. Motivated by techniques for extending the context length of large language models, we focus on the moderate interaction regime, where the number of tokens is large and the inverse temperature parameter scales accordingly. In this setting, the dynamics exhibit a multiscale structure. Using PDE analysis, we identify different phases depending on the choice of parameters.

Speaker: Giuseppe Bruno

10:45 Coffee Break

3 Mean-Field Transformer Dynamics with Gaussian Inputs

Transformers, that underlie the recent successes of large language models, represent the data as sequences of vectors called tokens. This representation is leveraged by the attention function, which learns dependencies between tokens and is key to the success of Transformers. However, the dynamics induced by the iterative application of attention across layers remain to be fully understood. To analyze these dynamics, we identify each input sequence with a probability measure, thus handling input sequences of arbitrary length, and model its evolution as a Vlasov equation called Transformer PDE, whose velocity field is non-linear in the probability measure. For compactly supported initial data and several self-attention variants, we show the Transformer PDE is well-posed and is the mean-field limit of an interacting particle system. We also study the case of Gaussian initial data, which has the nice property of staying Gaussian across the dynamics. This allows us to identify typical behaviors theoretically and numerically, and to highlight a clustering phenomenon that parallels previous results in the discrete case.

Speaker: Valérie Castin

12:00 Discussion

12:30 Lunch Break

4 Autoregressive in-context learning with Transformers

Speaker: Michael Sander

5 TBA

Speaker: Subhabrata Dutta

15:30 World Café (discussion format)