Dynamic metastability in self-attention dynamics

• Borjan Geshkovski

Room: Building 1b, Seminar Room 4ab

A multiscale analysis of mean-field transformers in the moderate interaction regime

• Giuseppe Bruno

Room: Building 1b, Seminar Room 4ab In this talk, we study the evolution of tokens across the depth of encoder-only transformer models at inference time, modeling them as a system of interacting particles in the infinite-depth limit. Motivated by techniques for extending the context length of large language models, we focus on the moderate interaction regime, where the number of tokens is large and the inverse temperature parameter scales accordingly. In this setting, the dynamics exhibit a multiscale structure. Using PDE analysis, we identify different phases depending on the choice of parameters.

Mean-Field Transformer Dynamics with Gaussian Inputs

• Valérie Castin

Room: Building 1b, Seminar Room 4ab Transformers, that underlie the recent successes of large language models, represent the data as sequences of vectors called tokens. This representation is leveraged by the attention function, which learns dependencies between tokens and is key to the success of Transformers. However, the dynamics induced by the iterative application of attention across layers remain to be fully understood. To analyze these dynamics, we identify each input sequence with a probability measure, thus handling input sequences of arbitrary length, and model its evolution as a Vlasov equation called Transformer PDE, whose velocity field is non-linear in the probability measure. For compactly supported initial data and several self-attention variants, we show the Transformer PDE is well-posed and is the mean-field limit of an interacting particle system. We also study the case of Gaussian initial data, which has the nice property of staying Gaussian across the dynamics. This allows us to identify typical behaviors theoretically and numerically, and to highlight a clustering phenomenon that parallels previous results in the discrete case.

Autoregressive in-context learning with Transformers

• Michael Sander

Room: Building 1b, Seminar Room 4ab

TBA

• Subhabrata Dutta

Room: Building 1b, Seminar Room 4ab