On an Inefficiency in the Linear Attention Projections, and How to Fix It

---

Start: 20.05.2026 @ 18:00

---

We are pleased to invite you to the next PyData meetup, which will take place on Wednesday, May 20, 2026, starting at 18:00 at Base42. This time, we will have the opportunity to host a talk titled “On an Inefficiency in the Linear Attention Projections, and How to Fix It”, presented by Marko Karbevski.

The talk will offer an interesting combination of deep theoretical insights into Transformer architectures and personal experience from conducting independent ML research. During the presentation, Marko will introduce research showing that the Query weights in standard self-attention can be completely eliminated without affecting the functionality of the model. This reduces both the number of parameters and computational complexity by more than 8%, while replacing the redundant linear layer with a small and highly efficient nonlinear module that enables more stable training and improved performance on GPT-style models.

In addition, the talk will cover the mathematical foundations behind the research, including new insights into the role of skip connections in MLP networks and how they fundamentally affect the functions these models are able to represent.

Beyond the technical aspects, Marko will also speak about the challenges of conducting independent ML research — from experiments and hyperparameter sweeps to the experience of participating in the DeLTa and GRaM workshops at this year’s ICLR in Rio de Janeiro 🇧🇷. ICLR is widely recognized as one of the top three scientific machine learning conferences in the world.

About the speaker

---

Marko Karbevski holds a master’s degree in theoretical mathematics and has eight years of experience as a Data Scientist. He graduated in theoretical mathematics and completed his master’s studies at Sorbonne University in France. His interests are focused on the theoretical aspects of machine learning and foundational ML architectures.

Sign up to speak on the next PyData:

./speak.sh