Street View

Back to the whiteboard with 3Blue1Brown

In our recent roundtable with Grant Sanderson of 3Blue1Brown, Alok, a researcher on our ML team, mentioned that he likes to give impromptu math and physics talks. So he and Grant headed to the whiteboard to explore a question Alok had been thinking about.

In this new video, Alok and Grant walk through his recent blog post, Using group theory to explore the space of positional encodings for attention. Attention has no built-in notion of position, so models need some way to encode it, and Alok wanted to know what the full menu of options is. It turns out to be a short menu: assume a few reasonable properties and everything collapses to a one-parameter group, which leaves room for growth, decay, and rotation, and not much else. Along the way the two of them derive RoPE, recover ALiBi, and find a use for a frictionless hockey puck.

Watch the full session now!