The first in a series of problems that give a glimpse into the work we do at Cursor. Setup When using a language model for code completion, we typically want the model to produce a completion that begins with what the user has typed. However, modern language models operate on sequences of tokens, not characters, so naively tokenizing the user's input and sending it to the model produces wrong results if the user's cursor doesn't happen to lie on a token boundary. Instead, we need an algorithm th...