The tokenizer currently only holds information about the current line number (along with the starting line number for strings that span multiple lines). This makes computing column offset harder, since it has to be done with pointer arithmetic using pointers to the beginning and end of the token (even more complicated when line continuations or multiline tokens happen).

Feature or enhancement

All of this can be significantly simplified, if we keep a column offset counter in the tokenizer state.