A font where every token is exactly the same width
Font where each token is equal-width
Developer &. and Astra built a font that gives every tokenizer token equal width, so you can see text the way a language model actually reads it. An early prototype drew huge attention, and a web tool now lets anyone combine a base font with a tokenizer of their choice. Several tokenizers are over 99% accurate, though GLM-5 and LLaMa 3 remain unreliable. The font uses OpenType GSUB and cmap tricks, since OpenType caps ligatures at 65,536.
OpenType has a 65,536 ligature limit and tokenizers are fucked up so this had to be implemented using GSUB and cmap.
- gchamonlive
Don't you have anywhere else you can share it other than twitter?
Inb4 I'm not going to use life support systems like xcancel, xdying is long overdue
- happytoexplain
Ha, I like that, but to a layman it looks like the author is trying to say "it can't count the R's in strawberry because the letters are overlapping", which is funny.
- dahart
Cute idea, but is this fixed width words, and not fixed width tokens? Isn’t an ‘ing’ ending normal a separate token from the root? Isn’t ‘strawberry’ two tokens?