GigaToken: ~1000x faster Language model tokenization
By syrusakbary · 2026-07-22 · 220 points · 45 comments
https://github.com/marcelroed/gigatoken/
By syrusakbary · 220 points · 45 comments · on Hacker News, read on BetterNews.
Open the full discussion on BetterNews