Understanding GPT tokenizers
Understanding GPT tokenizers
simonwillison.net
Understanding GPT tokenizers
Large language models such as GPT-3/4, LLaMA and PaLM work in terms of tokens. They take text, convert it into tokens (integers), then predict which tokens should come next. Playing …
Saw this on HN and thought it was very interesting. Also wanted to test creating a post on Lemmy :)