A token is a unit produced by a model tokenizer from text or other input. It may be a word, part of a word, punctuation or multimodal data; token count is not the same as word count.
Token (LLM)
A token is a unit produced by a model tokenizer from text or other input. It may be a word, part of a word, punctuation or multimodal data; token count is not the same as word count.