- extend-tiktoken.ipynb contains optional (bonus) code to explain how we can add special tokens to a tokenizer implemented via
tiktokenand how to update the LLM accordingly
Repository navigation
09_extending-tokenizers
Directory actions
More options
Directory actions
More options
09_extending-tokenizers
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|
parent directory.. | ||||