A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language
Models
![[R logo]](../../../doc/html/Rlogo.svg)
![[Up]](../../../doc/html/left.jpg)
![[Top]](../../../doc/html/up.jpg)
Documentation for package ‘rtiktoken’ version 0.0.7
Help Pages
| decode_tokens | Decodes tokens back to text |
| get_tokens | Converts text to tokens |
| get_token_count | Returns the number of tokens in a text |
| model_to_tokenizer | Gets the name of the tokenizer used by a model |