Pure-Ruby facade over Hugging Face `tokenizers`, OpenAI `tiktoken_ruby`, and SentencePiece bindings that maps ruby_llm model identifiers (gpt-4o, llama-3, mistral, ...) to the correct tokenizer and exposes a small API for counting, analyzing, and truncating text against a model's context window. Includes an opt-in approximation backend for models with no published tokenizer (Claude).
Required Ruby Version
>= 3.3.0
Authors
Sal Scotto