Pure-Ruby facade over Hugging Face `tokenizers`, OpenAI `tiktoken_ruby`, and SentencePiece bindings that maps ruby_llm model identifiers (gpt-4o, llama-3, mistral, ...) to the correct tokenizer and exposes a small API for counting, analyzing, and truncating text against a model's context window. Includes an opt-in approximation backend for models with no published tokenizer (Claude).

Required Ruby Version

>= 3.3.0

Authors

Sal Scotto

Versions

  1. 0.1.2 June 13, 2026 (16.5 KB)
  2. 0.1.1 June 11, 2026 (16 KB)
  3. 0.1.0 June 05, 2026 (14.5 KB)

Pushed by

SHA 256 checksum