A Ruby gem providing a consistent interface for various AI/ML tokenizers including OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, Mistral, Qwen, and embedding models like BERT, BGE, and multilingual-E5. Features caching, truncation, token counting, and error handling across different tokenization libraries.
Required Ruby Version
>= 3.2.0
Authors
Rafael Silva
Versions
- 0.4.2 February 27, 2026 (20.6 MB)
- 0.4.1 February 26, 2026 (20.6 MB)
- 0.4 January 06, 2026 (20.6 MB)
- 0.3.2 December 10, 2025 (20.6 MB)
- 0.3.1 July 07, 2025 (20.6 MB)
