RuToken is a Ruby gem that wraps the tiktoken Rust library, enabling fast and efficient tokenization for OpenAI models. It supports multiple models including o200k_base, cl100k_base, p50k_base, and r50k_base.
Required Ruby Version
>= 3.2, < 3.5.dev
Required Rubygems Version
>= 3.0.0
Authors
Logan Bresnahan
Versions
- 0.1.23 August 03, 2025 (13.5 KB)
- 0.1.22 August 03, 2025 (13.5 KB)
- 0.1.21 August 01, 2025 (13.5 KB)
- 0.1.20 August 01, 2025 x86_64-linux (24.6 MB)
- 0.1.20 August 01, 2025 arm64-darwin (23.9 MB)
- 0.1.8 July 31, 2025 x86_64-darwin (7.98 MB)
