JRRToken is a Ruby gem that wraps the tiktoken Rust library, enabling fast and efficient tokenization for OpenAI models. It supports multiple models including o200k_base, cl100k_base, p50k_base, p50k_edit, and r50k_base.
Required Ruby Version
>= 2.7, < 3.5.dev
Required Rubygems Version
>= 3.0.0
Authors
Logan Bresnahan
Versions
- 1.2.1 May 13, 2026 x86_64-linux (24.5 MB)
- 1.2.1 May 13, 2026 arm64-darwin (23.8 MB)
- 1.2.1 May 13, 2026 x86_64-darwin (23.8 MB)
- 1.2.1 May 13, 2026 x86_64-linux-musl (24.5 MB)
- 1.2.1 May 13, 2026 aarch64-linux (24.2 MB)
- 1.0.0 August 01, 2025 aarch64-linux (24.3 MB)
