TokenKit provides lightweight, Unicode-aware word-level tokenization with pattern preservation, backed by Rust for performance.
Required Ruby Version
>= 3.1, < 3.5.dev
Authors
Chris Petersen
Versions
- 0.1.0.pre.2 June 20, 2026 arm64-darwin (3.62 MB)
- 0.1.0.pre.2 June 20, 2026 aarch64-linux (3.88 MB)
- 0.1.0.pre.2 June 20, 2026 x86_64-linux (4.05 MB)
- 0.1.0.pre.2 June 20, 2026 (885 KB)
- 0.1.0.pre.1 September 29, 2025 (38.5 KB)
Requirements
Rust >= 1.85
