tokenizer-ruby 0.2.0

Fast tokenization for Ruby using HuggingFace's Rust-powered tokenizers library. Supports GPT, BERT, LLaMA, Claude, and any HuggingFace tokenizer.

Gemfile:
=

安装:
=

版本列表:

0.2.0 March 11, 2026 (10.5 KB)
0.1.1 March 10, 2026 (10.5 KB)
0.1.0 March 07, 2026 (9.5 KB)

Runtime 依赖 (1):

rb_sys ~> 0.9

显示所有传递性依赖

所有者:

推送者:

作者:

Johannes Dwi Cahyo

SHA 256 校验和:

=