riktoken 0.0.1
A pure Ruby implementation of OpenAI's tiktoken library for BPE tokenization
Gemfile:
=
instalar:
=
Versões:
- 0.0.1 June 04, 2025 (14,5 KB)
Runtime Dependencies (1):
base64
>= 0
Donos:
Pushed by:
Autores:
-
moznion
SHA 256 checksum:
=