byte-fallback

Here is 1 public repository matching this topic...

High-precision, Byte-Fallback Unigram tokenizer with dual-offset tracking, arithmetic isolation, and multilingual Unicode protection.

nlp rust deep-learning tokenizer help-wanted up-for-grabs hacktoberfest unigram good-first-issue pyo3 subwords byte-fallback

To associate your repository with the byte-fallback topic, visit your repo's landing page and select "manage topics."