Skip to content

Release v3.0.1
Compare
Choose a tag to compare
@himkt himkt released this 27 Sep 17:13
v3.0.1
145f4e0
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
  • #41 Add whitespace tokenizer
In [1]: from tiny_tokenizer import WordTokenizer
In [2]: tk = WordTokenizer("whitespace")
In [3]: tk.tokenize("γ‚γŸγ— は 猫")
Out[3]: [私, は, 猫]
Assets 2
Loading