sentencepiece
From the author
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
Install
npx skills add Orchestra-Research/AI-Research-SKILLs --skill sentencepiece -a claude-code
Installs to .claude/skills/sentencepiece/ · Each command fetches the skill straight from GitHub with the open skills CLI.
Byte-identical to the scanned commit, with the repository’s licence and a SOURCE.txt.
Want your agent to suggest skills while you type? See how to use skills from your IDE.
Safety checks on SKILL.md
- Uses sudoLow
Why 91
aiqbench computes the trust score from public signals. It is not a security audit, so read a skill before you install it.
Similar skills
Closest to this one by what they do and when they are used.
- huggingface-tokenizersOrchestra-Research/AI-Research-SKILLsFast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.Science★ 13.8K
- sentence-transformersOrchestra-Research/AI-Research-SKILLsFramework for state-of-the-art sentence, text, and image embeddings. Provides 5000+ pre-trained models for semantic similarity, clustering, and retrieval. Supports multilingual, domain-specific, and multimodal models. Use for generating embeddings for RAG, semantic search, or similarity tasks. Best for production embedding generation.Media★ 13.8K
- humanize-chinesebrycewang-stanford/Auto-Empirical-Research-SkillsDetect and humanize AI-generated Chinese text. 20+ rule detection categories plus statistical features (sentence-length CV, short-sentence fraction, comma density, perplexity, GLTR, DivEye) plus scene-aware LR fusion (rule × 0.2 + LR × 0.8) trained on three scenes: general / academic / longform 长文本 (≥1500 字)。Unified CLI: ./humanize {detect,rewrite,academic,style,compare}. 8 style transforms (casual/zhihu/xiaohongshu/wechat/academic/literary/weibo/novel)。 Multi-paragraph rewriting (paragraph length CV、跨段 trigram 重复) plus best-of-N humanize (默认 N=10 取最低 LR)。165 replacement patterns + CiLin 同义词词林 38873 with collision blacklist。 Academic paper AIGC reduction for CNKI/VIP/Wanfang (知网/维普/万方 AIGC 检测降重)。 Pure Python, no dependencies, offline。v5.0.0 — HC3 fused 准确率 95%、学术 hero 100→35 (-65)、 工作汇报 96→13 (-83)、长篇博客 96→41 (-55)。 Use when user says: "去AI味", "降AIGC", "人性化文本", "humanize chinese", "AI检测", "AIGC降重", "去除AI痕迹", "文本改写", "论文降重", "知网检测", "维普检测", "AI写作检测", "让文字更自然", "detect AI text", "humanize text", "reduce AIGC scWriting★ 4.7K
- analyzing-dotnet-performancedotnet/skillsScans .NET code for ~50 performance anti-patterns across async, memory, strings, collections, LINQ, regex, serialization, and I/O with tiered severity classification. Use when analyzing .NET code for optimization opportunities, reviewing hot paths, or auditing allocation-heavy patterns.Engineering★ 5.7K
- recallmaxsickn33/agentic-awesome-skillsFREE — God-tier long-context memory for AI agents. Injects 500K-1M clean tokens, auto-summarizes with tone/intent preservation, compresses 14-turn history into 800 tokens.Writing★ 48.2K
- tao-train-ocrnetNVIDIA/skillsOCRNet for scene text recognition. Recognizes text content from cropped text-region images and supports CTC and attention-based decoders. Use when training, evaluating, exporting, pruning, quantizing, retraining, or running inference for a TAO OCRNet model. Trigger phrases include "train OCRNet", "scene text recognition", "OCR cropped text", "CTC / attention text decoder".Other★ 3.7K
Ratings & reviews
—
no reviews
Tried this skill? Rate it to help others. Sign in
Discussion
Join the discussion. Sign in