llm-calibration-logprobs
From the author
Reads model uncertainty from token log-probabilities. Covers collecting logprobs and aggregating multi-token labels, confidence tiers and margins for triage, calibration assessment with ECE, Brier scores, and reliability diagrams against human labels, responsible downstream use of confidence, and reproducibility archives. Use when the user asks about classifier confidence by item, logprobs, top-k tokens, calibration, ECE, Brier, reliability diagrams, or routing low-confidence cases to human review. Codebook and validation design go to text-classification.
Install
npx skills add scdenney/open-science-skills --skill llm-calibration-logprobs -a claude-code
Installs to .claude/skills/llm-calibration-logprobs/ · Each command fetches the skill straight from GitHub with the open skills CLI.
No download: this skill’s licence (Custom or unclear) does not let us redistribute its files. The install commands fetch them from GitHub instead.
Want your agent to suggest skills while you type? See how to use skills from your IDE.
Safety checks on SKILL.md
No warnings.
Why 77
aiqbench computes the trust score from public signals. It is not a security audit, so read a skill before you install it.
Similar skills
Closest to this one by what they do and when they are used.
- text-classificationscdenney/open-science-skillsAdvises on setting up LLM-based text classification as a validated measurement design. Covers codebook construction, learning-regime and model choice, prompt construction, validation against human coding with kappa and F1, hybrid human-LLM review, downstream measurement error, and reporting, plus a short resumable batch pattern with a rule-based baseline for repeated free text against a closed codebook. Use when the user wants to classify, code, or label text at scale. Category discovery goes to topic-modeling.Design★ 69
- mllm-evalAperivue/medsci-skillsUse when designing or auditing how an LLM or multimodal LLM is evaluated on a clinical task (report generation, VQA, text extraction). Covers reference standard, clinical-efficacy metrics beyond BLEU/ROUGE, hallucination, contamination and prompt sensitivity. Imaging models are /model-assessment.Science★ 349
- book-toc-labhashgraph-online/awesome-codex-pluginsDesign and pressure-test useful nonfiction book promises, scopes, recommendation loops, and takeaway-first tables of contents before drafting. Use when planning, outlining, scoping, restructuring, validating, or testing a practical book, guide, manual, course-like book, or other reader-outcome-focused long-form nonfiction.Testing★ 1.4K
- llm-testingnaodeng/awesome-qa-skillsUse this skill when you need to test LLM behavior, failure modes, and evidence-based quality boundaries; triggers include LLM 测试 and llm testing.Testing★ 279
- llm-evaluation-designnaodeng/awesome-qa-skillsUse this skill when you need to design LLM evaluation datasets, judges, metrics, and human-review boundaries; triggers include LLM Eval 设计 and llm evaluation design.Design★ 279
- topic-modelingscdenney/open-science-skillsAdvises on setting up and validating topic models for survey and experimental text: discovery vs. measurement, STM/LDA/BERTopic choice, preprocessing, covariate formulas, choosing K across coherence, FREX, held-out likelihood, and residuals, validating topics against human coding, robustness, and reporting. Coding against a fixed codebook goes to text-classification.Engineering★ 69
Ratings & reviews
—
no reviews
Tried this skill? Rate it to help others. Sign in
Discussion
Join the discussion. Sign in