How to Gate Pull Requests on LLM Evals in CI

Changing one line in a support agent’s system prompt can ship an agent that tells customers the refund window is 30 days when your policy says 14. Nothing in a normal CI pipeline checks what the model says, so the build passes and the first person to see the wrong answer is a customer. Gating a pull request on a fixed eval set works the same way as gating on a failing unit test. You keep test cases in the repository, run them when a prompt changes, and block the merge when too many fail. In this guide, you write an eval set for a support agent and score it with a script that calls OpenRouter. You measure how much the results move between runs with nothing changed, then wire the script into GitHub Actions as a required status check. A fixed eval set is a list of test cases committed to your repository. It changes only through a reviewed pull request. Filter at the job level rather than the workflow level. GitHub reports a job skipped by an if condition as a passing check, while a workflow skipped by a path filter leaves a required check pending and blocks…