Case study · 2026 · AI/ML · Full-Stack

AnswerGate — Regression Tests for AI Assistants

An open-source CI check that catches the moment an AI assistant's answers get worse after a code change, and fails the pull request with the exact questions that got worse.

Watch · 1:35 · sound on

36/42

Injected faults caught

False alarms, 1,070 questions

19/44

Questions flagged by one check

00 How it works

Pull request · “Smaller chunks for more precise retrieval”

examples/fastapi_qdrant/rag.py

−CHUNK_SIZE = 1000 # characters per chunk

+CHUNK_SIZE = 250 # characters per chunk

One line. Looks harmless in review.

How it works

A pull request changes one line in an AI assistant

01 The problem

Teams tune AI assistants all the time — chunk size, prompts, models. Each change can quietly make answers worse, and nothing notices: the code review looks fine and the unit tests still pass. Teams find out from users.

02 Approach

01
Baseline diff: every run is compared with the last good one, question by question. A drop beyond the allowed margin fails the gate, and every change comes with a 95% confidence interval; the gate can be set to fail only on significant drops.
02
Private by default: answers are graded by a local LLM through Ollama, calibrated on labelled cases before it's trusted (granite4.1:8b scored 32/32).
03
Evidence-checked grading: every 'supported' verdict must quote the retrieved text, and code verifies the quote, so a judge can't wave a hallucination through.
04
Built-in guardrails for PII leaks, prompt injection hidden in documents and off-topic requests, plus sharding, resume and a judge cache for large test sets.
05
Change tracking: the assistant reports its settings, and AnswerGate names what changed since the last good run. With repeats, it asks each question several times and scores how consistent the answers are.

03 Architecture

01Pull requestGitHub ActionAnswerGateyour AI assistant (HTTP or Python)

02Answers + retrieved passagesretrieval metrics + local LLM judgeevidence check

03Scoresbaseline diff + 95% confidencepass / fail gate

04VerdictPR comment + HTML report

04 Results

Caught 36 of 42 faults injected into six industry test pipelines across 1,070 edge-case questions, with zero false alarms. The other six are hallucinations that need a judge to catch.
On a real FastAPI + Qdrant assistant, a one-line chunk-size change kept search at a 0.975 hit rate but dropped correctness from 0.925 to 0.856 and relevance from 0.988 to 0.900. AnswerGate failed the check and listed the 19 questions that got worse.
Testing on real documents exposed three bugs in AnswerGate's own checks — fixed, which raised measured faithfulness from 0.881 to 0.935.
Raising the model's temperature from 0 to 1 left correctness, faithfulness and relevance within noise, but answer consistency fell from 0.848 to 0.509. AnswerGate failed the check and named the setting: temperature 0 → 1.
Released on PyPI as answergate (now 0.3.0) through GitHub trusted publishing, with 221 automated tests running in CI on Linux and Windows.

Next case study

Exsy — Family Money Book

02 Approach

01
Baseline diff: every run is compared with the last good one, question by question. A drop beyond the allowed margin fails the gate, and every change comes with a 95% confidence interval; the gate can be set to fail only on significant drops.
02
Private by default: answers are graded by a local LLM through Ollama, calibrated on labelled cases before it's trusted (granite4.1:8b scored 32/32).
03
Evidence-checked grading: every 'supported' verdict must quote the retrieved text, and code verifies the quote, so a judge can't wave a hallucination through.
04
Built-in guardrails for PII leaks, prompt injection hidden in documents and off-topic requests, plus sharding, resume and a judge cache for large test sets.
05
Change tracking: the assistant reports its settings, and AnswerGate names what changed since the last good run. With repeats, it asks each question several times and scores how consistent the answers are.

04 Results

Caught 36 of 42 faults injected into six industry test pipelines across 1,070 edge-case questions, with zero false alarms. The other six are hallucinations that need a judge to catch.
On a real FastAPI + Qdrant assistant, a one-line chunk-size change kept search at a 0.975 hit rate but dropped correctness from 0.925 to 0.856 and relevance from 0.988 to 0.900. AnswerGate failed the check and listed the 19 questions that got worse.
Testing on real documents exposed three bugs in AnswerGate's own checks — fixed, which raised measured faithfulness from 0.881 to 0.935.
Raising the model's temperature from 0 to 1 left correctness, faithfulness and relevance within noise, but answer consistency fell from 0.848 to 0.509. AnswerGate failed the check and named the setting: temperature 0 → 1.
Released on PyPI as answergate (now 0.3.0) through GitHub trusted publishing, with 221 automated tests running in CI on Linux and Windows.