Loading…
Plumbline · Aurelle Skin
How Plumbline was tested against a simpler AI approach, the results, and what data leaves the system.
Step 1 of 6
Paste or upload
Text, DOCX, PDF or an image; images are read with OCR.
Step 2 of 6
Code checks
Rule patterns, term list, contrast, alt text, and confidential facts, matched in code only.
Step 3 of 6
AI checks
Claim extraction and judgement rules (Groq, then Gemini if it fails).
Step 4 of 6
Validators
Every AI finding must cite an ID that exists and a quote that is really there; a second model checks it.
Step 5 of 6
Findings and fixes
Cited findings, approved-wording fixes, and a list of what wasn't checked.
Step 6 of 6
Review and audit
Routed approvals, row-level security, and a hash-chained audit log.
Web app on Vercel (Next.js) · checker service on Render (FastAPI, Tesseract) · Supabase Postgres in Mumbai with row-level security. The checker reads and writes as the signed-in user, so the database decides what it can see.
Dev set written by the builder; held-out human-labelled set arrives in Round 3. 40 posts: 25 with 43 labelled problems in all, 10 clean controls and 5 copies with prompt-injection text added. A finding counts only if it names the right rule or fact and points at the right words. The baseline is one model call per post with the whole brand book and no validators; like Plumbline, it is never shown confidential facts. Run on 23 Oct 2026, 11:43 pm IST with rulebook v1.3; dates are judged as of 2026-10-23.
Share of labelled problems flagged at the right words. Higher is better.
Share of findings on labelled posts that matched a label. Higher is better.
Clean controls that got at least one finding. Lower is better.
Findings whose rule or fact ID exists and whose quote is really in the post. Higher is better.
Recall on the injected copies minus recall on their originals. Zero is best; negative means the injection worked.
From text in to findings out, including model calls. Lower is better.
Every model call is logged with its provider, model, region, purpose, character count and outcome; the log has no field for the text itself. Confidential and embargoed facts are matched in code and never sent to a model, and Plumbline masks names and contact details before any text leaves (the baseline sends posts as they are). These are the calls this evaluation made.
| Provider | Model | Region | Calls | Characters sent | Purpose | Outcome |
|---|---|---|---|---|---|---|
| groq | openai/gpt-oss-20b | us | 68 | 60,474 | extract 47, support 21 | ok 58, http_429 10 |
| groq | qwen/qwen3.8-27b | us | 75 | 1,54,293 | judge 63, rewrite 12 | ok 37, http_429 38 |
| groq | openai/gpt-oss-120b | us | 12 | 27,841 | judge 12 | ok 12 |
| Provider | Model | Region | Calls | Characters sent | Purpose | Outcome |
|---|---|---|---|---|---|---|
| groq | qwen/qwen3.8-27b | us | 75 | 4,48,674 | judge 75 | ok 11, http_429 64 |
| groq | openai/gpt-oss-120b | us | 30 | 1,79,462 | judge 30 | ok 28, http_429 2 |
| gemini | gemini-2.5-flash | global | 1 | 5,957 | judge 1 | ok 1 |
Aurelle, its products, people and figures are fictional. No real customer or personal data is used anywhere in this demo.