Before generation: paste the question and the chunks your retriever returned. Sieve tells you whether the context can answer the question (retrieval_miss, answerability_gap), whether any chunk is unsafe to pass on, and how much of the context is bloat. Everything runs in this browser tab.
After generation: add the answer your model produced. Sieve traces every number, date, quote and name in the answer back to a chunk, flags claims no chunk supports (unsupported_answer), and measures which chunks were never used and what they cost.
Retrieved documents are untrusted input. Paste one chunk to see prompt-injection, hidden-text, exfiltration and secret findings with their exact positions. Sieve reports and locates; whether to pass the chunk on is your call.
Why Sieve exists
Most RAG failures start in retrieval, not in the generator. Before you rewrite the prompt, check whether the retriever brought back what the question needs. Sieve attributes each failure to one side:
- retrieval_miss (retrieval) — the chunks do not contain the question's key terms.
- answerability_gap (retrieval) — the chunks are on topic but lack what the question asks for: a number for "how much", a date for "when", a name for "who is".
- unsupported_answer (generation) — the answer states numbers, dates, names or a negation that no chunk supports.
- bloated_context (cost) — most of the context tokens were never used or were duplicates.
- unsafe_context (security) — a chunk carries a prompt injection, hidden instructions, an exfiltration lure or a high-confidence secret. Always reported first.
Three rules the code follows
- Sieve never retrieves, never generates and never accepts provider credentials. You send the question, the chunks and the answer from your own stack; Sieve only diagnoses them.
- Everything is local and deterministic. Rules, not an LLM judge: the same input always gets the same verdict at no per-call cost.
- Unverified is never shown as clean. A check that failed is
"error", a check that did not apply isnull, and neither is ever rounded to "no problem".
How this page works
This is a static Space. sieve.py — byte-identical to app.py of the Gradio build — runs
under Pyodide in a Web Worker in your browser. Nothing you type is sent anywhere.
Each call has a time limit; if it is exceeded, the worker is restarted and the result is an error, never a
clean audit. The first visit downloads the Python runtime and pandas (about 20 MB).
The in-browser demo keeps no history and has no REST API or batch mode. For the dashboard, CSV batch audits,
corpus hygiene and the HTTP API, run the Gradio build: pip install -r requirements.txt && python app.py.
Links
- Source, Gradio build and tests: github.com/NagaYu/sieve
- Engine and rule inventory: NagaYu/sieve-rag-auditor
- Benchmark with a held-out split: NagaYu/sieve-eval
Sieve checks properties it can verify: key-term coverage, required elements, claim traceability, known threat patterns and duplicate text. A clean audit does not mean the answer is right, and pattern-based threat detection can be evaded and can misfire.