Info Desk
Which claims are actually supported by the sources? Info Desk puts a finding beside its original passage, keeps disagreements visible and lets a person review the result.
Python · FastAPI · SQLite · limited tools
The problem
A fluent summary can make different sources sound as though they agree. A government announcement, a license list and a news report do not necessarily support the same claim. Reading the summary alone can hide those differences.
My contribution
I built a comparison table that links findings to exact passages, plus checks for cited numbers and a database approval step in the local app. This page includes the recorded case below. Fresh source collection and AI answers run in a separate local service; they are not running inside this public example.
From question to something you can check
| Question | What I built | Why it helps |
|---|---|---|
| Do the sources agree? | Compare each claim with the actual passages. | Different wording and missing information remain visible instead of becoming one confident summary. |
| Can I check an AI draft? | Attach references and reject unknown citations or unsupported numbers. | A traceable quote is a starting point for review, not proof that the interpretation is correct. |
| Who decides what gets saved? | Keep drafts separate from approved notes in the local database. | Model output cannot approve its own write. The public recording does not perform database writes. |
Try the recorded case
Select sources, compare a claim, then open its supporting passage. This is a dated recording, not live research or a live model response.
The example preserves the recorded outcomes. Fresh research and database approval require the local service described below.
How I built it: tools, database, evaluation and limitations
What the evaluation found
On 17 development questions over recorded excerpts, expected evidence ranked first for 10/13 answerable questions. All three cross-language questions missed that first position; 3/4 unanswerable questions still returned passages. These are retrieval checks, not answer-accuracy scores or independent test results.
Skills this work used
| Skill | Where it was used |
|---|---|
| Python | Claims table, named quantities, OFAC license parse, negation so “does not name” is not a claim. |
| Limited tools | fetch_source, lookup_license, search_prior_notes. No free-form write. |
| SQLite | Drafts stay pending. A note row exists only after a human approves publish_draft. |
| Eval harness | Six Inspect-style cases on the same three recordings. CI reads SQLite, not a model saying done. |
| Optional Ollama | Local LLM may draft JSON. Python still checks numbers. Off in CI. |
Try it step by step
- Open the desk. You should see three public URLs: OFAC, White House, AP — not toy pages.
- Read the claims table. The OFAC recording does not name NABEP / 17 fields; White House and AP state them. $200bn royalties sit on the fact sheet only.
- Check the ranking row: “second-largest private producer” vs “second largest operator, behind Chevron.” The desk must not pick one.
- Read the OFAC license table (46D, 50C, 52B). A license list is not the fact-sheet deal.
- If you run it locally, reject the draft and check SQLite: approved writes stay 0. Approve is refused unless the action is publish_draft.
What depends on what
| Component | Depends on | Responsibility |
|---|---|---|
| Desk loop | Three tools plus extractors | Fetch, license lookup, prior notes. No other writes. |
| Claims table | The three dated recordings | Stated / absent / denied / attributed. Not a model summary. |
| Heuristic interpreter | Extracted quantities, ranking, scope, attribution | CI path. Same actions as the regex baseline. |
| Ollama (optional) | Local generate API | May propose JSON. Dropped if the host is down. |
| Store | SQLite | approve() is the only insert into notes. |
How the data was organised
The public GitHub Pages view is the recorded claims table (case.json). The repository also has a local FastAPI live workspace with versioned SQLite captures. It is not publicly hosted, so the public card remains an inspectable recorded case rather than a live service.
| Record | Depends on | Responsibility |
|---|---|---|
| recordings/ | Public URL plus fetch date | OFAC and White House as text. AP as quotes only, not a republication. |
| sources / fetches | Allowlisted recording id | Each fetch stores status and time, including a 429 before retry. |
| drafts | case_id plus body hash | Pending proposal. Duplicate body hash does not insert a second row. |
| notes | draft_id, unique | Created only by approve() on action publish_draft. |
| approvals | draft_id plus decision | Human approve or reject. Reject writes zero notes. |
Implementation, step by step
- Pin the tools. Anything else, including the model, cannot write.
- Compare the three recordings in Python: licenses, named quantities, ranking phrases, and negated sentences.
- Score the SQLite snapshot in the harness: notes, approved writes, fetch statuses.
- Label every page PUBLIC_RECORDING. Do not invent barrel figures.
- Run pytest in CI. Do not call a hosted LLM there.
Run the public code locally
python -m pip install -e ".[dev]"
python -m pytest
python -m infodesk.harness
python -m infodesk.appUse the repository README for environment setup and supported versions.
Limits and unfinished work
The public page is a recorded case. The live workspace runs locally and requires an appropriate hosted backend, access controls, persistent storage and an inference configuration before it can be offered as a public service. CI uses dated recordings. A fact sheet is not signed contracts, and AP quotes are not the full article.