Human Review Packet

An agent asking a human to validate rows is asking for the most expensive attention in the pipeline. The reviewer is slow, finite, easily discouraged, and the only thing standing between a confident extraction and a database full of plausible fiction.

The failure is almost never that the human refuses. It is that the artifact makes reviewing cost nearly as much as doing the work again — so it gets postponed, and the postponement is invisible because nothing errors.

The rule this skill exists to enforce: never ask a human to check a claim without putting the evidence for that claim next to it. Not a list of pages the agent read. Not "source: their website". The specific URL, and a local snapshot of the text that was actually read at extraction time.

Onboarding — start here

1. What this skill does

Turns "can you check these rows?" into a packet the reviewer can work through without opening a browser, running a search, or asking you what a column means. It covers what goes in the artifact, how to sample, how to avoid anchoring the reviewer to the model's answer, and what to do with the marks when they come back.

It applies to any human-validation step: ICP extractions, headline classifications, magnet detection, enrichment output, employer verification, scraped-field spot checks.

2. What it can't do — say this before promising anything

Reach for gsheets-api when the packet should land in a spreadsheet with enforced dropdowns rather than a CSV.

3. Setup

Reads whatever the extraction step already wrote, so the real requirement is upstream: the fetch or scrape step must persist, per record, the source URL and the raw text it read. If that is missing, that is the bug — fix it there.

For delivery it needs token.json (gitignored) for the Sheets API. Never hand over a CSV and ask someone to import it — see "Delivery" below.

python scripts/push_to_sheets.py --csv data/packet.csv --title "Review" \
    --checkbox checked --checkbox supported --out data/packet-sheet.json

Verify the data can support a packet at all, before promising one:

python -c "
import json,sys
recs=json.load(open(sys.argv[1]))
n=len(recs)
u=sum(1 for r in recs if r.get('source_urls'))
print(f'{n} records, {u} with source_urls  ->', 'OK' if u==n else 'PROVENANCE GAP')
" data/icp-batch3.json

4. How to invoke

It should fire on its own from the description. Ask for it directly with "build the review packet" or "use human-review-packet".

5. Cost

Free. No API calls, no credits. The cost it manages is the reviewer's attention, which is the scarcest input in the pipeline and the only one with no rate card.

6. The rest of the skill

• references/failure-modes.md — eight review failures observed in this repo, each with what it looked like at the time and the guard that prevents it
• references/packet-spec.md — the required columns, the snapshot convention, and the blind/reviewed split
• references/sampling.md — fixed draws, what a spot check can and cannot claim

The procedure

Step 1 — Check provenance exists before promising a packet

Every claim needs three things: the value, the verbatim span that supports it, and the URL that span is on. If any is missing, stop and say so. A packet built on two of three teaches the reviewer to trust unverifiable rows.

A populated value beside an empty evidence span is not a minor gap. It is the model asserting something the source never said, and in the artifact it looks exactly like a supported value. Sort those to the top and mark them explicitly — they are the highest-yield rows in the review.

Step 2 — Resolve each span to the exact page it is on

Not the list of pages read. If a record cites three URLs, "it came from one of these" hands the search back to the reviewer, which is the thing that makes review feel like redoing the work.

Spans are verbatim by construction, so the page is findable:

• Collapse whitespace on both sides before matching — extracted spans and cached markdown differ in whitespace often enough to break a literal find.
• Split multi-span values on their joiner (commonly " / ") and match the longest single span. The joined string was never on any page, so searching for it finds nothing and reports a false "unlocatable".
• A span that genuinely matches no cached page is a real finding. Surface it as such rather than leaving the cell blank.

Step 3 — Ship the snapshot, not just the link

Include a path to the locally cached text that was read at extraction time.

The live page is not the evidence. It changes, and when it does the reviewer marks correct extractions as misses and loses confidence in the whole set. The snapshot is what the claim was actually made against.

Convention already used in this repo: data/icp-pages/<sha1(url)[:12]>.md. Keep the keying stable so any packet can resolve a URL to its snapshot.

Step 4 — Put the columns the reviewer types into first

mark and note are column A and B. Everything else is context.

This sounds cosmetic. It is not — the review CSV this replaced had the mark column past 60 columns of context, and the check was deferred four sessions.

Step 5 — Constrain the answer space

Free text where an enum was needed produces answers that cannot be joined back to anything. Give the reviewer the closed vocabulary, in the packet, and enforce it with dropdowns when the packet is a sheet.

Watch for the reviewer answering a different question than the column asks — axis-1 reasoning appearing in an axis-2 column is the tell. Say in the header exactly what is being judged and what is explicitly not.

Step 6 — Decide blind vs reviewed, and never mix them

• Blind — no model answer shown. Measures the human against the source.
• Reviewed — model answer shown, human marks hit/pass. Faster, but anchored.

Both are legitimate; they measure different things. Mixing them in one file produces a number belonging to neither. If a person is being re-reviewed on rows they labelled before, their prior answers stay hidden.

Step 7 — Deliver a link, never a file to import

A CSV handed to a human is not a delivered packet. Importing it into Sheets needs an explicit File > Import > Replace current sheet; a plain Drive upload lands as a blob that will not open as a spreadsheet at all. Both failures are silent, and the reviewer concludes the data "didn't show up" — which is the point at which the review stops.

Push it through the Sheets API and hand back a URL that opens into the work:

python scripts/push_to_sheets.py --csv data/packet.csv \
    --title "ICP hand-check 11" --checkbox checked --checkbox supported \
    --dropdown verdict=hit,miss --out data/packet-sheet.json

That script applies the things this skill requires and a CSV cannot carry: checkboxes on answer columns, strict dropdowns on enum columns (strict, not warn — a warning still lets an out-of-vocabulary answer through), a frozen header, a wrapped quote column, and column widths that keep the answer cells on screen.

Answer columns come in pairs. One checkbox cannot distinguish "this is wrong" from "I haven't reached this row yet", and those two produce very different loads. Use checked (I have looked at this) alongside supported (the evidence holds). A dropdown is the alternative when there are more than two outcomes.

Persist the spreadsheet id and url next to the CSV so the next revision can update that sheet rather than creating a second one — see versioned-artifacts for what happens when it doesn't.

Step 8 — Explain the task in the conversation. The sheet holds rows only.

The instructions go where the reviewer already is, which is the chat — not in the spreadsheet. The spreadsheet is the place they score; it is not a document. Splitting attention across both means one of them gets skipped, and putting prose in a sheet makes the sheet feel like reading rather than ticking.

So: write the explanation as a message, and hand over a link to a sheet that contains data and nothing else. No START HERE tab, no instruction rows above the header, no notes column pre-filled with guidance.

(push_to_sheets.py --readme exists for the case where the reviewer is not in the conversation — an outside labeller, someone picking it up next week. That is the exception. Default to the chat.)

The explanation in chat must contain, in this order:

1. The one question, stated in a single sentence, in the reviewer's language — not the schema's. "Does that sentence actually prove that claim?" beats "verify enum-quote correspondence".
2. What each answer column means, including why there are two boxes rather than one.
3. Two worked examples, from the reviewer's own data — one obvious pass, one real failure, both copied from actual rows. Invented examples are ignored; examples the reviewer recognises are read. The failure example should be the characteristic mistake, because it teaches the pattern rather than the rule.
4. What they can skip, with a count. Rows already decided (unquoted values, pre-filled notes) should be named and dismissed — "14 of these are automatic, that leaves 35 to think about" turns an intimidating grid into a short job.
5. How to check the source themselves, and when to use the snapshot instead of the live URL.
6. What happens with the marks, and what a clustered set of misses means.
7. A time estimate.

Write it at the reading level of someone who did not build the pipeline. No field names in the prose where a plain phrase will do, no reference to the schema, no "simply" or "just".

Hide the columns the reviewer does not need (--hide). Context they cannot act on still competes for attention. It stays in the sheet; it just is not on screen.

python scripts/push_to_sheets.py --csv data/packet.csv --title "…" \
    --checkbox checked --checkbox supported --link source_url \
    --readme data/packet-readme.txt \
    --hide cached_page --hide confidence

Step 9 — State the return path in the artifact itself

The last line of the packet says what happens with the marks: "tell me the misses and I load all 45". A review with no stated consequence is a review that feels optional.

Step 8 — Report what the check can support

A ten-row gate answers "is this generation safe to load", not "what is the error rate". Report it as the former. If the misses cluster in one field, that field is the finding — fix the extractor for it rather than hand-correcting rows.