Data Curation Pipelines

The gate that separates reviewed data from extracted data

A pipeline without an explicit human approval step has produced nothing reviewed — it extracted and stored. Those are useful operations, and they are not the same thing. The three-week pipeline that opens this book was functional: well-formed output, passing validators, exit code 0 on every run. Structural checks and truth checks are different operations, and a validator built for structure will produce clean failures and miss semantic ones every time.

This book builds the pipeline with the gate in place from the start, treating each stage — crawler, drafter, automated pre-gate, human review interface, publisher — as distinct for distinct reasons, so the review step is fast enough to survive daily use instead of getting abandoned after the first day.

Nine stages, nine working decisions

• the-gate-that-makes-it-curation — What separates a reviewed dataset from an extracted one, and why removing the human decision step turns curation into automated ingestion.
• the-crawler — Seed design as the variable that predicts output quality more than almost anything else, and how to point the crawler at sources worth extracting.
• claude-as-drafter — What makes a system prompt produce output a reviewer can actually evaluate, rather than something merely well-formed.
• the-automated-pre-gate — Length checks, forbidden patterns, and verbatim checks that catch structural defects before a human opens the queue — and why none of it is the real gate.
• calibrating-the-pipeline — Tuning threshold values so the pre-gate catches genuine defects instead of quietly suppressing ambiguous entries to shrink the queue.
• the-review-interface — Why a gate abandoned after the first day isn’t a gate: building a review tool fast enough to survive daily use.
• publishing-the-accepted-dataset — Turning accepted entries into a versioned, queryable artifact with its own metadata and schema.
• the-overnight-runner — Wiring crawler, drafter, pre-gate, and publisher into an unattended batch that still stops for a human sign-off before anything gets promoted.
• reading-the-signal — Rejection rate as a diagnostic: distinguishing drift in the source material from drift in the prompting strategy from drift in Claude’s behavior.

A preview

“Finalized blocks cannot be altered after confirmation.” The quote anchor said the blocks were resistant to reorganization. The source had never used the word “cannot.” The validator had checked structure. It had not checked whether the two claims meant the same thing. They do not.

Anyone whose pipeline has already generalized past its source

Builders of automated content or training-data pipelines who have watched Claude produce something plausible, coherent, and wrong — and are ready to build the gate that should have been there from the start, instead of auditing three weeks of unreviewed output after the fact.

A longer excerpt is available to newsletter subscribers.

Buy Ebook on Amazon