Part 8. Sources and Schema Manifest

Profiling sources is an observation mode, not a design mode. The agent should read raw file headers, sample rows, and simple statistics, but should not decide business meaning on its own. At this step we collect facts and open questions, so that we do not write a contract on guesses.

What to profile

For each source:

• file or table name;
• list of fields;
• observed type;
• empty values and fields that allow null;
• candidate keys;
• PII candidates;
• freshness or extraction date, if available;
• questions for a human.

Example:

Source: raw/open_api_consents.csv
Fields: consent_id, customer_id, scope, granted_at, revoked_at
Key candidate: consent_id
PII candidates: customer_id — synthetic identifier, no direct PII
Open question: does empty revoked_at mean an active consent?

Why the manifest is separate from the contract

The manifest describes what is observed. The contract describes what is promised. If a source has a field pii_email, the manifest is required to show it. The contract may forbid it from reaching the mart. If you mix these layers, the agent will start "cleaning" the source before a human has approved the policy.

Qwen query

Read raw CSV headers and the first rows.
Compose a Schema Manifest: fields, observed types, candidate keys, PII candidates,
freshness hints, and open questions.
Do not write dbt models. Do not decide grain without approval.

Minimum output

Draft a manifest for at least two sources. In the runnable example, the following files already exist:

specs/source-manifests/card_transactions.md
specs/source-manifests/open_api_consents.md

Format:

# Schema Manifest: card_transactions

## Observed fields
## Key candidates
## PII candidates
## Quality notes
## Open questions

Open questions are not a weakness. They are protection against hidden guesses.

Breakdown for the reader

Profiling a source requires restraint. The agent sees the field risk_flag and wants to immediately assign business meaning to it. Sees an empty revoked_at and wants to decide that the consent is active. Sees stage and may recall a credit methodology. At that point, the most useful thing is to forbid yourself from interpreting. Profiling sources is responsible only for observation: what fields exist, which types look real, which values are empty, which keys appear stable.

The manifest protects the raw layer from premature cleanup. If something is strange in a source, it must first be recorded, not fixed. An empty date could be an error, an acceptable way to encode an active consent, or a consequence of data loss. These options require different decisions. If the agent immediately replaces an empty value with a convenient one, the reviewer will no longer see the question that was supposed to reach a human.

The "open questions" section is therefore not a sign of weak work. On the contrary, it shows that the author is not passing off a guess as a fact. In data engineering, many mistakes happen not because someone did not know SQL, but because a question was closed too early. "Looks like a key" became a key. "Looks like PII" was left without a policy. "Looks like an active consent" ended up in a report.

A good manifest reads like a crime scene protocol: here is what was found, here is what was measured, here is what looks suspicious, here is what cannot be decided without the source owner. Such a document does not replace the contract. It makes the contract more honest, because the contract starts not from the desired picture, but from what actually arrived.

Practice

Open specs/source-manifests/card_transactions.md and specs/source-manifests/open_api_consents.md. For each open question, write what could break if the agent decides it on its own.

Typical mistake

Believing that a field's name fully defines its meaning. risk_flag can be a source flag, an antifraud rule, or a teaching stub. Until a human approves it, it is only a candidate.

Control questions

1. Why should source profiling not write SQL right away?
2. How does an observed type differ from a contract type?
3. What questions should be asked about an empty revoked_at?