Topic: Part 8. Sources and Schema Manifest
Difficulty level: Medium
Estimated study time: 2.5–3 hours
Prerequisites: Basic understanding of SQL and CSV files
Familiarity with ETL/ELT concepts and layered data architecture
Understanding the difference between the raw layer and the mart layer
Introductory knowledge of dbt and its role in data transformation
Basic understanding of PII (Personally Identifiable Information) and data governance
Learning objectives: Explain the difference between observation mode and design mode when working with data sources
Compose a Schema Manifest for at least two sources using a given template
Identify candidate keys, PII candidates, and null values in a dataset without interpreting business meaning
Formulate at least five open questions to the source owner as a safeguard against hidden assumptions
Argue why the manifest and contract should exist as separate documents
Overview: Source profiling is the first and fundamentally important stage of an agent's work with raw data. Unlike the design stage, where the agent creates models and contracts, profiling operates strictly in observation mode: the agent records what actually arrived in the raw layer, without trying to assign business meaning to it. The main tool of this stage is the Schema Manifest. This is a document that describes the observed fields, their types, null values, candidate keys, PII candidates, and freshness hints. An important part of the manifest is the open questions section — it protects against premature interpretation. The manifest describes what is observed, while the contract describes what is promised. If you mix these layers, the agent will begin to "clean up" the source before a human has approved the policy, which leads to the loss of important signals about data quality and to errors in reporting. The goal of this section is to learn restraint when working with raw data and to understand that open questions are not a weakness, but protection against hidden assumptions.
Key concepts: Source profiling: An observation mode for raw data, in which the agent reads file headers, the first rows, and computes simple statistics. The main rule is not to decide business meaning on your own and not to write dbt models at this stage.
Schema manifest: A document that describes what is observed in the source. It consists of sections: observed fields, candidate keys, PII candidates, quality notes, and open questions. This is a data inspection protocol, not a data transformation design.
Observation mode vs design mode: In observation mode, the agent only describes what it has seen. In design mode, it creates models, contracts, and makes architectural decisions. Mixing these modes leads to premature optimization and loss of context.
Manifest vs contract: The manifest records what is observed (what actually arrived in the raw layer). The contract records what is promised (what the source guarantees and what is permitted for use in the mart). If the source contains a pii_email field, the manifest must show it, while the contract may prohibit it from reaching the mart.
Candidate key: A field or combination of fields that, according to the agent's observations, look like stable identifiers. Until approved by a human, this is only a candidate, not a key.
PII candidates: Fields that, by their name or content, resemble personal data. Until approved by the source owner and a formal processing policy, this is only a suspicion, not a fact.
Data freshness: Hints about how up-to-date the data is: extraction date, presence of timestamps, update frequency. This is an observable indicator, not a contractual guarantee.
Open questions: A section of the manifest where all the ambiguities and oddities of the source that cannot be resolved without the owner are recorded. This protects against an assumption becoming a fact.
Agent restraint: A principled commitment not to interpret data until a human has approved it. The agent sees risk_flag and should not immediately assign it business meaning — only record the field's existence.
Crime scene inspection protocol: A metaphor for a quality manifest: just as a forensic investigator at a crime scene describes what they found without drawing conclusions about the perpetrator, the agent describes data without drawing conclusions about its meaning.
Raw layer protection: The manifest protects raw data from premature "cleanup". If there is an oddity, it must first be recorded and a question asked, not corrected, because a null date can mean three different things: an error, an acceptable way of encoding, or data loss.
Observed type: The data type that the agent sees in the source (for example, string, number, date as a string). Not to be confused with the contract type — the one that the source promises or that the mart requires.
Source grain: The level of detail of the data (one row = one transaction, one customer, etc.). The agent must not decide the grain without human approval, even if the answer seems obvious.
Important dates: Course context: Part 8 is the stage after receiving raw data and before designing models in the data engineering course using dbt and LLM agents
Order of work: First the Schema Manifest (observation), then the contract (design), then the models (implementation)
Practice exercises: Name: Composing a manifest for card_transactions
Problem: Open specs/source-manifests/card_transactions.md. Study the observed fields, candidate keys, PII candidates, quality notes, and open questions. Based on this template, compose your own draft manifest for a fictional source raw/orders.csv with the following columns: order_id, customer_id, order_date, total_amount, status, risk_flag, promo_code. Specify the observed types, mark null values, suggest a candidate key and PII candidates, and formulate at least four open questions.
Solution: Step 1. Create the file specs/source-manifests/orders.md. Step 2. Describe the observed fields with their types, for example: order_id (int64), customer_id (int64), order_date (string in YYYY-MM-DD format), total_amount (float64), status (string), risk_flag (bool or string), promo_code (string with a large number of nulls). Step 3. The candidate key is order_id (looks like a stable identifier). Step 4. PII candidates — customer_id (synthetic or real?). Step 5. Quality notes — promo_code is empty in most rows. Step 6. Open questions: what does an empty promo_code mean? Is risk_flag a source flag or the result of an antifraud system? What does status encode — final state or current state? Freshness — is there a loaded_at field?
Complexity: intermediate
Name: Analysis of an open question about revoked_at
Problem: In the source open_api_consents.csv there is a revoked_at field that is often null. Formulate what could break if the agent resolves this question on its own, without contacting the source owner. Describe three possible interpretations of the null value and the consequences of each.
Solution: Step 1. Formulate three hypotheses about an empty revoked_at: (a) the consent is active and has never been revoked, (b) the consent was revoked, but the date was not recorded due to a bug in the source, (c) the field is used in a special way of encoding. Step 2. Describe the consequences of a mistaken interpretation: if the agent decides that an empty revoked_at = active consent, while in reality it is a bug, then rows with outdated consents will end up in the mart, which will violate GDPR/152-FZ requirements. Step 3. Draw a conclusion: an open question in the manifest should explicitly point out this ambiguity and require confirmation from the owner.
Complexity: intermediate
Name: Separation of manifest and contract
Problem: Given a source that contains the fields: pii_email, pii_phone, customer_segment, lifetime_value, registration_date. Compose two separate documents: a Schema Manifest (observed) and a contract draft (promised). Show that the manifest must record all fields, including pii_email and pii_phone, while the contract must explicitly indicate which of them are permitted in the mart.
Solution: Step 1. The Schema Manifest includes all fields with types, marking pii_email and pii_phone as PII candidates. Step 2. In the open questions section of the manifest, a question appears: "Is there a policy for masking pii_email and pii_phone before they reach the mart?". Step 3. The contract contains explicit permissions and prohibitions: for example, pii_email — prohibited in the mart, allowed only in a protected layer with masking; customer_segment — allowed in the mart; lifetime_value — allowed in the mart. Step 4. Draw a conclusion: if the document were one, the agent could "forget" about pii_email, or, conversely, start aggressively cleaning the source before the policy was approved.
Complexity: intermediate
Name: Searching for hidden assumptions in your own manifest
Problem: Take your draft manifest from exercise 1. Read it carefully and find at least three places where you, without noticing, assigned business meaning to a field before human approval. Rewrite these places in a neutral observation form.
Solution: Step 1. A typical hidden assumption is calling status the "order status" instead of the "status field". Step 2. A second assumption is deciding that risk_flag "means high risk", although this is only the field's name. Step 3. A third is writing "customer_id is the customer", although customer_id could be a session, device, or account identifier. Step 4. Rewrite in a neutral form: "the status field takes values from the set {new, paid, cancelled, refunded}", "the risk_flag field has type bool, observed in 12% of rows", "the customer_id field is a numeric identifier, meaning not confirmed". Step 5. Add these ambiguities to the open questions section.
Complexity: intermediate
Name: Composing a Qwen query for profiling
Problem: Formulate a prompt for Qwen (or another LLM agent) that correctly sets the task of profiling a new source raw/loan_applications.csv. The prompt must explicitly prohibit the agent from writing dbt models and deciding the grain without approval.
Solution: Step 1. Start with an explicit statement of the task: "Read the headers of raw/loan_applications.csv and the first 100 rows". Step 2. List what needs to be done: "Compose a Schema Manifest: fields, observed types, candidate keys, PII candidates, freshness hints, and open questions". Step 3. Explicitly prohibit unnecessary actions: "Do not write dbt models. Do not decide the grain without approval. Do not replace null values. Do not interpret the business meaning of fields". Step 4. Specify the output format: "The result is a markdown document with sections: observed fields, candidate keys, PII candidates, quality notes, open questions". Step 5. Make sure the prompt does not contain phrases that provoke interpretation, for example "describe how this data is used in the business".
Complexity: advanced
Case studies: Name: Case: GDPR audit and premature cleanup of a consent field
Scenario: A European fintech startup was preparing a mart for marketing analytics. The agent received a source with the fields customer_id, consent_marketing, consent_at, revoked_at. The team asked the agent to "clean the data and prepare a showcase of active newsletter subscriptions".
Challenge: The agent saw that revoked_at was often null and decided that this meant "consent is active". Without contacting the data owner, it built a model that interpreted an empty revoked_at as active consent. Six months later, a GDPR audit arrived, and it turned out that some of the "active" subscriptions had actually been revoked, but due to a bug in the upstream system, revoked_at was not being recorded. The company received a fine and was required to notify all affected customers.
Solution: The team reworked the process: they introduced a mandatory Schema Manifest step, in which the open question "what does an empty revoked_at mean?" was explicitly recorded. Before building the mart, the agent was required to obtain confirmation from the source owner. The manifest explicitly stated that revoked_at was a candidate for encoding active consent, but not a confirmed fact.
Result: After the process change, such bugs began to be detected before they reached reporting. The legal department gained the ability to approve interpretations before release. There were no more fines, and the process of onboarding new sources began to take 2–3 days longer, but it reduced the number of incidents by 80%.
Lessons learned: Premature interpretation of data in code leads to legal and reputational risks that are not visible at the development stage
The Schema Manifest is not a formality, but a legal artifact that protects the team and shows that the interpretation was approved by a human
Open questions are documented caution, not a weakness of the agent's work
Separating the manifest and contract avoids situations where "cleaned" data hides real source problems
Related concepts: Manifest vs contract
Open questions
Agent restraint
PII candidates
Raw layer protection
Name: Case: a suspicious risk_flag field in an antifraud system
Scenario: A risk analytics team received a transaction dump with a risk_flag field. Two junior analysts interpreted this field differently: one believed it was the "final decision of the antifraud system", the other believed it was an "input parameter for a rule". Both began building reports without clarification.
Challenge: Three months later, it turned out that the analysts' reports contradicted each other. For one, the share of "risky" transactions was 4%, for the other — 11%. The reason was different interpretations of the same field. Without a Schema Manifest, neither analyst had a documented basis for their interpretation, and the source owner could not quickly resolve the dispute.
Solution: The team introduced the practice of a mandatory Schema Manifest for every new source. In the manifest for transactions, risk_flag was recorded as "a field of type bool, occurs in 100% of rows, a candidate for business meaning — a risk flag, but the exact interpretation requires confirmation from the owner". An open question was posed to the source owner, and a week later the answer came: "risk_flag is the result of the rule engine before the ML model is applied, it cannot be used as an independent risk feature".
Result: After the practice was introduced, disputes about field interpretation decreased. The onboarding time for a new analyst on the team was reduced from two weeks to three days, because the Schema Manifest served as a "map" of the source. The team also discovered two more fields with ambiguous interpretation that had previously simply been used "as is".
Lessons learned: A field's name does not define its meaning — even obvious names like risk_flag can have different interpretations
The Schema Manifest captures ambiguities and makes them visible to all team members
An open question without an answer is better than an assumption that became a fact
Documenting raw data is an investment that pays off in any dispute about interpretation
Related concepts: Agent restraint
Open questions
Crime scene inspection protocol
Observed type
Name: Case: source migration and questions saved from the manifest
Scenario: A large retailer migrated its loyalty system from one CRM to another. The source customer_profiles.csv had 47 fields, many of which were unclear to the new team. The previous team left a Schema Manifest with 23 open questions.
Challenge: The new team could have ignored these questions and started building the mart "as best they could". But they saw that many of the open questions concerned critical fields: is_vip, churn_score, lifetime_segment. Without answers, they could not guarantee that the reporting would match the old one.
Solution: The new team contacted the source owners for each of the 23 open questions. 14 answers turned out to be critical for reporting. For example, it turned out that churn_score was calculated using an outdated formula, and using it in the mart without correction would have led to an 18% discrepancy with historical dashboards.
Result: The Schema Manifest with open questions saved the team at least 2 months of investigation and prevented the publication of incorrect reporting. After the migration, the team made the practice mandatory: every source must have an up-to-date Schema Manifest with an open questions section, even if at the time of creating the manifest "everything is clear".
Lessons learned: Open questions have long-term value — they help not only the current team, but also those who come after
The Schema Manifest is an artifact for knowledge transfer between teams
Even if it seems that "everything is obvious", documenting ambiguities safeguards against errors in the future
Migrations and changes in sources are moments when the Schema Manifest is especially valuable
Related concepts: Open questions
Raw layer protection
Manifest vs contract
Source profiling
Study tips: Practice restraint: every time you want to write "this field means X", rephrase it as "this field is called X, its meaning requires confirmation".
Keep a personal list of "dangerous assumptions" — fields in which you are confident, but the confidence is not confirmed by the source owner.
Formulate open questions as questions to the owner, not as rhetorical or self-evident statements. For example, not "probably active consent", but "empty revoked_at: what does it mean — active consent, a source bug, or special encoding?".
Do not try to write a dbt model immediately after the manifest. First show the manifest to a human and get confirmation of the interpretations.
Re-read your manifest after a day — hidden interpretations are often discovered that are not visible right away.
Use a unified manifest template for all sources — this simplifies review and comparison of sources with each other.
If a field looks "too obvious" (for example, customer_id), this is not a reason to skip it in the manifest. Obviousness is a trap.
Separate what is observed (manifest) and what is promised (contract) physically — in different files, with different owners and different update cadences.
When reviewing someone else's manifest, pay attention to missing open questions. If there are none, this is a reason to ask questions yourself.
Remember that the manifest is read not only by engineers, but also by analysts, lawyers, and managers. Write so that the document is understandable without deep technical context.
Additional resources: Specs/source-manifests/card transactions.md: A sample manifest for a transaction source, available in the course repository
Specs/source-manifests/open api consents.md: A sample manifest for a consents source with an example of open questions
Dbt documentation: sources: https://docs.getdbt.com/docs/build/sources — official documentation on sources in dbt, which is logically related to the idea of a manifest
Dbt-utils: source freshness checks: https://github.com/dbt-labs/dbt-utils — tools for checking source freshness
Gdpr official text: https://gdpr-info.eu — regulation that explains why PII and consents require special attention in the manifest
Data quality fundamentals (o'reilly): A book that reveals the principles of data profiling and working with quality
Fundamentals of data engineering (joe reis, matt housley): A book with a detailed description of layered architecture and the place of the manifest in it
Course on data observability (monte carlo, bigeye): A series of articles and videos on data observability, which is conceptually close to the idea of a Schema Manifest
Summary: Part 8 of the course is devoted to the Schema Manifest — a key document that records the observed state of raw data. The main principle of this stage is restraint: the agent describes what it sees and does not decide business meaning until a human has approved it. The manifest includes observed fields, candidate keys, PII candidates, quality notes, and, most importantly, open questions. Open questions are not a weakness, but protection against hidden assumptions: an empty field can mean three different things, and only the source owner can confirm the correct interpretation. The manifest is fundamentally separated from the contract: the manifest describes what is observed, the contract describes what is promised. Mixing these layers leads to premature cleanup of the source and loss of data quality signals. As a result of studying this part, you will learn to compose a Schema Manifest for real sources, formulate open questions, separate observation from interpretation, and justify the architectural separation of the manifest and contract.