Topic: Part 1. Introduction: SDD for Data
Difficulty level: Medium
Estimated study time: 2–3 hours (theory + practice with a local project)
Prerequisites: Basic knowledge of SQL (SELECT, JOIN, GROUP BY, aggregate functions)
Understanding of the concept of data warehouses and marts
Familiarity with the Spec-Driven Development (SDD) methodology from the first volume
Basic understanding of dbt (models, sources, tests)
Confident command-line and Git skills
Learning objectives: Formulate the main difference between SDD Data and a traditional dbt course: the workflow goes from mission and specification to SQL, not the other way around.
Explain why an invisible change in the meaning of data (grain, metric, freshness, PII) is more dangerous than a bug in a web application code.
Define the role of the data product specification as the single source of truth and the place of dbt as the executor of a fixed promise.
List the three basic rules of SDD Data: the specification lives in the repository, the verification fact has a command or a reviewer step, and the author and the reviewer work in different modes.
Run a local smoke test of the educational DataLakeHouse on DuckDB and record your own reading mode for the course.
Overview: The first part of the book introduces the Spec-Driven Development methodology as applied to data platforms. While in the first volume SDD protected web application features from agent errors in routes and UI, here it protects the meaning of data: the grain of a mart, the correctness of metrics, freshness, personal data policy, and compatibility with downstream consumers. The main thesis of the part: dbt remains the primary transformation tool, but it is not the source of meaning. Meaning lives in specifications, and SQL is obliged to execute a fixed promise, not to create it post-factum.
The educational project is a DataLakeHouse of a fictional bank "Bank Northern Bridge". By the end of the book, the reader will build a local dbt project on DuckDB that produces five final marts: mart_customer_360, mart_card_turnover_daily, mart_payment_risk_signals, mart_credit_portfolio_quality, and mart_open_api_consents. All data is synthetic, and the Russian banking context serves as a set of constraints: PII, payments, consents, risk signals, regulatory reporting. This is a reproducible educational architecture, not a legal instruction.
The part sets the rhythm of the entire volume: first the reader records the promise of the data product, then makes it executable through dbt, then proves it with facts and submits it for independent review. At the level of the first chapter, the reader is required to choose a reading mode, run a smoke test, and record in working notes the understanding that the book builds one reproducible data product process, not a "bank in general".
Key concepts: Sdd data: A development cycle in which the agent does not start with implementation. First, the human and the agent fix the intent, boundaries, decisions, and verification facts, then the agent writes code or SQL based on approved files, after which a separate review checks not only the passing of tests, but also the match of the result with the specification. For data, the cycle looks like this: describe the data product → define the contract → implement the dbt model → execute verification facts → evaluate impact on the contract → collect release evidence.
Data product specification: A document that records the promise to the consumer: which grain is promised, which fields are mandatory, which are forbidden, which checks allow change, who confirms drift or breaking changes. It lives in the repository, not in a chat, and serves as the single source of truth.
Data contract: A formal agreement between the supplier and the consumers of a mart about its structure and semantics. Any change to the contract must be explicitly recorded and approved, otherwise it is considered breaking.
Mart grain: A semantic unit describing what a single row in the table represents (for example, "one customer" or "one customer-product"). Changing the grain is a typical example of an invisible breaking change of meaning.
Invisible change of meaning: A defect in which SQL compiles and dbt tests pass, but the downstream consumer receives a different meaning of rows. In a web application, such a defect is often visible immediately; in data, it can get into a report, KPI, or risk model within a week.
Verification fact: A command, SQL query, or manual reviewer step that proves that a dbt model fulfilled the specification, not merely passed tests. Without a verification fact, a green dbt build does not equal a correct mart.
Dbt as the executor of a promise: The positioning of dbt in this book: a transformation tool that makes behavior verifiable, but does not decide what should be true for the business. An analogue of a test framework in a regular application.
Datalakehouse of the "northern bridge" bank: An educational project on DuckDB, reproducible locally. Includes five marts: mart_customer_360, mart_card_turnover_daily, mart_payment_risk_signals, mart_credit_portfolio_quality, mart_open_api_consents. All data is synthetic.
Three basic rules of sdd data: (1) The specification lives in the repository, not in a chat. (2) A verification fact must have a command, SQL, or manual reviewer step. (3) The author and the reviewer must work in different modes, even if it is the same CLI agent after /clear.
Pii and personal data policy: The class of constraints for which the book introduces the specification before SQL. In the DataLakeHouse, personal data, consents, and risk signals are areas where "quick SQL" makes decisions it has no right to make.
Consumer chain: A metaphor explaining the spread of a meaning error: the operations team unloads the source, the data engineer normalizes types, the analyst builds the report, the risk team looks at the signal, the executive reads the aggregate. A local SQL error turns into a systemic meaning error.
Smoke test and local track: The bash smoke_all.sh command from book3/examples/bank-lakehouse. Without dbt, it generates synthetic data and checks the structure; with dbt, it performs a full dbt build. This is the first practical step of the first part.
Contract drift and breaking change: Drift is a gradual shift of the actual behavior of a mart from the specification. A breaking change is a change to the grain, mandatory fields, or metric semantics that violates the contract with consumers. Both require explicit confirmation from the owner.
Important dates: Stage 0: run smoke all.sh: The first practical step: run a local smoke test and verify that the DataLakeHouse is reproducible on the reader's machine.
Stage 1: fix the reading mode: The decision on which track the reader follows: the educational local one on DuckDB or mapping against the production environment as a reference layer.
Stage 2: working notes: Fixing in a markdown file: "First pass: building a local DataLakeHouse of the bank on dbt-duckdb; I will not introduce mapping to the production environment until the practical credit."
Practice exercises: Name: Exercise 1. Running a local smoke test of the DataLakeHouse
Problem: Clone the book repository, go to the book3/examples/bank-lakehouse directory, and run bash smoke_all.sh. First without dbt installed — verify that the script generates synthetic data and checks the structure. Then install the dependencies (dbt-core, dbt-duckdb) and run the full dbt build. Record in your notes the list of created tables and the execution time.
Solution: Step 1: git clone <book_repository> && cd <book_directory>. Step 2: cd book3/examples/bank-lakehouse. Step 3: bash smoke_all.sh — without dbt, the script should generate synthetic parquet/csv into the data/ directory and print a structure report. Step 4: pip install dbt-core dbt-duckdb (or the equivalent for your package manager). Step 5: re-run bash smoke_all.sh — it should perform dbt build and finish green. Step 6: examine the dbt_project.yml file and the models/ directory, find the five final marts. Write down in your notes observations about the build time and the number of rows in each mart table.
Complexity: beginner
Name: Exercise 2. The "invisible grain change" scenario
Problem: Open the specification (or description) of the mart_customer_360 mart and fix its promise in writing: what grain, which mandatory fields, which forbidden fields, who the consumer is. Then mentally add the product_code field to the SELECT for the agent and describe exactly how the contract with the consumer who counted active customers on this mart will be broken. Prepare an argument: why even a green dbt build will not protect the report.
Solution: Step 1: write down the grain — "one row per customer_id". Step 2: list the mandatory fields, for example: customer_id, full_name, birth_date, segment, opened_at. Step 3: list the forbidden fields, for example: card_pan, phone, email, consents. Step 4: describe the consumer — for example, the report "Number of active customers by segments" that executes select segment, count(distinct customer_id) from mart_customer_360 group by segment. Step 5: after adding product_code, the same query without distinct will start counting customer-product pairs, and with distinct it will stop accounting for the fact that a customer has several products of different types. Step 6: argument: dbt tests like not_null and unique on customer_id will continue to pass, because customer_id remains unique (if there are no customer duplicates in the data), but the report will show an inflated number and change its meaning.
Complexity: intermediate
Name: Exercise 3. Designing a verification fact
Problem: For the mart_card_turnover_daily mart, write a verification fact as a SQL query that proves that the sum of turnovers for a day in the mart matches the sum of turnovers in the raw source for the same day, with a tolerance of ±0.01. The query should be saved in the file tests/facts/mart_card_turnover_daily_recon.sql or equivalent. Formulate in text what exactly it proves and at what discrepancy it is considered breaking.
Solution: Step 1: identify the source — for example, stg_card_transactions. Step 2: write the reconciliation: with mart_sum as (select transaction_date, sum(amount) as mart_total from {{ ref('mart_card_turnover_daily') }} group by transaction_date), src_sum as (select transaction_date, sum(amount) as src_total from {{ source('core','card_transactions') }} group by transaction_date) select m.transaction_date, m.mart_total, s.src_total, abs(m.mart_total - s.src_total) as diff from mart_sum m join src_sum s using(transaction_date) where abs(m.mart_total - s.src_total) > 0.01. Step 3: an empty result is success. Step 4: fix in text: "If diff > 0.01 for any day, the fact is considered failed, the release is blocked, the owner is the risk team". Step 5: add this file to sources.yml as data_tests or hook it up in CI.
Complexity: intermediate
Name: Exercise 4. Identifying PII in the mart
Problem: Open the schema (SQL or description) of the mart_open_api_consents mart. List the fields you would classify as PII or sensitive. Formulate which of them are acceptable to give to the consumer, which require masking, which are completely prohibited. Prepare a short list of checks that should accompany this mart.
Solution: Step 1: write out all the fields. Step 2: classify. For example: client_id — quasi-identifier, acceptable in aggregated slices; consent_id — acceptable; consent_type, granted_at, expires_at — acceptable; ip_address, user_agent, device_id — prohibited or masked; full_name, phone, email — prohibited; reason_for_withdrawal — acceptable in aggregates. Step 3: for each class, describe the handling. Step 4: formulate checks: a test for the absence of phone/email, a test for the presence of mandatory consent_id, a mart freshness check (no older than N hours), a check of the revoked vs granted share. Step 5: attach the list of owners: client_id — the data owner is the client service, consent_id — the open APIs product.
Complexity: intermediate
Name: Exercise 5. Review in a different mode
Problem: Imagine that you have written a specification and a dbt model for mart_payment_risk_signals. Now simulate a review "in a different mode": start a CLI agent session with /clear, explicitly forbid yourself from changing the model and ask only to check compliance with the specification. Write down what questions you would ask the reviewer and what evidence you would require.
Solution: Step 1: describe the artifact under review — path to the specification, path to the model, path to the tests. Step 2: fix the prohibition — "do not edit model.sql and the specification, only analyze". Step 3: list of review questions: does the grain match what was promised; are all mandatory fields present and not null; are there forbidden fields in the SELECT; does the verification fact pass; is there a discrepancy with the raw source in sums/counts. Step 4: required evidence — output of the verification fact, list of tests and their status, dbt project diff against main, link to the spec file with version. Step 5: fix the verdict — approved / changes_required / blocked.
Complexity: advanced
Case studies: Name: Case 1. The customer mart "bloated" after adding a product
Scenario: A mid-sized bank, a data platform on dbt + Snowflake. The mart_customer_360 mart has been serving the "Active customers by segments" dashboard and the regulatory reporting report for six months. The analytics team asked the data engineer to add the product_code field to the mart so as not to join the product directory on the BI side.
Challenge: The data engineer with an AI assistant added product_code to the SELECT, ran dbt build, tests for not_null and unique on customer_id passed (because one customer in the data had one product on the snapshot date). The release went into production. A week later, the regulatory report started showing 18% more "active customers" than the real number — rows multiplied for customers with two or more products. The segment dashboard continued to show correct numbers, because there was a manual distinct.
Solution: The team stopped the release, rolled back the change in the dbt project, wrote a separate analytical layer mart_customer_products with the grain "customer-product", and left the grain "customer" in mart_customer_360. After the incident, they introduced the SDD practice: for each mart in the repository, a YAML specification is stored with grain, mandatory and forbidden fields. Each PR is checked by a reviewer in a separate agent session, who compares the diff against the specification.
Result: The cost of the incident was three days of manual regulatory report recalculations and an explanatory note to compliance. After the introduction of SDD Data, such changes are blocked at the review stage: the reviewer notices that adding a field changes the grain and requires either moving it into a separate mart or explicitly describing the drift and getting the dashboard owner's signature.
Lessons learned: A green dbt build does not equal preservation of the mart's contract.
Grain is part of the specification, not a property of SQL.
A review in a "different mode" (a separate agent session) catches what the author misses due to context.
Related concepts: Mart grain
Invisible change of meaning
Data product specification
Contract drift and breaking change
Name: Case 2. PII leak through an analytical mart
Scenario: A large fintech, dbt + BigQuery. The open APIs team is preparing the mart_open_api_consents mart for external partners through a BI tool. The AI assistant, on the request "build the mart on consents with the maximum number of fields", included client_email and client_phone in the SELECT, considering them "useful for segmentation".
Challenge: Partners began seeing customer emails and phones in the dashboard, although consent to transfer this data to partners had not been obtained. The legal department recorded a personal data incident, followed by a fine and an obligation to notify customers.
Solution: The mart was rebuilt with a whitelist of fields described in the specification. A test was added in dbt that fails if a field from the "red list" appears in the schema. A YAML specification appeared in the repository with allowed_fields and forbidden_fields sections. Any PR that expands the SELECT is checked against the forbidden fields list on a mandatory basis.
Result: The cost of the incident was a regulatory fine, reputational losses, and rebuilding the partner API. After the introduction of SDD Data, the PII policy is described before SQL, not figured out in court after the release. The specification lives in the repository, next to the model, and any team member sees the boundaries.
Lessons learned: "Maximum fields" is a dangerous instruction for an agent without a specification.
A prohibition in SQL is easier to check than a prohibition in the analyst's head.
PII policy is a contract, not a best practice.
Related concepts: PII and personal data policy
Data contract
Data product specification
Three basic rules of SDD Data
Name: Case 3. A subtle freshness shift on payment risk signals
Scenario: The risk platform of a mid-sized bank, dbt + ClickHouse. The mart_payment_risk_signals mart is updated every 15 minutes. During DAG refactoring, the agent changed the schedule, and the actual freshness of the mart increased from 15 to 47 minutes. dbt tests continued to pass, because the data was fresh in absolute terms (no older than a day).
Challenge: The risk team discovered the shift only two weeks later, when the fraud scoring model began missing nighttime attack peaks: the mart simply did not have time to pick up the latest transactions. A direct incident was avoided, but the average fraud signal detection time increased.
Solution: An explicit freshness section with an upper limit of 20 minutes was added to the mart_payment_risk_signals specification. A verification fact appeared in the dbt project that compares max(updated_at) of the mart with the current time and fails if the gap is more than 20 minutes. This fact is included in the mandatory set of release evidence.
Result: The cost was two weeks of "blind zone" and dozens of hours of analysis by risk analysts. After the introduction of SDD Data, freshness shifts are detected within one release, not two weeks of operation. Freshness is part of the contract, not an operational detail.
Lessons learned: Freshness is a semantic promise, not an SLA in ITSM.
A "no older than a day" test does not protect business-critical marts.
The freshness verification fact should live in the repository next to the specification.
Related concepts: Verification fact
Data product specification
Contract drift and breaking change
Consumer chain
Study tips: Do not try to replace DuckDB with a production stack right away. In the first pass, it is more important to see the connection "specification → dbt → verification fact" than to assemble a production-grade infrastructure.
While reading, keep two things open: the chapter text and models/ in the local project. Check each theoretical statement with the corresponding file in the repository.
Before asking the agent to write SQL, write 5–10 lines of the specification yourself: grain, mandatory fields, forbidden fields, consumer, freshness. This exercise shows the fastest the difference between SDD Data and a regular dbt course.
Keep a separate file notes/sdd-data-checks.md and put the control questions from each part there. Before starting the next chapter, return to it and try to answer without hints.
When working with the agent, always open a new session (/clear) for the reviewer role. The author and the reviewer must work in different modes — this is rule #3, and it saves hours of analysis.
Run bash smoke_all.sh after every change. If the smoke test turns yellow or red, do not edit the dbt model "blindly" — first re-read the specification and the verification fact.
Re-read the section "The first risk: invisible change of meaning" before each new part. This scenario repeats itself in different forms, and it is easy to forget, drowning in mart details.
Do not mix the educational track and the production environment. If you have a working dbt project, map against it as a reference layer, but do not try to "apply" SDD Data to production in the first pass.
Additional resources: Book repository, directory book3/examples/bank-lakehouse: Contains the dbt project of the "Northern Bridge" bank's DataLakeHouse, synthetic data, and the smoke_all.sh script. This is the main practical artifact of the course.
Dbt documentation (dbt labs): A reference on models, tests, sources, exposures, and macros. Use it as a dictionary, not as a textbook — the conceptual framework is given in the book.
Dbt-utils and dbt-expectations: Packages with ready-made tests, useful for building verification facts (expect_column_values_to_be_between, recency, freshness).
The first volume of the series (sdd for web applications): The source of the SDD methodology. If you missed the first volume, read the introductory chapter — the cycle "describe → specify → implement → verify → review" is the same for both volumes.
Data product specification checklist (template in book3/templates/): A YAML/JSON template with sections on grain, mandatory/forbidden fields, freshness, owners, and verification facts. Copy and adapt for your project.
Cbr rf regulatory reporting standards (152-fz, information security regulations): The context in which the bank's DataLakeHouse lives. Not as a legal instruction, but as a source of constraints that make the specification mandatory.
Summary: Part 1 introduces the SDD Data methodology and fixes its key thesis: in a data platform, meaning is more important than code. An agent can invisibly change the grain, metric, freshness, PII policy, or compatibility with consumers, and a green dbt build will not catch this. Therefore, the workflow is opposite to a regular dbt course: first the platform mission, then the data product specification and contract, then raw/staging, then marts, then verification facts and independent review. dbt remains the primary transformation tool, but the source of meaning is the specification living in the repository. Three rules must be remembered from the first chapter: the specification lives in the repository, the verification fact has a command or a reviewer step, the author and the reviewer work in different modes. The educational project is the "Northern Bridge" bank's DataLakeHouse on DuckDB, which builds five marts (mart_customer_360, mart_card_turnover_daily, mart_payment_risk_signals, mart_credit_portfolio_quality, mart_open_api_consents). The minimum output of the part is to choose a reading mode, run a local smoke test, and record in notes that the book builds one reproducible data product process, not a "bank in general".