Part 3. DataLakeHouse Process Overview

SDD Data uses the same principle as the base SDD: first intent and verifiable facts, then implementation. The difference is that there are more artifacts. In a regular feature, requirements.md, plan.md, and validation.md are enough. In a data project, a source profile, Schema Manifest, ODCS/ODPS, model specification, dbt tests, lineage, and a reviewer report appear.

Main Flow

mission -> source profiling -> Schema Manifest -> data product specification
-> contract -> dbt model -> verifiable facts -> reviewer report

This flow does not have to be heavy. For a learning example, each artifact is short, but it answers a separate question:

• mission: why the platform exists;
• source profiling: what actually came in;
• Schema Manifest: which fields we observe and what is still unknown;
• data product specification: what we promise to the consumer;
• contract: which fields, SLAs, and quality must be preserved;
• dbt model: how we build the result;
• verifiable facts: how we prove readiness;
• reviewer report: who checked the change and where the risk remains.

Agent and Human Roles

The agent is good at mechanical work: reads CSV, suggests a manifest, writes SQL, finds discrepancies between the specification and the model. The human is responsible for meaning: grain, PII policy, business owner, breaking changes, release approval. If the agent itself decided that customer_id can be replaced with account_id, the process has already broken down.

Separate the roles:

• the profiler reads sources and returns facts;
• the builder writes models based on approved specifications;
• the reviewer compares the change against contracts and verifiable facts.

Even if all roles are performed by one tool, run them in different sessions after /clear. This tests whether the context is sufficiently written down in files.

Minimal Repository Structure

specs/
  mission.md
  tech-stack.md
  roadmap.md
  customer_360_product.odps.yaml
  customer_360_contract.odcs.yaml
  models/
    mart_customer_360.md
  validation/
    customer_360.md
models/
  staging/
  intermediate/
  marts/
tests/

If an important decision lives only in chat, the next agent will lose it. If it lies in the specification, the reviewer can check it independently.

Walkthrough for the Reader

The main danger of the DataLakeHouse process is that it looks like a linear ladder: source, manifest, contract, model, validation, review. In reality it is not a ladder but a system with feedback. Source profiling may reveal empty dates. The contract may find that the product is promised too broadly. The review may send the author back to grain. SDD does not forbid going back; it makes the return visible.

Each artifact in this chapter answers its own question. The manifest answers: "what did we see?" The contract answers: "what do we promise?" The model answers: "how do we build it?" The validation answers: "how do we prove it?" The reviewer report answers: "who looked at the whole chain?" If we combine these answers into one big document, the reader will first find it easier, and then harder: source, promise, and validation will start quietly substituting for one another.

Role separation is needed for the same reason. The model author naturally defends their decision. Even if the author is an agent, it has already walked the chosen path and tends to explain it as correct. A reviewer in read-only mode looks differently: they do not try to rescue the implementation, but check whether it holds the promise. After

/clear this check becomes more honest, because the reviewer does not remember the conversation where the author might have explained unrecorded decisions.

In practice, this chapter should teach the reader to see gaps. If SQL exists but there is no validation.md, it is not "almost ready." If ODCS exists but there is no validation of mandatory fields, it is not a contract but a declaration. If there is a reviewer report without references to specifications, it is an opinion, not evidence. SDD Data works only when the gaps become visible before the error reaches the mart.

Practice

Draw an artifact map for mart_customer_360: which files are responsible for source facts, product promise, contract, implementation, validation.md, and review. Separately mark gaps where a file has not yet been created or is not linked to validation.

Qwen Query for Overview

Read AGENTS.md, specs/mission.md, specs/tech-stack.md, and specs/roadmap.md.
Build a DataLakeHouse artifact map: which files are responsible for source facts,
product promise, contract, dbt implementation, and validation.
Do not change files.

Minimum Output

Create workflow-map.md or a working note:

Source facts:
Product promise:
Contract:
Implementation:
Validation:

Review:

For each item, specify the file or mark a gap. A gap is better seen now than after the SQL is already written.

Typical Mistake

Running a single /sdd:do-everything command. It is convenient in a demo, but destroys human control. In banking data it is dangerous when one agent profiles the source on its own, decides grain on its own, writes SQL on its own, and confirms readiness on its own.

Check Questions

1. How does the Schema Manifest differ from the data product specification?
2. Why must the builder and reviewer be separate roles?
3. How does /clear help verify that the process is written down in files?