Topic: Part 21. Qwen Code process for DataLakeHouse
Difficulty level: Medium
Estimated study time: 90-120 minutes (theory: 45 minutes, practice: 45-75 minutes)
Prerequisites: Basic understanding of Data Lakehouse architecture
Familiarity with dbt (data build tool) and its models
Understanding of the data contracts concept (ODCS, ODPS)
Experience with version control systems (Git)
General understanding of AI agents and LLM tools
Understanding of the separation of duties principles in a banking environment
Learning objectives: Explain why the SDD process should be split into separate commands with human confirmation points, rather than executed by a single command
Describe the purpose and responsibility zone of each of the five commands: /sdd:profile, /sdd:contract, /sdd:build, /sdd:verify, /sdd:release
Distinguish between skills and sub-agents and explain why the generator and reviewer must be different roles
Determine in which cases hooks should be introduced and which classes of changes they should block
Build a minimal process map with artifacts for the DataLakeHouse team
Overview: Chapter 21 brings together the material from the entire volume into a working cycle of interaction with Qwen Code when working with DataLakeHouse. The focus is on the five-step process /sdd:profile → /sdd:contract → /sdd:build → /sdd:verify → /sdd:release, in which each command has a narrow role and a human confirmation point. The main idea of the chapter: a single command should never execute the entire process, especially on banking data, where contract drift may be discovered too late. It discusses skills (data-profiler), sub-agents (data-contract-review), the concept of hooks as a disciplinary layer, and a typical mistake — creating a single /sdd:do-everything command that erases human confirmations and turns SDD into autopilot again. The chapter concludes the volume by giving the reader the ability to consciously choose which command is responsible for what, and which artifact should record the decision.
Key concepts: Sdd (specification-driven development): An approach in which work with data begins with a formal specification (contract) of the source, rather than direct modification of models. In the context of Qwen Code, this means that the agent's actions are determined by an explicit contract between the source and the consumer, not by LLM improvisation.
Five-step process /sdd:profile → /sdd:contract → /sdd:build → /sdd:verify → /sdd:release: Splitting the entire data work cycle into five separate commands, each with a narrow responsibility and a human confirmation point. This decomposition prevents the agent from having unlimited power over the data.
/sdd:profile: A source observation command. Reads metadata, schemas, volumes, statistics, and produces a Schema Manifest — an artifact with facts about the data without writing models. Has no right to change the structure of marts or contracts.
/sdd:contract: A command for formulating or verifying the promise. Works with ODCS/ODPS documents that describe what the source guarantees to the consumer. Does not touch models — only specifications.
/sdd:build: A command for making changes to dbt models. Triggered after the contract is approved by a human. Generates SQL, but should not validate itself.
/sdd:verify: A command for running checks and searching for drift. Performs dbt build, tests, compares actual data against the contract. Looks for discrepancies between the promise and reality.
/sdd:release: A command for assembling an evidence package. Produces artifacts for stakeholders: what changed, which tests passed, who confirmed.
Data-profiler (skill): A narrow skill for reading sources and producing a manifest. Returns only facts, has no right to write models. An example of a skill that limits the agent's role.
Data-contract-review (sub-agent): A read-only sub-agent that applies a review checklist to proposed changes. Cannot edit SQL — only review.
Separation of generator and reviewer roles: The principle that the author of changes and the checker must be different entities. Otherwise the agent verifies its own guesses, which is dangerous on banking data.
Human confirmation points: Places in the process where the agent must stop and hand the decision over to a living specialist. Without these points, SDD degrades into autopilot.
Hooks: A mechanism for automatically blocking dangerous classes of changes: PII fields in models/marts/, changes to the contract YAML without a confirmation tag, changes to grain without an impact note. Hooks do not replace review — they prevent obvious errors from reaching it.
Artifact: The tangible result of a command's work (Schema Manifest, updated ODCS, modified dbt model, reviewer's report). Rule: if a command leaves no artifact, it is not needed on the first pass.
Process map: The minimal visual output of the chapter — a table mapping commands to artifacts. Allows you to quickly explain to the team who is responsible for what.
Contract drift: The discrepancy between what the source promises (contract) and what actually happens with the data. Especially critical in a banking environment, where drift is discovered too late.
Pii (personally identifiable information): Customers' personal data. Adding fields with the pii_ prefix to marts models is a typical dangerous class of changes that should be blocked by a hook.
Grain (model granularity): The level of detail of records in a dbt model (for example, "one customer — one row"). Changing the grain without an explicit note about downstream impact is a frequent source of errors.
Important dates: Adoption of sdd in datalakehouse: A maturity stage of the process at which recurring violations become visible — a sign that it is time to write hooks
First practical milestone: The moment after which the team already understands typical violations and can formulate rules for hooks
After /clear: The moment when a reviewer in read-only mode is especially important: if they cannot understand the change from the files, it means the author left the meaning in the chat — a signal of poor discipline
Practice exercises: Name: Exercise 1: Building a process map for your project
Problem: You work on a banking DataLakeHouse team that is transitioning to SDD with Qwen Code. Describe a process map: which five commands you will use, what artifact each of them leaves, and where the human confirmation point is located. Use the table format from the chapter.
Solution: Step 1. Write out the five commands in the correct order: /sdd:profile, /sdd:contract, /sdd:build, /sdd:verify, /sdd:release. Step 2. For each command, determine the artifact: profile → Schema Manifest; contract → ODCS/ODPS update; build → dbt model change; verify → dbt build + verification facts; release → evidence package. Step 3. Add a sixth row for data-contract-review → reviewer report. Step 4. Mark with arrows where the human confirmation is located (between contract and build, between verify and release). Step 5. Check yourself: does each command have an artifact? If not, that command is unnecessary on the first pass.
Complexity: beginner
Name: Exercise 2: Separation of author and reviewer roles
Problem: Run the same request to change a dbt model (for example, adding the customer_segment field to a marts table) in two modes: (a) one agent does everything itself — profiles the source, proposes the change, writes SQL, and reviews it; (b) separation of roles: the author agent proposes the change, the reviewer agent in read-only mode checks it. Compare which comments appear only when roles are separated.
Solution: Step 1. Prepare the context: source description, current contract, existing dbt model. Step 2. Mode (a): ask one agent to execute the entire cycle. Record which risks it allowed itself. Step 3. Mode (b): launch the author — it writes the change. Launch the reviewer in read-only mode (files only, no chat) — it applies the checklist. Step 4. Compare the results. Typical comments that appear only with separation: "impact on grain not described", "ODCS not updated", "no NULL test on the new field", "not verified who the consumers of this column are", "not noted that the field may contain PII". Step 5. Formulate a conclusion: separation of roles reveals risks that the author does not notice, because they are "beyond its optimism".
Complexity: intermediate
Name: Exercise 3: Designing the first hook
Problem: The team has run three sprints on SDD. A review analysis showed that three violations recur most often: (1) fields with the pii_ prefix appear in models/marts/; (2) contract YAML is changed without a human confirmation tag; (3) model grain is changed without a note in the PR description. Design one hook that blocks all three classes of changes. Describe it conceptually (without code).
Solution: Step 1. Formulate the blocking condition: the hook triggers when attempting to write files to models/marts/. Step 2. Rule 1: if a new or modified file contains a field starting with pii_ — block the commit with the message "PII in marts requires separate approval". Step 3. Rule 2: if any file in the contracts/ directory or .odcs.yaml is changed in the PR, and the commit/PR message does not contain the "contract-approved" marker — block. Step 4. Rule 3: if a file in models/marts/ is changed, and the line {{ config(materialized=..., grain=...) }} or its equivalent is changed in it — require the PR description to contain the string "grain-impact:". Step 5. Decide in which order the hook emits errors and which are only warnings. Step 6. Record: the hook does not replace review — it prevents obvious errors from reaching it.
Complexity: intermediate
Name: Exercise 4: Diagnosing the "/sdd:do-everything" error
Problem: The team created a single /sdd:do-everything command that profiles the source, updates the contract, generates a dbt model, runs tests, and publishes a report. After a month, a field with PII that was not in the contract made it into production. Conduct an analysis: which human confirmation points were erased and which artifacts were missing?
Solution: Step 1. List the confirmation points the command should have had: (a) after profile — confirmation that the Manifest is correct; (b) after contract — confirmation that the promise is agreed with source stakeholders; (c) after build — confirmation that the change is correct; (d) after verify — confirmation that the evidence package is complete. Step 2. In the /sdd:do-everything command, all these points are merged into a single automatic execution. The person sees only the final report, not the intermediate artifacts. Step 3. Which artifacts were missing: Schema Manifest was not reviewed by a source expert; ODCS update was not signed by the data owner; the contract reviewer did not receive a separate ticket; there is no human confirmation tag in the logs. Step 4. Where exactly was the PII "missed": the agent added the field to marts because it saw it in the source, and the contract for that source either was not updated or was generated automatically. Step 5. Conclusion: a single command erases human confirmations and turns SDD into autopilot. The solution is to split it into five commands and explicitly require manual confirmation between them.
Complexity: advanced
Case studies: Name: Case 1: Adopting the five-step SDD in a banking DataLakeHouse
Scenario: A mid-size retail bank (5 million customers) adopts Qwen Code to accelerate the development of dbt models in the corporate DataLakeHouse. Before adoption, analysts spent up to 40% of their time describing sources and agreeing on contracts, and changes often bypassed formal procedures. A pilot team of 6 data engineers starts by creating five separate commands: /sdd:profile, /sdd:contract, /sdd:build, /sdd:verify, /sdd:release.
Challenge: Main challenges: (1) how to convince the team not to collapse the process into one command due to the "inconvenience" of switching; (2) how to formalize human confirmation points so that they do not turn into formal checkboxes; (3) how to set up different roles for the generator and reviewer so that the latter does not depend on chat context; (4) how to choose the first class of changes to block with a hook without overcomplicating the process.
Solution: The team went through several stages. Stage 1: they created the files .qwen/commands/sdd/profile.md, contract.md, build.md, verify.md, release.md — each with its own narrow instruction. Stage 2: they introduced the data-profiler skill, which returns only a Schema Manifest with no right to write models. Stage 3: they separated the data-contract-review sub-agent, working only with repository files (read-only mode) and applying a checklist. Stage 4: after the first month, they analyzed recurring violations — it turned out that in 4 out of 11 cases, PII fields ended up in marts without a note in the ODCS. This became the basis for the first hook, blocking the writing of fields with the pii_ prefix to models/marts/. Stage 5: they recorded the process map and made it a mandatory part of onboarding new analysts.
Result: After three months: the average time from change request to release decreased by 28% (from 9 to 6.5 days), the number of regressions from non-contract fields dropped to zero, and the time spent describing sources decreased by 35%. The main qualitative effect: an explicit owner appeared for each artifact and an explicit point where the decision is made by a human, not an LLM.
Lessons learned: Splitting into five commands is not bureaucracy, but a way to make responsibility visible: the profile does not write models, the reviewer does not edit SQL, the generator does not certify its own safety
Hooks are useful only after the team itself has seen recurring violations — otherwise the hook is either too weak or too annoying
A read-only reviewer is especially valuable after /clear: if they cannot understand the change from the files, it means the author left the meaning in the chat — a signal of poor discipline that is easy to catch with a formal check
The process map is a minimal but mandatory artifact: without it, new team members do not understand where their confirmation point is
Related concepts: Five-step process /sdd:profile → /sdd:contract → /sdd:build → /sdd:verify → /sdd:release
Skills and sub-agents
Separation of generator and reviewer roles
Human confirmation points
Hooks
Name: Case 2: The /sdd:do-everything single-command error in a core banking DataLake
Scenario: A large bank (top-10), in an attempt to "accelerate DataOps", created a single /sdd:do-everything command that itself profiled the source, updated the contract, generated a dbt model, ran tests, and published a report. The idea was that a single analyst request would turn into a complete change in one cycle. The first two months went smoothly.
Challenge: In the third month, a field passport_series with the pii_ prefix, which was not in the source's ODCS contract, ended up in the marts table mart_customer_360. The field leaked into three downstream reports, including the Central Bank's regulatory dashboard. The complexity: (1) the command left no intermediate artifacts — there was only the final report; (2) the Schema Manifest was not reviewed by a human source expert; (3) the ODCS update was generated automatically, without the data owner's signature; (4) there was no manual confirmation tag in the logs. Root-cause analysis took more than a week.
Solution: The bank abandoned /sdd:do-everything and moved to five commands. Profile and contract were transferred to a separate role — Data Steward. The dbt-model generator received a limited scope (build only) and a prohibition on updating contracts/. The data-contract-review reviewer was launched as a separate sub-agent in read-only mode, receiving only a diff from Git, not chat context. A hook was written immediately: it blocks any write to models/marts/ with a field starting with pii_, and requires the contract-approved tag on any changes to ODCS files. Human confirmation points were enshrined in the security policy: no dbt model goes to main without a Data Steward's signature in the PR description.
Result: No similar incidents recurred during the next quarter. Central Bank auditors separately noted the presence of an explicit process map with artifacts as a positive signal of DataOps maturity. Development speed recovered 6 weeks after the initial decline — analysts acknowledged that five short steps with explicit confirmations are faster than one long command with incident analysis in case of error.
Lessons learned: The single /sdd:do-everything command is convenient in demos but dangerous on banking data: human confirmation disappears, and contract drift is discovered too late
An artifact is not a report, but a pivot point for investigation: when there is no Schema Manifest and no signed ODCS, there is no one to hold accountable and no way to prove what was promised
Separation of roles is not a formality, but a way to find out who should have noticed the problem: after the error, it is precisely the role that shows where the step was missed
Hooks in a banking environment should block not "everything suspicious", but a narrow known dangerous class — otherwise people will start bypassing them
Related concepts: Typical error: /sdd:do-everything
Contract drift
PII in marts
Artifact as a pivot point
Hooks as a disciplinary layer
Name: Case 3: Evolution from skill to sub-agent in a risk-modeling team
Scenario: A risk-modeling team at an investment bank uses Qwen Code to speed up the preparation of data marts for credit scoring models. At the first stage, they described data-profiler as a skill, and data-contract-review as a simple instruction in chat. After two months it became clear that "review in chat" is easily ignored: the author simply replies "everything is ok" and continues.
Challenge: Problem: the reviewer must be a structurally separate role, not a behavior mode of the same agent. When review is a chat instruction, the agent remains in the same session and inherits the author's context, including their confidence in the correctness of the decision. In addition, the reviewer must see only the files, not the correspondence — otherwise they will evaluate not the code, but the intentions.
Solution: The team moved the review into a separate data-contract-review sub-agent with its own system prompt and read-only mode on the repository. The only input is a diff from Git, no chat context. The review checklist was formalized and now includes items: "field with the pii_ prefix — forbidden in marts", "grain change — requires grain-impact in the description", "ODCS update — requires contract-approved in the commit", "new dependency on a source — requires profile again". Outside a /clear session, the reviewer remains fully functional, which is what was required.
Result: Over the quarter, the average number of comments found by the reviewer increased from 0.4 to 2.7 per PR. Particularly valuable was the comment "unclear from the files what the change does" — it appeared for authors who were used to explaining logic in chat rather than in commits. This forced the team to tighten the practice of commit messages and PR descriptions, which improved the overall code quality regardless of the AI tool.
Lessons learned: Skill and sub-agent are different levels of isolation: a skill narrows the role in the same session, a sub-agent provides separate context and mode
A read-only reviewer works best after /clear: they cannot be persuaded, because they have no access to the chat
The review checklist must be explicit, otherwise review degrades into "everything looks fine"
The "unclear from the files" comment is not a reviewer bug, but a feature: it catches poor change documentation
Related concepts: Skills and sub-agents
data-contract-review
data-profiler
Read-only mode
Review checklist
Study tips: First, build the process map on paper before creating the command files: five commands, five artifacts, explicit confirmation points. It will take 20 minutes, but save weeks of fighting /sdd:do-everything
Run one task in two modes — one command and five commands — and record the difference in the number of risks found. This is more illustrative than any theory
Do not write hooks before you see recurring violations: spend the first two or three sprints gathering statistics, and only then formulate rules
Always keep the reviewer in read-only mode and without access to chat: this is the only way to force the author to write clear commits and PR descriptions
After /clear, check whether the reviewer can understand the change from the files. If not, it is a signal that the author left the meaning in the chat, and discipline requires correction
The principle "if a command leaves no artifact, it is not needed on the first pass" is a good filter: five commands produce five artifacts; a command without an artifact is either an extra step or a poorly defined role
Link each rule to a class of changes: PII in marts, grain change, ODCS update without a tag. Abstract rules like "the code should be of high quality" do not work in review or in hooks
Use the chapter's terminology: "human confirmation point", "artifact", "contract drift", "grain". This is a common language that allows the team to quickly agree on changes
Additional resources: Open data contract standard (odcs): A standard for describing data contracts used in the chapter. Helps formalize the source's promise to the consumer. https://bitol-io.github.io/open-data-contract-standard/
Dbt (data build tool): A data transformation tool mentioned in the context of the /sdd:build and /sdd:verify commands. Documentation: https://docs.getdbt.com/
Pii in data lakehouse: An overview of practices for working with personal data in data lakehouses. Useful for understanding why PII in marts is a typical dangerous class
Data contracts (mark lopez-restrepo et al.): The conceptual basis for data contracts as a way to record promises between data producers and consumers
Qwen code and qwen3-coder: Documentation for the Qwen Code AI tool, including work with commands, skills, and sub-agents. https://qwenlm.github.io/
Spec-driven development (sdd): The methodology underlying the chapter. Related to BDD and TDD, but with a focus on data specifications and contracts
Separation of duties (sod) in a banking environment: A concept from the area of internal banking control that explains why the generator and reviewer must be different roles
Summary: Chapter 21 assembles the entire volume into a working Qwen Code process cycle for DataLakeHouse: five commands (/sdd:profile, /sdd:contract, /sdd:build, /sdd:verify, /sdd:release), each with a narrow role, an explicit artifact, and a human confirmation point between them. The main idea: one command should not do the entire process — otherwise human confirmations disappear, and contract drift is discovered too late, which is especially critical on banking data. Skills (data-profiler) and sub-agents (data-contract-review) are useful when they narrow the role: the profiler does not write models, the reviewer does not edit SQL, the generator does not certify its own safety. Hooks are the next level of discipline, but they cannot be written before the process: first the team must see recurring violations (PII in marts, ODCS change without a tag, grain change without a note), and only then formalize them into rules. The minimal output of the chapter is a process map with five commands, five artifacts, and one row for the reviewer report. After reading, the reader should confidently answer three control questions: why the process cannot be done with one command, how a skill differs from a sub-agent, and where the human confirmation should be located.