Reading: Part 4. Environment: Qwen Code, dbt, DuckDB

Lesson 1 of 5 in module «Part 4. Environment: Qwen Code, dbt, DuckDB»

You are viewing the lesson without signing in. Sign in to save progress and take tests.

Source

Part 4. Environment: Qwen Code, dbt, DuckDB

The environment must be local, reproducible, and small enough that the student sees the SDD cycle, not a fight with infrastructure. That is why the tutorial track uses Python, DuckDB, dbt-core, and dbt-duckdb.

Minimum Requirements

Python 3.11+;
venv;
a shell with bash;
Qwen Code CLI or a compatible agent tool;
access to the local repository.

Verification:

python3 --version
python3 -m venv /tmp/sdd-data-check
. /tmp/sdd-data-check/bin/activate
pip --version

Example Installation

cd book3/examples/bank-lakehouse
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
bash smoke_all.sh

A full smoke run must generate raw/seeds, run dbt parse and dbt build. If dbt is not installed, the fallback mode will check only the structure. This is a partial verification, not proof that the tutorial is ready.

Why DuckDB

DuckDB makes it possible to show SQL, types, aggregates, dbt tests, and local storage without an external service. This makes mistakes cheaper. If revoked_at is empty and a type cast breaks, the student sees the problem in the source/staging contract, not in cluster settings.

DuckDB does not replace a production environment. It replaces the first tutorial environment. Once grain, PII policy, and contracts are understood locally, the same schema can be moved to Trino, Spark, Databricks, or Snowflake.

Agent Setup

The example includes AGENTS.md and .qwen/commands/sdd/. Before any change to models, the agent must read:

AGENTS.md;
specs/mission.md;
specs/tech-stack.md;
the relevant ODCS/ODPS;
the relevant model specification;
verification facts.

Do not start with "fix dbt". Start with "compare the error to the contract and show which fact is violated".

Reader Walkthrough

The local environment in this book is intentionally modest. This is not an attempt to pretend that a production environment is simple. Quite the opposite: the small stack is needed to separate a tutorial mistake from infrastructure noise. When dbt build fails on an empty date, the reader should see the problem in staging and the contract, not wrestle with the cluster, secrets, permissions, the catalog, and network policies.

DuckDB is useful precisely as a laboratory. It is similar enough to an analytics engine to show types, aggregates, materialization, and tests. At the same time it is close enough that a student can delete the database, rebuild the data, and get the same result again. That reproducibility matters more than the feeling of a "real" platform. If the first tutorial environment is not reproducible locally, moving to Iceberg or Spark will only add new reasons for accidental discrepancies.

A separate venv is also part of the discipline. It fixes the boundaries of the example: here are the tool versions, here is the install command, here is the verification command. When pins are updated, this must be confirmed by a smoke run, not by trust in the word "latest". This is especially important for a tutorial: the reader copies the commands, and if they break, the reader stops trusting not only the environment but also the process.

Finally, the environment sets the right attitude toward the agent. The agent should not receive the command "fix everything" when dbt does not run. First you need to understand which boundary is violated: a dependency is not installed, raw data is missing, staging misinterprets an empty value, the contract does not match the model. The smaller the environment, the easier it is to learn this diagnosis.

Practice

Run smoke_all.sh in two modes: without an activated venv and after installing dependencies. Record which mode is partial, which is full, and which output lines prove a successful dbt build.

Minimum Output

After the chapter you should have:

a locally installed environment;
a successful or honestly recorded smoke_all.sh;
an understanding of which verification mode was full and which was partial;
a note on where specs, models, tests, and reviewer artifacts are stored.

Common Mistake

Ignoring the fallback smoke run. The message "structure and data generation check passed" does not mean that the dbt models are correct. It only checks that the example can be opened and raw files generated.