Topic: Part 5. Initial Project Setup
Difficulty level: Medium
Estimated study time: 1.5–2 hours
Prerequisites: Basic understanding of dbt and model layers (staging, intermediate, marts)
Familiarity with the concept of an AI agent as a development participant
Understanding of version control principles (Git) and working with a repository
Experience with relational data warehouses and SQL
Understanding of what a model specification and a mart contract are
Learning objectives: Explain why initial project setup is more important in data projects than in regular applications
Create a minimal project directory structure according to the course standard
Formulate five short behavioral rules for the agent in AGENTS.md
Distinguish the areas of responsibility of AGENTS.md and specs/tech-stack.md
Build a project roadmap based on artifacts and verification facts
Formulate a Qwen query for auditing the project scaffold without changing files
Verify the project setup with the criterion "after /clear, a new agent understands the project without a chat recap"
Overview: The chapter describes the minimal but mandatory initial setup of a data project in which an AI agent and a human work as a pair. The goal of the setup is to transfer decision memory from the chat into repository files: mission, technology stack, roadmap, model specifications, verification notes, and reviewer reports. The chapter breaks down why the specs/, models/, and tests/ directories are needed, what exactly to write in AGENTS.md, why a roadmap by layers is bad, and which query to use to commission an audit of the scaffold from the agent. After the chapter, the repository should contain AGENTS.md, specs/mission.md, specs/tech-stack.md, specs/roadmap.md, directories for model specifications, verification notes and reviewer reports, as well as a dbt project scaffold. The chapter teaches the main principle: at the setup stage, not finished artifacts are created, but "places" where these artifacts will end up. Setup is not finished when the folders are created, but when they start holding decisions.
Key concepts: Project memory vs chat memory: Project memory is the folder structure and files (mission.md, tech-stack.md, roadmap.md, model specifications, verification notes, reviewer reports). Chat memory is the content of the dialog with the agent, which disappears on the first /clear. Before the context is reset, chat memory seems convenient, but for the next human and the next reviewer it is useless. Initial setup is needed so that decisions live in files, not in the dialog window.
Minimal directory structure: Standard repository layout: specs/ (with files mission.md, tech-stack.md, roadmap.md and subdirectories models/, validation/, reviewer-reports/), models/ (with subdirectories staging/, intermediate/, marts/), tests/, AGENTS.md, dbt_project.yml, profiles.yml. At the setup stage, you need not artifacts ready for a production environment, but "places" where decisions will land. If, for example, there is no specs/validation/ folder, verification facts will almost certainly remain in the chat or in the PR description.
Agents.md as a behavioral file: This is a short file that defines the agent's behavior before changes: what to read (mission, tech-stack, roadmap, relevant specifications), what not to do (do not disclose direct PII, do not change grain, SLA, or contract fields without confirmation), which checks to run (dbt build), how to report (with verification facts, not promises). AGENTS.md is not an encyclopedia, but a list of behavioral rules. The shorter the rules, the higher the chance that the agent and the human will actually use them.
Separation of agents.md and specs: AGENTS.md describes how the agent should behave. specs/mission.md describes what the project is for. specs/tech-stack.md describes which stack is allowed and why. specs/roadmap.md describes which artifacts must appear. specs/models/ stores model contracts. specs/validation/ stores verification notes. specs/reviewer-reports/ stores reviewer reports. The product promise, contract fields, and acceptance facts must live in specs/, not in AGENTS.md.
Roadmap by artifacts: In a data project, it is better to keep the roadmap based on evidence, not layers. An example of a bad formulation is "the marts/customer_360.sql model is written." An example of a good formulation is "the Customer 360 mart has been verified by a reviewer." This way it is clear that "the model is written" does not equal "the product is ready." A concrete example from the chapter: 1) raw/staging are reproducible; 2) the Customer 360 contract is approved; 3) the Customer 360 mart is verified; 4) the risk, loans, and consents marts have been verified by a reviewer; 5) release evidence for the practical credit is ready.
Layers of meaning, verification, and review: The folders specs/models/, specs/validation/, and specs/reviewer-reports/ tell the agent that meaning, verification, and review are separate layers, not an afterthought to SQL. If these folders do not exist, the source of truth will appear wherever the agent finds it convenient: in a comment, in command output, in the response text. Restoring it later is very difficult.
Containers for decisions: At the setup stage, you need to create containers for decisions, not make the decisions themselves. The task is to agree in advance where the mission, stack, roadmap, specifications, verification notes, and reports will live. The contents of these folders are decided later, in the following chapters of the course.
Qwen query for setup audit: A query that instructs the agent to read the current project structure, check for places for mission, tech-stack, roadmap, model specifications, verification facts, dbt models, and reviewer reports, and compose a list of setup gaps without changing files. After the list of gaps, you can ask the agent to create only the missing directories and empty templates. Asking it to write SQL at this stage is not allowed.
Setup readiness criterion: A good setup gives a simple test: after clearing the context, a new agent should understand where the mission is, which stack is allowed, which marts are being built, and how readiness is proven. If you need to recap the chat to continue work, then the project is not yet set up. Setup is not finished when the folders are created, but when they start holding decisions.
Typical setup mistake: Immediately asking the agent to "assemble a lakehouse" is too broad a request. At the setup stage, you need to create containers for decisions, not make the decisions themselves. Such a request leads the agent to a discussion of the stack, formats, and orchestrators in the chat, and not a single structural unit appears in the repository.
Practice exercises: Name: Creating a minimal project scaffold
Problem: In an empty repository, create the following directory structure: specs/ (with subdirectories models/, validation/, reviewer-reports/), models/ (with subdirectories staging/, intermediate/, marts/), tests/. Add empty stub files: specs/mission.md, specs/tech-stack.md, specs/roadmap.md, AGENTS.md, dbt_project.yml, profiles.yml. Describe which commands you use and in what order, and explain why the order of creating directories matters for further work with the agent.
Solution: Step 1. Create the root directories with a single command: mkdir -p specs/models specs/validation specs/reviewer-reports models/staging models/intermediate models/marts tests. The -p flag is needed to create nested directories in a single command. Step 2. Create empty stub files: touch specs/mission.md specs/tech-stack.md specs/roadmap.md AGENTS.md dbt_project.yml profiles.yml. Step 3. Fix the structure in Git: git init && git add . && git commit -m 'scaffold: minimal project structure'. Step 4. Explain the order: specs/ is the contract, models/ is the implementation, tests/ is the verification. Creating specs/ before models/ fixes the priority of meaning over code. If you create models/ first, the agent will start writing SQL before places for the mission and specifications appear, and the source of truth will move into comments and chat responses.
Complexity: intermediate
Name: Writing AGENTS.md according to the minimal template
Problem: Fill AGENTS.md with the five rules from the chapter. Formulate them unambiguously and concisely: 1) what the agent must read before changing dbt models; 2) what must not be disclosed; 3) what must not be changed without confirmation; 4) which command to run; 5) how to report. Explain why each rule should be short and why AGENTS.md is a bad place to describe marts.
Solution: Step 1. The first rule is preliminary reading: "Before changing dbt models, read the mission, tech-stack, roadmap, and the relevant model specifications." Step 2. The second rule is a PII restriction: "Do not disclose direct PII fields in marts." Step 3. The third rule is a prohibition on silent contract changes: "Do not change grain, SLA, or contract fields without confirmation." Step 4. The fourth rule is mandatory verification: "Run dbt build when dbt is available." Step 5. The fifth rule is the report format: "Report with verification facts, not promises." Step 6. Explanation: rules should be short, because a long AGENTS.md turns into an encyclopedia that nobody reads — neither the human nor the agent. Mart descriptions should live in specs/models/, not in AGENTS.md, because this is product knowledge, not agent behavior.
Complexity: intermediate
Name: Auditing the scaffold via a Qwen query
Problem: Formulate a query to the AI agent that must: 1) read the current project structure; 2) check for places for mission, tech-stack, roadmap, model specifications, verification facts, dbt models, and reviewer reports; 3) compose a list of setup gaps without changing files. Then formulate a second query that, based on the list of gaps, creates only the missing directories and empty templates, without writing SQL and without filling in the contents of specifications.
Solution: Step 1. First query (read only): "Read the current project structure. Check whether there are places for mission, tech-stack, roadmap, model specifications, verification facts, dbt models, and reviewer reports. Compose a list of setup gaps. Do not change files." Step 2. Second query (containers only): "Based on the list of gaps, create only the missing directories and empty templates. Do not write SQL, do not fill in the contents of specifications." Step 3. Explanation: the first query is separated from mutations, so the agent will not accidentally overwrite existing files and the team will see a clean list of gaps. The second query is limited to creating empty containers, so the agent will not slip into writing models and discussing the stack prematurely. This is the principle of "containers for decisions, not the decisions themselves."
Complexity: intermediate
Name: Building a roadmap by artifacts
Problem: For a fictional banking data project "Customer 360," compose a roadmap of five items. Each item must name an artifact or verification fact, not a layer. Explain why the formulation "the marts/customer_360.sql model is written" is bad, and "the Customer 360 mart has been verified by a reviewer" is good. Add to each item the expected evidence (where it will live).
Solution: Step 1. Item 1: "raw/staging are reproducible" — the evidence lives in specs/validation/ (a report on a reproducible run) and in tests/ (passed dbt build tests). Step 2. Item 2: "the Customer 360 contract is approved" — the evidence lives in specs/models/customer_360.md with the signature of the mart owner. Step 3. Item 3: "the Customer 360 mart is verified" — the evidence lives in specs/validation/customer_360.md (a report on field, grain, and SLA verification). Step 4. Item 4: "the risk, loans, and consents marts have been verified by a reviewer" — the evidence lives in specs/reviewer-reports/. Step 5. Item 5: "release evidence for the practical credit is ready" — this is an aggregate item referring to all previous folders. Step 6. Explanation: the formulation "the model is written" describes an action, not a result; the formulation "verified by a reviewer" describes evidence. A roadmap by artifacts immediately shows which evidence has not yet been collected and where it should live.
Complexity: intermediate
Name: Separating AGENTS.md and specs zones
Problem: You are given four statements: 1) "PostgreSQL is the allowed storage layer because it is already used in the bank"; 2) "The agent must not run dbt build without reading mission.md"; 3) "Customer 360 contains the fields customer_id, full_name, birth_date, risk_score"; 4) "Every PR for a mart must be accompanied by a reviewer report." Distribute them across the correct files in the repository and explain why AGENTS.md is a bad place for statement 1 and a good place for statement 2.
Solution: Step 1. Statement 1 ("PostgreSQL is allowed because it is used in the bank") goes into specs/tech-stack.md — this is a description of the allowed stack and the rationale for the choice, not agent behavior. Step 2. Statement 2 ("the agent must not run dbt build without reading mission.md") goes into AGENTS.md — this is a rule of agent behavior before an action. Step 3. Statement 3 ("Customer 360 contains the fields...") goes into specs/models/customer_360.md — this is the contract of a specific model, not a behavior rule. Step 4. Statement 4 ("every PR must be accompanied by a reviewer report") goes into AGENTS.md as a behavioral rule for the agent and into specs/roadmap.md as a process requirement. Step 5. Explanation: AGENTS.md is a bad place for statement 1 because it describes the stack, not behavior; mixing creates a bloated AGENTS.md that the agent stops reading. AGENTS.md is a good place for statement 2 because it is a "read before acting" rule, i.e., behavior.
Complexity: advanced
Case studies: Name: Banking Customer 360: how the chat "ate" a forgotten PII restriction
Scenario: A team of three analysts and an AI agent was building the Customer 360 mart for the bank's risk department. At the start of the project, the team agreed in the chat that the email field could not be exported to marts without masking, and that the mart grain was "one customer — one row per snapshot date." The agreements remained in the dialog window. Two weeks later, a new analyst joined the repository and connected a fresh context to the agent.
Challenge: After the /clear command, the agent did not find any mentions of email masking or mart grain in the project. It generated a mart in which email was output in cleartext, and the grain blurred to "one row per customer operation" — because that was more convenient for the current risk department request. The SQL passed all dbt build tests and gave a green result, but violated agreements that lived only in the chat. The risk department received a mart with PII and incorrect grain, which made any aggregations meaningless.
Solution: The team stopped the release, manually restored the chat history, and fixed two new folders in the repository: specs/models/customer_360.md with grain, SLA, field list, and an explicit ban on cleartext email, and specs/validation/ for verification notes. Two rules were added to AGENTS.md: "Do not disclose direct PII fields in marts" and "Do not change grain, SLA, or contract fields without confirmation." specs/tech-stack.md recorded that PII masking is a mandatory pipeline step, not optional. The roadmap was rewritten by artifacts: "contract approved → mart verified → mart approved by reviewer."
Result: A week later, the mart was rewritten: email was output only in masked form, and the grain was restored to "one customer — one row per snapshot date." A new analyst joining the project for the first time was able to continue work without a chat recap: he read specs/models/customer_360.md and immediately saw the restrictions. A green dbt build ceased to be a sign of readiness — readiness was now proven by the contents of specs/validation/ and specs/reviewer-reports/.
Lessons learned: Agreements left in the chat disappear at the first /clear and are invisible to a new person
A green dbt build does not equal product readiness — specifications and verification facts are needed
AGENTS.md should be a short list of behavioral rules, not a copy of specifications
A roadmap by artifacts immediately shows which evidence has not yet been collected
The specs/models/, specs/validation/, and specs/reviewer-reports/ folders must exist before the first SQL model
Related concepts: Project memory vs chat memory
Separation of AGENTS.md and specs
Layers of meaning, verification, and review
Roadmap by artifacts
Setup readiness criterion
Name: E-commerce analytics: the attempt to "assemble a lakehouse right away"
Scenario: An e-commerce project manager asked the AI agent to "assemble a lakehouse for sales analytics." The project had neither mission.md, nor tech-stack.md, nor specs/. The repository contained only an empty README and a data/ folder with raw dumps.
Challenge: The agent began by trying to choose between Spark, ClickHouse, BigQuery, and Snowflake, discussing data models, file formats, and orchestrators right in the chat. After an hour, the dialog had grown to hundreds of messages, but not a single structural unit other than README had appeared in the repository. When a second analyst arrived, he could not understand which decisions had already been made, because they were smeared across the chat. Any /clear erased the context, and the work had to start over.
Solution: The manager stopped the dialog and began setting up the project according to the rules of the chapter. He created specs/mission.md ("e-commerce sales analytics, focus on repeat purchases"), specs/tech-stack.md ("ClickHouse as the storage, dbt as the transformation layer, Airflow as the orchestrator"), specs/roadmap.md by artifacts, and AGENTS.md with five short behavioral rules. After that, a Qwen query was sent to the agent for a scaffold audit: "Read the current project structure. Check whether there are places for mission, tech-stack, roadmap, model specifications, verification facts, dbt models, and reviewer reports. Compose a list of setup gaps. Do not change files." Based on the list of gaps, the agent was instructed to create only the missing directories and empty templates, without writing SQL.
Result: After 30 minutes the scaffold was ready, and the second analyst entered the project already after setup. He read mission.md, tech-stack.md, and roadmap.md in 10 minutes and immediately understood what was being done and why. After that, the team moved on to model specifications and only then to SQL. The onboarding time for a new team member dropped from several days to one hour. A new agent after /clear also understood the project without a chat recap, which confirmed the setup readiness criterion.
Lessons learned: Too broad a query to the agent ("assemble a lakehouse") leads to a dialog without artifacts in the repository
Project setup is creating containers for decisions, not making the decisions themselves
A Qwen query with a ban on changes is well suited for scaffold auditing
A roadmap by artifacts helps avoid falling into "layers for the sake of layers"
A short AGENTS.md works better than a long instruction
Related concepts: Containers for decisions
Qwen query for setup audit
Minimal directory structure
Roadmap by artifacts
Typical setup mistake
Name: Marketplace: AGENTS.md as an encyclopedia and broken onboarding
Scenario: A marketplace analyst wrote AGENTS.md 12 pages long: it included dbt instructions, descriptions of all marts, security requirements, data schemas, and even pieces of SQL. The file was regularly updated, but nobody read it in full — neither the human nor the agent.
Challenge: When a new agent joined the project, it ignored the long AGENTS.md and started relying on its own "common sense." In particular, it began changing the orders mart grain without confirmation, because the rule banning grain changes had sunk between schema descriptions and SQL examples. The reviewer noticed the problem only after the mart had gone into production and broken the revenue report. At the same time, verification facts were scattered across PR comments and the chat, so it was impossible to reconstruct the course of the agent's reasoning.
Solution: The team rewrote AGENTS.md according to the rules of the chapter: left five short behavioral rules (what to read, what not to disclose, what not to change without confirmation, which command to run, how to report). All product descriptions, mart descriptions, contract fields, and acceptance facts were moved into specs/: into specs/mission.md, specs/tech-stack.md, specs/roadmap.md, specs/models/, and specs/validation/. The specs/reviewer-reports/ folder was recreated, and the reviewer was instructed to write reports there, not in the chat. The roadmap was rewritten by artifacts with an explicit reference to specs/validation/ as the evidence repository.
Result: A month later, the number of contract violations (grain changes, PII leaks, SLA changes without confirmation) dropped to almost zero. The new agent began following the short rules of AGENTS.md and referring to specs/ for details. The review time for a single PR shortened because verification facts lived in specs/validation/, rather than being scattered across the chat and comments. For the first time, the team was able to verify the setup readiness criterion: a new agent after /clear understood the project without a chat recap.
Lessons learned: A long AGENTS.md turns into an encyclopedia that nobody reads
Behavioral rules and product descriptions must live in different files
Verification facts should be in specs/validation/, not in PR comments
A short AGENTS.md increases the likelihood that the agent and the human will use it
A reviewer report is a separate artifact (specs/reviewer-reports/), not a chat message
Related concepts: AGENTS.md as a behavioral file
Separation of AGENTS.md and specs
Layers of meaning, verification, and review
Roadmap by artifacts
Project memory vs chat memory
Study tips: Do not try to fill specs/ with content right away — at the setup stage you only need empty containers and headers
Before every change in the project, ask yourself: "If I do /clear now, will a new agent understand where to find this decision?" If the answer is "no" — the decision must live in a file, not in the chat
Keep AGENTS.md short: better five short rules than twelve pages of instructions
Build the roadmap by artifacts and verification facts, not by "layers for the sake of layers"
Use the Qwen query from the chapter as a template for scaffold auditing: the first query is read-only, the second is only creating empty directories
Separate agent behavior (AGENTS.md) from product description (specs/); mixing creates bloated files that nobody reads
Check the setup readiness criterion: a new agent after /clear should understand the mission, stack, marts, and how to prove readiness without a chat recap
Do the setup before the first SQL model; otherwise the source of truth will appear in comments and agent responses, and it will be hard to restore
Avoid broad queries like "assemble a lakehouse" — at the setup stage you need containers for decisions, not the decisions themselves
Fix every significant decision in specs/ immediately after it is made, not "when there is time"
Additional resources: Dbt documentation — project structure: https://docs.getdbt.com/docs/build/projects — official guide to dbt project structure, dbt_project.yml and profiles.yml files
Dbt labs — model contracts: https://docs.getdbt.com/docs/collaborate/govern/model-contracts — description of model contracts, fixing grain and field composition
Anthropic — claude documentation: https://docs.anthropic.com — recommendations for formulating behavioral rules for AI agents
Conventional commits: https://www.conventionalcommits.org — commit formatting standard, useful for fixing scaffold setup stages
Markdown guide: https://www.markdownguide.org — reference for markup of mission.md, tech-stack.md, roadmap.md, and AGENTS.md
Git book — recording changes to the repository: https://git-scm.com/book — basics of working with Git, necessary for fixing the scaffold and its versioning
Summary: The chapter "Part 5. Initial Project Setup" teaches transferring decision memory from the chat into repository files. The minimal scaffold is specs/ (mission.md, tech-stack.md, roadmap.md and subdirectories models/, validation/, reviewer-reports/), models/ (staging, intermediate, marts), tests/, AGENTS.md, dbt_project.yml, and profiles.yml. AGENTS.md sets short behavioral rules: what to read before a change, what not to disclose, what not to change without confirmation, which command to run, and how to report with verification facts. The specifications of mission, stack, roadmap, model contracts, and acceptance facts live in specs/, not in AGENTS.md. It is better to keep the roadmap by artifacts and verification facts, so that it is clear that "the model is written" does not equal "the product is ready." A Qwen query with a ban on changes is used for scaffold auditing; after the list of gaps, the agent is instructed to create only the missing directories and empty templates, without writing SQL. A typical mistake is too broad a query like "assemble a lakehouse," which leads the agent to discuss the stack in the chat, while not a single structural unit appears in the repository. Setup is finished when, after /clear, a new agent understands the mission, stack, marts, and how to prove readiness without a chat recap: the folders start holding decisions.