Topic: Part 9. Raw and Bronze Landing
Difficulty level: Medium
Estimated study time: 2-3 hours (theory + practice)
Prerequisites: Basic understanding of Data Lakehouse architecture (raw, bronze, silver, gold layers)
Experience with dbt or similar data transformation tools
Knowledge of SQL and data schema concepts
Understanding of lineage and metadata principles
Basic familiarity with Python (for generating synthetic data)
Learning objectives: Distinguish the responsibilities of the Raw and Bronze layers and articulate which transformations are permissible on each
Explain why data in the raw layer cannot be modified, deleted, or hidden without explicit business justification
Compose acceptance facts to verify the reproducibility of input landing
Describe the semantics of empty values and null through the source manifest rather than manual edits
Build an evidence chain raw → manifest → specification → staging → test to protect product decisions
Overview: Chapter 9 is devoted to the first two layers of the Lakehouse architecture — Raw and Bronze — which are responsible for reproducibility of data intake. In the educational example, raw is represented by CSV files generated by the script scripts/generate_data.py; in production, this can be a landing zone in object storage, a Kafka topic, a CDC table, or an external API. Despite the difference in physical implementation, the principle is one: at the first step, the meaning of the data cannot be changed. Raw answers the question "what arrived" and preserves the input as is — including empty values, PII, and contradictions. Bronze adds minimal technical normalization (file format, typing, load metadata, lineage), but does not make product decisions. The goal of the chapter is to teach the reader to distinguish between describing data and correcting it, understand that a "boring and honest" raw is insurance against future disputes about data provenance (especially critical in a banking context), and build a short evidence chain that allows any reviewer to trace the path from source to mart.
Key concepts: Raw layer: A landing zone that answers the question "what arrived". Preserves data in its original form: the same field names, the same order, the same empty values and null markers. It is forbidden to rename business fields, delete rows, hide PII, correct values without evidence, and perform any aggregations. The raw layer is intentionally "boring": its job is to be an honest mirror of the source, not a filter or normalizer.
Bronze layer: An intermediate technical layer that adds minimal normalization without product logic: file format (Parquet, Delta), basic field typing, load metadata (time, source, batch_id), column presence checks, and basic lineage. Bronze should not make business decisions — for example, deciding that an empty revoked_at means active consent. This decision is moved into the staging specification.
Input reproducibility: A key property of Lakehouse: the ability to regenerate raw data and walk through the chain of transformations up to the mart with the same result. Achieved through versioning of generation scripts, source manifests, header checks, and freezing synthetic data.
Raw/bronze acceptance facts: The minimal set of checks confirming that the input is correct: source files exist and are accessible, headers match the expected source schema from the domain map, synthetic data generation is reproducible, the semantics of empty values and null are described in the manifest, and PII is not lost before the processing policy is approved.
Semantics of empty values: A rule for interpreting empty cells and NULL values, fixed in the source manifest rather than in code or data. For example, an empty revoked_at may mean "consent is active" or "consent was revoked before the field was introduced" — the choice is made by the business and recorded explicitly, not implied by the convenience of passing a test.
Lineage and load metadata: Technical information added at the Bronze level: source name, path to the file or topic, load timestamp, batch identifier, schema version. Allows tracing the origin of any row in the mart back to the original event.
Source manifest: A document that fixes the expected structure and semantics of raw data: a list of columns with types, a description of empty values and null markers, business owner, update policy, and open questions on disputed fields. Links the data source to the rest of the architecture.
Sdd data (specification-driven data): An approach in which data decisions are fixed in specifications and manifests, rather than in transformation code or, moreover, in the data itself. Raw/Bronze is where SDD protects the future conversation with a reviewer or regulator.
Typical "green test" mistake: The temptation to fix the source CSV so that dbt build passes. Externally the error disappears, but with it disappears the evidence that the source sent an ambiguous value. The correct path: freeze the input, describe the interpretation in the staging contract, add a test.
Domain map: A map of the subject area describing entities, relationships between them, data owners, and expected sources. Used as a reference when reconciling headers of raw files.
Important dates: Lakehouse evolution stage (approximately 2019-2020): The emergence of the medallion architecture terminology (Bronze/Silver/Gold) at Databricks and the consolidation of the idea of layered data processing with explicit separation of technical normalization and business logic.
Adoption of dbt in sdd practice (since 2021): The rise in popularity of dbt as a tool that made staging and mart layers "speaking SQL," increased attention to model specifications and tests, which raised the importance of an honest raw layer.
Regulatory changes on pii (gdpr since 2018, updates to federal law-152 in the rf): Tightening of requirements for processing personal data, due to which the rule "do not hide PII before approving the policy" became especially critical in banking and financial sectors.
Practice exercises: Name: Finding a field with ambiguous emptiness semantics
Problem: Open the catalog book3/examples/bank-lakehouse/raw and study all CSV files. Find one field in which both empty strings and explicit null markers occur (or a value that looks like 'unknown', 'n/a', etc.). For the found field, describe: (1) how it should be preserved as a raw fact; (2) where and how technical typing will be performed in Bronze; (3) in which staging specification the business rule of interpretation will appear; (4) which dbt test should confirm the correctness of the interpretation. Hint: start with the file consents.csv and the field revoked_at.
Solution: Step 1. Open raw/consents.csv and find rows where revoked_at is empty. Step 2. Fix the raw fact: an empty cell is preserved as empty (or as NULL after reading with Pandas/duckdb, depending on the parser), without substituting default values. Step 3. In the Bronze model stg_consents (or equivalent), perform type casting to DATE or TIMESTAMP, without interpreting emptiness. Step 4. In the staging specification (e.g., in schema.yml or in _models/_docs), add a rule: empty revoked_at in staging logic is treated as "consent is active at the time of export". Step 5. Write a dbt test that checks that after transformation for all active consents, the value is_active = true corresponds to the expected share (e.g., not_null or accepted_values). Step 6. Fix the decision in a note to the test explaining why exactly so.
Complexity: intermediate
Name: Reconciling headers with the domain map
Problem: Using Qwen (or manually), execute the query from the "Qwen query" section: check the raw sources in book3/examples/bank-lakehouse/raw, compare the headers with the expected sources from the domain map, and compile a list of source facts and open questions. Do not change dbt models. The goal is to fix discrepancies before they lead to errors in staging.
Solution: Step 1. Open each CSV in the raw catalog and write down the actual headers. Step 2. Open the domain map (e.g., docs/domain_map.md or a similar file in the repository) and write down the expected headers for each source. Step 3. Build a discrepancy table: which field is missing, which is renamed, which has an unexpected type. Step 4. For each discrepancy, classify: (a) real source error → open question to the owner; (b) terminology discrepancy → clarify the manifest; (c) the domain model changed → update the map. Step 5. Compose a markdown document "Source facts and open questions" that will become the entry point for the reviewer.
Complexity: intermediate
Name: Anti-pattern: fixing CSV instead of fixing the contract
Problem: Simulate the situation: you run dbt build, and the model stg_consents fails with the error "NULL value not allowed for column revoked_at_iso". The raw file does have rows with empty revoked_at. Simulate two solution paths — wrong and right — and describe the consequences of each after 1, 6, and 12 months.
Solution: Wrong path: open the CSV, replace empty strings with a placeholder date (e.g., 1900-01-01), resave the file, rerun dbt build — it is green. After 1 month, no one remembers the substitution. After 6 months, an auditor arrives and asks why the system has 12% of consents with a revocation date in 1900. After 12 months, you cannot recover which clients actually revoked consent and which did not. Right path: freeze raw as is; in the staging model use safe_cast or try_cast, which turns emptiness into NULL; in the manifest describe that empty revoked_at is interpreted as "active consent"; add a dbt test not_null on the derived field is_active and a test on the consistency is_active ↔ revoked_at. After 1, 6, and 12 months, the evidence chain remains intact: raw → manifest → staging → test.
Complexity: intermediate
Name: Designing acceptance facts for a new source
Problem: The team is connecting a new source — an export of card operations from external processing. The source arrives as a parquet file in S3 daily. Design a set of acceptance facts for the raw and bronze layers that will be checked automatically (e.g., Great Expectations, dbt-source-freshness, or a custom Python script).
Solution: Raw acceptance facts: (1) the file exists at the expected path s3://bucket/raw/card_ops/yyyy/mm/dd/; (2) file size > 0; (3) headers match the manifest (card_id, txn_id, amount, currency, txn_ts, mcc); (4) checksum matches the manifest (protection against substitution); (5) no unexpected columns; (6) PII fields (card_id) are present and not masked. Bronze acceptance facts: (1) schema is cast to expected types (amount → DECIMAL, txn_ts → TIMESTAMP); (2) load metadata is added (loaded_at, source_batch_id, source_file_path); (3) raw and bronze row counts match (protection against losses during conversion); (4) freshness: data is no older than N hours; (5) lineage is recorded in the meta table. All checks are run before dbt build starts and block it on failure.
Complexity: advanced
Case studies: Name: Dispute over a client's consent half a year after export
Scenario: A retail bank receives a complaint from a client: "I revoked consent to the processing of personal data back in March, and they keep calling me with promotional offers in August." The legal department requests from the data team confirmation of the consent status for every day from March to August. The data comes from an external CRM as daily CSV exports that are placed in the raw layer and then transformed through dbt into the consents mart.
Challenge: At the initial load of the CSV file for March, the revoked_at field was partially empty (the source sent data in a transitional period when some systems were not yet transmitting the revocation date). To make the green dbt build pass, the team opened the file and replaced empty cells with the date 1900-01-01. Technically the data looked "clean," but the client who actually revoked consent in mid-March is now listed as active until August — because the "revocation" in the system occurred in 1900.
Solution: The data team's audit reconstructs the chain of events: finds in git history the commit in which the CSV file was resaved with the substituted date; checks the staging model and sees that the interpretation revoked_at = 1900-01-01 → is_active = false was not documented. The team redoes the process: (1) restores the original CSV from an S3 backup with versioning; (2) in the source manifest fixes that empty revoked_at is treated as "no data," not as "active"; (3) in staging logic adds an explicit rule: is_active = (revoked_at IS NULL AND consent_given_at IS NOT NULL); (4) adds a dbt test prohibiting the appearance of the date 1900-01-01 in production data; (5) reissues the consents mart and provides the lawyers with the correct history.
Result: The legal department receives the correct timeline: the client actually revoked consent on March 14, and from that date onward they should not have received marketing communications. The bank settles the dispute without a fine. The internal process is revised: any manual editing of raw files is prohibited, a check is added to CI that the git history of CSV is empty after the initial commit, and all interpretations of empty values go through manifest review.
Lessons learned: "Green test" is not proof of correctness; editing source data substitutes the signal and kills reproducibility
The semantics of empty values should live in the specification, not in code or in the data itself
In the banking context, a dispute arises not today, but in months — therefore the evidence chain raw → manifest → staging → test must be restorable at any moment
Versioning raw data (S3 versioning, DVC, git-lfs) is insurance that allows rolling back a "manual fix" and seeing what the source actually sent
Related concepts: Raw layer
Semantics of empty values
Source manifest
SDD Data
Typical "green test" mistake
Input reproducibility
Name: Connecting a Kafka topic as a new raw source
Scenario: The team is moving from batch CSV exports to streaming reception of payment events through Kafka. The topic payments.events is written to object storage through Kafka Connect in Avro format. The team must decide where the boundary between raw and bronze lies, and who is responsible for the Avro schema.
Challenge: New questions arise: (1) should parquet files in S3 generated by Kafka Connect be considered already bronze layer (because there is a schema and metadata there), or is it still raw? (2) is it necessary to keep the raw Avro byte stream for audit? (3) how to describe the semantics of empty values in streaming data, where "absence of event" is also a fact?
Solution: The team makes the following decision: the raw Avro snapshot of each batch is saved in S3 in the catalog raw/payments_avro/ for audit purposes — this is the true raw. Parquet files generated by Kafka Connect are placed in the catalog raw/payments/ and are considered a raw fact: they contain Kafka technical metadata (offset, partition, timestamp), but no business interpretation. The Bronze model payments_bronze performs only: (1) casting Avro schema to dbt types; (2) adding loaded_at, source_partition; (3) checking the presence of required fields (event_id, amount, currency); (4) a control check that the number of records in the Avro snapshot and in parquet matches. The semantics of empty values are fixed in the manifest: empty merchant_category means "MCC not defined by the source system," not "category is missing by business decision."
Result: The team gets two independent layers of evidence: Avro snapshots for deep audit and parquet for daily transformation. For any discrepancy between the mart and the source, it is possible to recover in hours what exactly came from Kafka. The costs of storing Avro snapshots are justified by the reduction in regulatory risk.
Lessons learned: The physical file format (Parquet, Avro, CSV) does not automatically determine the layer — what matters is the role of the data in the pipeline
Raw must be preserved even when downstream tools (Kafka Connect, Fivetran) already add technical metadata
In streaming data, "absence of event" is the same kind of fact as the event itself, and must be described in the manifest
Bronze should not take on responsibility for interpreting the source schema — this is the task of the staging specification
Related concepts: Bronze layer
Lineage and load metadata
Raw/Bronze acceptance facts
Input reproducibility
Domain map
Name: CDC table as raw: boundary between technical and product
Scenario: The team is moving to CDC (Change Data Capture) from PostgreSQL: Debezium writes changes to a topic, from there the data ends up in an Iceberg table raw.customers_cdc. The table contains all operations (INSERT, UPDATE, DELETE) with metadata __op, __ts_ms, __source_ts_ms. Analysts need only the current version of the client.
Challenge: Where should the rule "keep only the latest version of the client" live — in bronze or in staging? If in bronze, bronze starts making a product decision. If in staging, staging receives a stream of operations and must figure out CDC semantics. In addition, it is necessary to decide what to do with DELETE operations — treat them as "client deletion" or as "operation in the source system".
Solution: The team fixes: raw.customers_cdc stores all operations as they arrived, without filtering. This means that the table retains INSERT, UPDATE, DELETE, and even TOAST updates. The Bronze model customers_cdc_bronze performs only: (1) explicit type casting (especially for JSON fields in PostgreSQL); (2) renaming technical CDC fields into lineage fields (cdc_op, cdc_ts_ms); (3) adding loaded_at, source_db, source_table. All decisions about "latest version," "active client," "whether to apply DELETE" move into the staging specification stg_customers, where they are explicitly described and covered by tests. For example, a test checks that after applying CDC operations, the number of active clients matches the operational system.
Result: The team gets a "boring" raw that shows all the nuances of the source: for example, it turns out that UPDATE operations sometimes arrive earlier than INSERT (due to Kafka partition ordering), and this requires special logic in staging. Without the raw fact, this anomaly would be hidden by a "clean" bronze layer and would manifest itself only in disputes with the regulator about the correctness of balance recalculation.
Lessons learned: CDC data is always raw, because it reflects the internal workings of the source, not the product representation
"Latest version of an entity" is a staging decision that must be documented and covered by a test
Technical CDC metadata (__op, __ts_ms) is important to preserve as lineage, not discard at the bronze stage
The raw layer protects against loss of context when staging logic changes: if tomorrow the business changes the rule "what to consider an active client," the mart can be recalculated without losing the original operations
Related concepts: Raw layer
Bronze layer
Lineage and load metadata
SDD Data
Input reproducibility
Study tips: Before reading the chapter, open the catalog book3/examples/bank-lakehouse/raw and skim through all CSV files — this will help see real examples of empty values and contradictions discussed in the text.
Keep a separate notebook "Open questions about raw" where you write down all disputed fields as you read. This notebook will become the basis for the source manifests in your own project.
Say out loud the rule: "Raw answers the question WHAT arrived, staging answers WHAT IT MEANS". If in class you forgot the boundary — return to this phrase.
When studying cases, first try to propose the "wrong path" yourself, and then compare with the author's — this way the anti-pattern is more actively formed, and in production the hand will not reach to edit CSV.
Use the "reverse lineage" technique: from any field in the mart, go up to the raw file and check that every transformation is documented. If somewhere the chain breaks — this is a signal that there is hidden product logic in bronze or raw.
Formulate a personal rule: "Before editing a file, ask yourself — is this describing data or fixing data?" If fixing — stop and revisit the staging contract.
Make yourself a checklist of 5 raw acceptance facts and hang it next to the monitor — this turns abstract requirements into a daily habit when receiving new sources.
To memorize the raw/bronze difference, use the mnemonic: "Raw is the mirror, Bronze is the frame around the mirror". The frame can improve the view, but should not change the reflection.
Additional resources: Book 'fundamentals of data engineering' (joe reis, matt housley, o'reilly 2022): The chapter on storage and ingestion examines in detail why the first layer should be "boring" and how raw, bronze, and the principle of immutability are related.
Dbt documentation: 'about dbt project structure' and 'source freshness': A description of how to properly declare sources and check freshness in dbt — this is a formalization of raw layer acceptance facts.
Article 'the medallion architecture: a guide to bronze, silver, and gold layers' (databricks): The original description of the medallion architecture with emphasis on the boundaries between layers and the prohibition on business logic in bronze.
Materials 'great expectations' (or substitute soda core): Tools for formalizing raw/bronze acceptance facts as code, so that checks are versioned and reproducible.
Book 'designing data-intensive applications' (martin kleppmann, chapter on stream processing): A deep explanation of what an "event arrival fact" is and why it is important to separate it from interpretation.
Standard federal law-152 (rf) and gdpr (eu) on pii processing: The regulatory basis explaining why PII cannot be hidden in raw before the policy is approved — this is a legal, not just an engineering norm.
Dbt-utils package: source freshness and accepted values tests: Ready-made tests for formalizing part of the raw/bronze acceptance facts in a standard dbt project.
Qwen (or similar llm assistant): A tool for automating routine checking of raw file headers against the domain map — an example query is given in the chapter.
Summary: Chapter 9 "Raw and Bronze Landing" formulates three key principles. First, raw is a "boring and honest" layer that preserves the input in its original form: it does not rename fields, delete rows, hide PII, correct values, or aggregate. Its task is to be a mirror of the source, not a filter. Second, bronze adds minimal technical normalization (file format, typing, load metadata, lineage), but should not make product decisions — the interpretation of disputed values is moved into the staging specification. Third, input reproducibility is verified through acceptance facts: the existence of files, the correctness of headers, the reproducibility of generation, the described semantics of empty values, and the preservation of PII before the policy is approved. The main typical mistake is editing raw data for the sake of a "green test," which externally removes the problem but destroys the evidence chain and makes it impossible to recover the truth in months (which is critical in a banking context). After studying the chapter, the reader should be able to distinguish describing data from correcting it, design acceptance facts for new sources, and build a short recoverable chain raw → manifest → specification → staging → test.