Topic: Part 15. Gold layer: business entities and grain
Difficulty level: Medium
Estimated study time: 2-3 hours
Prerequisites: Basic understanding of Data Warehouse architecture (bronze/silver/gold layers)
Experience with dbt or similar data transformation tools
SQL knowledge at the level of writing complex queries and window functions
Understanding of data lineage and data contracts concepts
Familiarity with data quality and data testing principles
Learning objectives: Define and formulate the grain (granularity) of a data mart as a single public promise
Distinguish between the technical readiness of a model and its readiness as a gold product for the consumer
Apply mandatory acceptance facts (not_null, unique, absence of PII) for the gold layer
Compile a gold layer registry indicating the consumer, granularity, mandatory fields, and PII policy
Conduct a strict review of gold models with a focus on the contract, SLA, and quality evidence
Overview: The Gold layer is the final layer in the data architecture, where engineering work turns into a public promise to consumers. Unlike the staging and silver layers, where intermediate calculations and technical normalization are acceptable, a gold model must answer a specific business question with a fixed granularity. The central concept is grain — defining what one row in the mart means. If grain is not recorded or verified, the consumer will assume it on their own, which leads to meaning drift. A gold model must be narrow, have a explicitly named consumer, mandatory fields, a PII policy, and verification facts. Placing a model in the models/marts/ folder does not automatically make it a gold data product — it is a promise secured by a specification, a contract, and verification.
Key concepts: Gold layer: The final layer of data products, where the consumer sees stable marts rather than intermediate calculations. Gold starts with grain and a contract, not with a convenient SELECT. It is not the models/marts/ folder, but a promise to the consumer.
Grain (granularity): The answer to the question "what does one row mean" in a data mart. Examples: one row per customer_id, transaction_id, scope, or per day and channel. Changing the grain is a change to the data API, not cosmetics.
Public promise of a mart: A single-line description of who the mart promises what to. A good formulation: "One row per customer for portfolio analytics without direct PII". A bad one: "A useful customer mart with accounts, payments, risks, and everything that might be needed".
Mart consumer: An explicitly named user or team: customer and portfolio analytics, risk signal monitoring, consent status control. Different consumers should not accidentally receive the same grain.
Acceptance facts: A set of checks for a gold model: the primary key of the grain passes not_null and unique (if one row per key is assumed), mandatory indicators pass not_null, no direct PII, contract fields are present, lineage is readable.
PII policy: Rules for handling personal data in the gold layer. Any extra column with PII may end up in a report if the policy is not verified. The absence of direct PII is a mandatory readiness condition.
Mart meaning drift: The gradual expansion of a mart to become "richer": adding products, channels, segments, recent operations, risk flags, credit stages. Each expansion changes the question the table answers, even though the name remains the same.
Data contract: An agreement between a supplier and a consumer about the set of fields, their types, and semantics. The contract fields must be present in the gold model and verified by tests.
Gold layer review: A stricter process than a staging review. Questions: who is the mart promised to, how will it be used, what are the SLAs, PII policy, grain, and evidence. If the reviewer cannot understand the consumer, the model is not yet gold.
Mart customer 360, mart payment risk signals, mart open api consents: Examples of specific marts with different consumers and grains: mart_customer_360 — for customer and portfolio analytics (grain: customer_id), mart_payment_risk_signals — for risk signal monitoring, mart_open_api_consents — for consent control.
Important dates: Emergence of the medallion architecture concept: Approximately 2020-2021 — consolidation of bronze/silver/gold layer terminology in the industry
Evolution of the data contracts approach: 2022-2024 — formation of the practice of explicit contracts between data suppliers and consumers
Standardization of PII policies: GDPR (2018) and similar regulations as a catalyst for formalizing privacy policies in the gold layer
Practice exercises: Name: Formulating a single-line mart promise
Problem: For each mart in your project (assume there are mart_customer_360, mart_payment_risk_signals, mart_open_api_consents), write one public promise in a single line. Check whether the promise contains the consumer and grain. If not, rephrase it.
Solution: Step 1: Identify the consumer for each mart (customer analyst, risk team, consent control). Step 2: Formulate the grain (one row per customer_id, per transaction_id, per scope). Step 3: Write the promise: 'mart_customer_360: one row per customer for portfolio analytics without direct PII'. Step 4: Check that the wording does not contain commas and exceptions that would indicate an overly broad mart.
Complexity: beginner
Name: Compiling a gold layer registry
Problem: Create a gold layer registry for the project following the template: Consumer, Granularity, Mandatory fields, PII policy, Verification facts. Fill it in for three marts.
Solution: Step 1: Create a markdown file 'Gold layer registry.md'. Step 2: For mart_customer_360: Consumer — customer and portfolio analytics, Granularity — one row per customer_id, Mandatory fields — customer_id, segment, LTV, last_activity, PII policy — only hashed identifiers, no email/phone, Verification facts — unique+not_null on customer_id, not_null on LTV. Step 3: Repeat for mart_payment_risk_signals (one row per transaction_id, risk flags are mandatory) and mart_open_api_consents (one row per scope, consent status is mandatory).
Complexity: intermediate
Name: Identifying marts with unproven grain
Problem: Using the Qwen query from the material, analyze the specs/models and models/marts folders. Find marts where grain is not proven by a test or a manual fact from the reviewer.
Solution: Step 1: Open the Qwen query: 'Read specs/models and models/marts. For each mart, determine the grain, main verification facts, PII policy, and impact on the contract. Find marts where grain is not proven by a test or a manual fact from the reviewer. Do not modify files.' Step 2: Review each model in models/marts/. Step 3: For each, check: is there a schema.yml with unique/not_null tests on the key grain field? Is there an explicit indication of grain in the README or description? Step 4: Compile a list of marts without grain evidence and submit them for review to add tests or documentation.
Complexity: intermediate
Name: Diagnosing mart meaning drift
Problem: The mart mart_customer_360 originally contained one row per customer. Over time, the following were added to it: customer products, interaction channels, recent operations, risk flags, credit stage. Name the problem and propose a solution.
Solution: Step 1: Identify the problem — this is mart meaning drift. The grain effectively became 'customer × product × channel', although the name implies one row per customer. Step 2: The consumer does not understand what they are getting: a customer slice or a customer-product slice. Step 3: The solution is to split the mart: return mart_customer_360 to the grain 'one row per customer' (only customer attributes), create separate marts: mart_customer_products (one row per customer-product), mart_customer_risk_signals (risk flags per customer), mart_customer_interactions (channels and operations). Step 4: Record the public promise of each mart in a single line.
Complexity: advanced
Case studies: Name: Meaning drift in a fintech startup's customer mart
Scenario: A fintech startup created the mart mart_customer_360 for the portfolio analytics team. Initially, the mart contained one row per customer_id with the following fields: customer segment, LTV, date of last transaction, balance. Analysts used it for a weekly portfolio report.
Challenge: Over time, the team began adding new fields to the mart: a list of customer products (JSON array), communication channels, credit stage, risk flags, the last 5 operations. The grain effectively changed from 'one row per customer' to 'customer with all activity'. The name remained the same, but the meaning changed. Analysts began getting ambiguous results: in the report, LTV was calculated correctly in some cases and with duplicates in others due to row multiplication across products. Additionally, a column with customer emails accidentally got into the mart, violating the PII policy.
Solution: The team conducted a gold layer review following the principles from the material. They formulated public promises: mart_customer_360 — 'one row per customer for portfolio analytics without direct PII'. They created separate marts: mart_customer_products (grain: customer_id × product_id), mart_customer_risk_signals (grain: customer_id), mart_customer_recent_activity (grain: customer_id × operation_id). They added acceptance tests: unique + not_null on customer_id in all marts, a check for the absence of PII via a dbt test, contract tests on mandatory fields. They conducted a review focused on the questions: who is the consumer? what is the grain? what are the SLAs?
Result: After splitting the marts, analysts got predictable results in weekly reports. LTV drift stopped. The PII policy was formalized: gold marts retained only hashed identifiers and aggregated metrics. Report debugging time decreased by 60%. The team adopted the practice: every mart in models/marts/ must now have a single-line public promise in the README indicating the consumer and grain.
Lessons learned: Placing a model in models/marts/ does not automatically make it a gold data product
Expanding a mart to become 'richer' silently changes the grain and meaning — a discipline of separation is needed
Acceptance facts (unique, not_null, absence of PII) must be mandatory for the gold layer
A single-line public promise is the best way to prevent drift
Related concepts: Grain as a defense against meaning drift
Gold layer acceptance facts
PII policy in marts
Strict review of gold models
Public promise of a mart
Name: Implicit grain in a payment system risk signals mart
Scenario: A payment system created the mart mart_payment_risk_signals for monitoring by the risk team. The mart was assembled from several sources: antifraud scoring, behavioral factors, geolocation. The model was assembled without an explicit definition of grain — just a convenient JOIN of all factors.
Challenge: The risk team began receiving duplicate signals: the same payment appeared in the mart several times because different factor sources fired at different times with different timestamps. Monitoring showed false positives. In addition, the mart contained direct PII (customer name, email), which the risk team was not supposed to see per the security policy. The staging review passed without comments, but the gold review was not conducted strictly.
Solution: The team applied the approach from the material: explicitly defined the grain — 'one row per transaction_id with the current state of risk factors at the time of the latest assessment'. Wrote a single-line promise: 'mart_payment_risk_signals: one row per transaction with current risk flags for monitoring without direct PII'. Added acceptance tests: unique + not_null on transaction_id, not_null on risk_score and risk_flags, a test for the absence of PII columns via metadata. Removed direct PII, replacing it with a hashed customer_hash. Conducted a review of each mart with the questions: who is it promised to? what is the grain? what are the SLAs? what is the evidence?
Result: Signal duplication stopped. Monitoring became deterministic: one transaction — one record with the current state. The risk team stopped seeing direct PII, eliminating the compliance risk. A practice was introduced: before merging into main, every mart goes through a 6-point gold model readiness checklist (consumer, grain, contract, PII, verification facts, review report).
Lessons learned: Without an explicit grain, the consumer assumes it on their own — this leads to ambiguous results
Staging and gold reviews must have different strictness: in gold, they ask about the consumer, contract, SLA
The PII policy must be verified automatically, not rely on manual control
Acceptance facts (unique on the grain key) are a mandatory readiness condition
Related concepts: Grain as part of the public contract
Gold layer acceptance facts
PII policy
Strict review of gold models
Qwen query for mart analysis
Study tips: Start by formulating a public promise: take any mart from your project and try to describe it in a single line. If you cannot do it without commas, the mart is too broad.
Use the gold model readiness checklist as a filter: apply the 6 points from the material (consumer, grain, contract, PII, verification facts, review report) to every model in models/marts/.
Do not confuse the folder with the promise: being located in models/marts/ is a technical detail. Gold is a contract with the consumer, secured by a specification and verification.
Practice identifying drift: take a mart with a history of changes and trace how the grain has changed. This will help you recognize drift at early stages.
Use the Qwen query from the material as a template for automated analysis: it finds marts with unproven grain and saves time on manual review.
Review strictness should grow from staging to gold: in gold, ask about the consumer, contract, SLA, PII, grain, and evidence — not just about the technical correctness of SQL.
Adopt a habit: for every mart expansion, first check with the question — does this change the grain? If yes, it is not cosmetics, it is a change to the data API.
Split marts by grain: it is better to have three narrow marts with a clear grain than one broad one that tries to be everything at once.
Additional resources: Dbt documentation on tests: https://docs.getdbt.com/docs/build/data-tests — official documentation on schema and custom tests in dbt
Data contracts (bitol): https://bitol.io — open data contracts specification by the Andrew Jones team
The medallion architecture (databricks): https://www.databricks.com/glossary/medallion-architecture — description of the bronze/silver/gold architecture by Databricks
Data quality fundamentals (great expectations): https://greatexpectations.io — tools and practices for ensuring data quality
Dbt-utils package: https://github.com/dbt-labs/dbt-utils — a set of useful tests and macros, including PII checks
Book 'Fundamentals of Data Engineering' (Joe Reis, Matt Housley): Chapter on data layers and contracts — a modern view of DWH architecture
Article 'What is a data contract?' (Monte Carlo): https://www.montecarlodata.com/blog-data-contracts-explained — a practical explanation of data contracts
Summary: The Gold layer is not the models/marts/ folder, but a public promise to the consumer. The key concept is grain — defining what one row means. A ready gold model has: an explicitly named consumer, a recorded and verified grain, mandatory fields tied to the contract, a PII policy, acceptance facts (unique+not_null on the key, absence of direct PII, contract fields), and a review report separating facts from assumptions. The main defense against meaning drift is narrow marts with a single-line public promise. If the promise cannot be formulated without commas, the mart is too broad and must be split. The gold review must be stricter than the staging review: it must ask about the consumer, contract, SLA, PII, grain, and evidence. Use the Qwen query to find marts with unproven grain and compile a gold layer registry indicating the consumer, granularity, mandatory fields, and PII policy.