White paper
Data quality in a medallion architecture
Where to check, what to check, and why the silver layer carries most of the weight.
Summary
- Quality checks belong in every layer of a medallion architecture, and each layer answers a different question.
- Bronze checks confirm that data arrived. Silver checks confirm that it is correct. Gold checks confirm that it is fit for the decision it supports.
- Most of the effort should go into silver. It is the first place data has enough structure to be tested for meaning, and the last place a fix reaches every downstream use.
The three layers, briefly
A medallion architecture organizes a lakehouse or warehouse into three layers. Bronze holds data as it arrived from source systems, with little or no change. Silver holds the same data cleaned, typed, deduplicated and conformed to shared definitions. Gold holds the tables and aggregates built for specific business uses, such as reports, dashboards and models.
The appeal is that each layer has one job, and data quality should follow the same division. A single block of checks at the end of the pipeline can show that a number is wrong. It cannot show where the number went wrong, and it cannot stop it reaching anyone.
A different question at each layer
| Layer | The question | Typical checks | When a check fails |
|---|---|---|---|
| Bronze | Did the data arrive, complete and in the expected shape? | Freshness, row and file counts against the source, schema changes, duplicate deliveries | Alert the pipeline owner and hold the load. Leave the data as received. |
| Silver | Is the data correct and consistent? | Unique keys, valid references between tables, allowed values and ranges, required fields, cross-column rules, row counts reconciled to bronze | Quarantine the failing rows or stop promotion, depending on severity. Route the issue to the data owner. |
| Gold | Is the result fit for the decision it supports? | Totals reconciled to silver and to systems of record, complete reporting periods, consistent metric definitions, shifts in distribution | Hold publication and tell the people who use the product. |
Why silver carries the weight
It is the first place data can be tested for meaning
Bronze data is raw by design. Types are loose, records from different sources have not been matched, and the same customer may appear three times. Rules about business meaning, such as "every order belongs to a known customer" or "a contract cannot end before it starts", need typed columns and resolved keys before they can be evaluated. Silver is where those first exist.
A fix in silver reaches every consumer
Gold tables are many and specific. Silver tables are few and shared. A duplicate removed or a code standardized in silver is corrected for every report, model and extract built on it. The same problem handled in gold has to be handled again in each product, and the versions drift apart.
Bronze should stay faithful to the source
The value of bronze is that it records what was actually received. That makes it possible to reprocess history when a rule changes, and to show an auditor what a source system sent. Repairing data in bronze destroys that record. Bronze checks should detect and report, and correction belongs one layer up.
Gold is too late and too summarized
By the time data reaches gold it has usually been joined and aggregated. A total can look reasonable while hiding thousands of bad rows that happen to offset each other, and the row-level detail needed to find the cause is gone. Problems found at this stage are also found by the business, often in a meeting.
What still belongs in bronze and gold
The other two layers still need checks of their own. Bronze checks are cheap and catch the most common failure of all, which is data that never arrived or arrived twice. They also give the earliest warning, before any transformation has run.
Gold checks are the last line before a decision is made. They are the only place to confirm that a figure agrees with the system of record, or with the number published last month.
Practices that make the checks useful
- Set a severity for every rule. Some failures should stop a load. Others should be recorded and reviewed. Deciding in advance prevents silent corruption, and it prevents pipelines that halt for trivia.
- Quarantine failing rows. Rows that fail a silver rule go to a holding table with the reason attached, so they can be corrected and replayed instead of being lost.
- Reconcile between layers. Compare counts and key totals from bronze to silver and from silver to gold on every run. An unexplained difference is the cheapest signal available.
- Store results as data. Keep every check result with its date, so quality can be trended and reported, and so there is evidence of whether it is improving.
- Give every dataset an owner. A failed check with no named person behind it becomes background noise within weeks.
- Monitor for what nobody anticipated. Rules cover known expectations. Baselines for volume, freshness and distribution catch the failures no one wrote a rule for.
- Keep lineage. When a gold figure is questioned, lineage turns a day of investigation into a few minutes.
Common mistakes
- Cleaning data in bronze, which removes the record of what the source sent.
- Putting all the checks in gold, where problems are found last and fixed many times over.
- Rules that only write to a log nobody reads.
- Thresholds set once and never revisited as volumes grow.
- Testing full reloads carefully while incremental loads run unchecked.
Where to start
- List the silver tables that feed the most gold products, and start with five to ten of them.
- For each one, name an owner and write down its keys, required fields and valid values.
- Add freshness, volume and schema checks on the bronze tables behind them.
- Turn the written expectations into silver rules, each with a severity.
- Add reconciliation between the layers.
- Review the results weekly for the first month and adjust the thresholds.
- Extend to the next group of tables.
How Datamite can help
Datamite Consulting designs and implements data quality across warehouse and lakehouse platforms, from a first assessment through to the rules, monitoring and ownership that keep working after handover. To talk through your own architecture, write to sales@datamiteconsulting.com.