White paper
Rules and observability: catching the data problems nobody predicted
Why rules alone leave gaps, what monitoring adds, and how to keep alerts useful.
Summary
- Data quality rules confirm the expectations someone thought to write down. They miss the failures nobody anticipated.
- Observability fills that gap. It learns how each dataset normally behaves and flags departures in freshness, volume, schema and distribution.
- Neither helps unless alerts are ranked, grouped by cause and sent to a named owner.
What rules do well
A data quality rule states an expectation and tests it. An order must have a customer. A date of birth cannot be in the future. A status must be one of five values.
Rules are precise and easy to explain to the business, and they map directly to the familiar quality dimensions of completeness, validity, uniqueness and consistency. When a rule fails, the failing rows can be listed, which makes the problem concrete and fixable.
Where rules stop
A rule exists only if someone anticipated the problem, and most serious data incidents are of a kind nobody anticipated. A source system changes a default value. A job runs successfully but loads a tenth of its usual rows. A new product code appears that every rule accepts as valid and no report knows how to group. Each record passes each rule, and the dataset as a whole is wrong.
Rules also need upkeep. An estate with thousands of tables cannot be covered one rule at a time, so coverage settles on the tables people already worry about and leaves the rest untested.
What observability adds
Observability treats each dataset as something with normal behavior and watches for departures from it. It needs no rule for each table, because the expectations are learned from history. Five signals do most of the work:
- Freshness. Has the data updated when it normally does?
- Volume. Did roughly the expected number of rows arrive?
- Schema. Have columns been added, removed or changed in type?
- Distribution. Have the values shifted, such as a jump in empty fields, a new category, or an average that has moved?
- Relationships. Do the tables that usually agree with each other still agree?
A useful baseline accounts for rhythm. Many datasets are quiet at weekends and heavy at month end. A monitor that compares a Sunday with a Wednesday raises a false alarm every week, so baselines should be learned separately for periods that behave differently.
The two work together
| Problem | Caught by a rule | Caught by monitoring |
|---|---|---|
| A required field is empty | Yes | Only if the share of empty values changes |
| A value is outside its allowed range | Yes | Sometimes |
| A business rule across two columns is broken | Yes | No |
| A load delivers a tenth of its normal volume | No | Yes |
| A table stops updating | No | Yes |
| A column changes type upstream | No | Yes |
| A new, valid-looking category appears | No | Yes |
The pattern is consistent. Rules catch violations of known business logic at the level of the row. Monitoring catches changes in behavior at the level of the dataset. A program with only one of them has a predictable blind spot.
Keeping alerts useful
These programs usually fail through sheer volume of alerts. A team that receives two hundred a day stops reading them. Four practices keep the number manageable:
- Rank by priority. Combine the size of the departure with the importance of the dataset, so the critical few are at the top.
- Group by cause. One failed upstream load can trip checks on every table below it. Lineage lets those be presented as one incident with one root cause.
- Route to an owner. Every alert should reach the person responsible for that dataset.
- Let people correct it. When an alert turns out to be expected behavior, marking it as such should adjust the baseline.
Making quality visible
Results are most useful when they are summarized into a health score for each dataset and rolled up by business domain. A score gives a business owner a single figure to track, shows whether quality is improving, and makes it possible to agree targets. The detail stays available underneath for the people who fix things.
It also helps to show results where people already look. When quality status appears in the data catalog next to a dataset's owner, description and lineage, someone deciding whether to use a table can see at once whether to trust it.
Where to start
- Pick the ten to twenty datasets that matter most to reporting or operations.
- Turn on freshness, volume and schema monitoring for all of them. It needs no rule writing and gives results within days.
- Let the baselines settle for two to four weeks before tuning them.
- Add rules for the business constraints each owner can name.
- Assign an owner and a priority to each dataset, and route alerts accordingly.
- Publish a simple scorecard and review it monthly.
How Datamite can help
Datamite Consulting sets up data quality and observability on the platforms clients already run, and brings its own tool, EDQ, where nothing is in place yet. To talk through your own situation, write to sales@datamiteconsulting.com.