Data quality & provenance

The binding was done. The graph built. The agent answered in a clean, confident sentence, and it was wrong.

Here is why. The freezers table had 4,100 rows. About twelve percent had no max-temp at all, just a blank the loader left null. Dozens more, from sensors caught mid-reboot, read −999, the sentinel value (a fake stand-in number) a sensor writes when it has no real reading to report. Every one of those rows became a node anyway, because nothing checked them on the way in. So when the agent averaged max-temp to find which freezers looked healthy, the −999s pulled the average down. It reported the freezers running much colder, and so much healthier, than they really were.

Nobody caught it for a week. And when someone finally asked "where did that number come from," there was no answer. The node was just a node. It no longer knew which row it came from.

A confident answer built on a bad row is still a bad answer. It is worse, actually, because it sounds exactly like a good one.

An answer is only as good as its worst row

You cannot average your way out of a −999. Aggregates do not soften bad data, they hide it. A single bad row disappears into a clean-looking number, and that number looks just as trustworthy as the good ones. That is the trap: the answer looks fine because the bad row is now invisible.

So the fix cannot be "check the answer." By then it is too late, the bad row is already inside. The fix is to check each row before it gets in, against a rule that says what a valid max-temp even is: it has to be there, it has to be a number, and it has to fall in a range a real freezer could reach. A blank fails the first test. A −999 fails the third. Neither one gets to be a node.

Notice where the rule lives. Not in the freezers pipeline, and separately in the shipments pipeline, and again in whatever notebook someone wrote last quarter. It lives once, on the concept, on max-temp itself. Every source that binds to that property inherits the same contract. Change the range in one place and every future load is checked against the new one.

Declare the rule once, not in ten pipelines

This is the part that stops quality from slipping over time. When each pipeline carries its own copy of "max-temp should be sensible," the copies drift apart. One team writes the range check, another forgets, a third writes it a little differently. Six months later you have three definitions of a valid temperature and no idea which rows passed which one.

Put the rule on the concept and there is one definition, and only one. The rule is not a step in a pipeline you might skip. It is a property of what a Freezer is. A freezer whose max-temp is −999 is not a freezer with bad data, it is not a valid freezer at all, and the model can say so before the row ever becomes a node.

Turn on the contract one rule at a time, and watch the same batch sort itself.

Run it with every rule off first. All eight rows pass, −999 and the blank included, and that is the whole problem in one screen: with no contract, everything is believed. Turn on required and the blank drops out. Turn on the range and the −999 and the stuck 214 drop out too. You did not clean the data. You declared what clean means, and the batch sorted itself against it.

Every fact should carry a return address

Catching the bad row is half the job. The other half is being able to answer "where did this come from" for the rows that passed. A trusted node is not enough. You need a node that can point back to the exact row it was built from, and an edge that can point back to the exact foreign key it came from.

That pointer is provenance. Some people call it lineage. Same idea: the fact keeps a record of where it came from.

Provenance is what turns "the agent said so" into "the agent said so, and here is the row." When an answer looks wrong, you do not argue about it, you trace it back to the source and look. And when the source is fixed, the fact updates, because it was never a copy in the first place, it was always a live pointer to that row. A fact with no provenance cannot be checked, cannot be corrected, and cannot be trusted any longer than the moment someone believes it.

Binding-level quality, plus lineage on every fact

In Microsoft Fabric IQ the ontology layer is where quality rules are meant to live, on the concept rather than in the pipelines that feed it. The lab shows this clearly at the validate-publish step: you declare that a property is required, numeric, and within a range, and the checks run over your real bound rows before you publish. So you find the null max-temp and the −999 while you are still building, not a week after the data agent gave a wrong answer built on them.

The constraints sit at the concept layer, not in the pipelines that feed it. That is the same idea you just built by hand, now in Fabric's terms: quality is a property of the concept, inherited by every source that binds to it, checked once against the data rather than rebuilt for every load.

Provenance is the other half, and Fabric IQ is designed to keep it. As a preview capability as of July 2026 the exact lineage shown may change, but the goal is clear: keep a path from each graph fact back to the OneLake rows and keys it came from, so an answer the data agent walks can be traced to its source. When the data agent answers by walking the graph, that path is also the trail of receipts: this node, from that row, this edge, from that foreign key. The semantic model gives you the shape to query; the ontology and its graph aim to give you facts you can both trust and trace. The pattern is set: declare quality where the meaning lives, and keep lineage on the facts the graph carries.

The one trap

Do not mistake a clean-looking answer for a true one. A number can be perfectly formatted and stated with full confidence and still be built on rows that never should have counted. The two questions that protect you are boring, and you cannot skip either one: does this value pass the rule for a valid fact, and can I say which row it came from. If either answer is no, you do not have a fact yet, you have a guess that only looks like a fact.

Next: some properties are not one value at all. A freezer does not have a temperature, it has a temperature every minute, a stream of readings across time, and binding that is its own kind of problem. That is time-series binding.