We shipped a way to tell whether our AI's advice actually did anything

I'm building an AI chief of staff for founder-operators — people running real businesses where every decision still routes through one person. This week we shipped something I'd been putting off for months, because I suspected the answer would be embarrassing.

We had no way to know whether our recommendations helped anyone.

Generating advice is trivial. Proving that someone saw it, acted on it, corrected it, or got a result is hard, and we'd been shipping the easy half.

Three things went out.

A permanent record of every meaningful recommendation, so it doesn't vanish when the underlying item gets reclassified. What we said, when we said it, whether it was viewed, what happened after.

Separate tracking for exposure and action. This is the one that surprised me. We'd been counting recommendations as ignored that nobody had ever scrolled to. Once we split those apart, a large chunk of "users ignore us" turned into "users never saw it" — a placement problem, not a quality problem. We were close to redesigning the wrong thing.

And a resolution engine that reads recorded events, like replies, status changes, and corrections, instead of asking the model whether the user followed its own advice. Asking the model means grading a recommendation using the same text that produced it.

What I'm still stuck on is the attribution window. If someone acts three weeks later, that wasn't us. Same day, probably was. I've picked a number somewhere in between and written it down, mostly so we're consistently wrong rather than randomly wrong.

If you've built measurement into an AI product, how are you drawing that line? And does anyone else separate "never saw it" from "saw it and passed"? I suspect a lot of products are quietly conflating those two the way we were.