Working on a new feature for PulseBoard
Most uptime tools send weekly reports full of numbers:
• Uptime %
• Response times
• Incident count
Useful... but they still leave you asking:
"What actually happened this week?"
Instead of another statistics email, I'm experimenting with something different.
Every week, users will receive an email with a single button:
🤖 Generate AI Weekly Summary
One click generates an AI-powered analysis of the week's monitoring data, highlighting trends, anomalies, reliability insights, and the events that actually mattered.
The idea is simple: spend less time reading metrics and more time understanding your infrastructure.
Still designing the experience, but I think weekly reports should explain—not just report.
Would you click a button like that instead of reading a page full of metrics?
RedMarker
Get your evenings back from manually checking maths homework.
Comments (11)
Md_haseeb, the shift from reporting to explaining is the right instinct. I ran into the same design question building FounderFlow, a page full of metrics is technically complete but nobody wants to be the one connecting the dots every Monday morning. What helped for us was tagging what changed and why it matters instead of just showing that a number moved. Are you thinking about ranking the anomalies by how much they actually affect uptime, or listing everything and letting the person reading it figure out what matters most?
Great point. I think listing everything would just recreate the same problem in another format.
The direction I'm exploring is having AI rank events by impact instead of only reporting changes.
For example:
- what changed
- why it matters
- confidence level
- whether the user should take action
The goal is not another weekly dashboard, but helping users understand what actually deserves attention.
Great experiment, Haseeb! One thought on the UX: instead of the button saying "Generate AI Weekly Summary" (which means the user has to wait for it to be created), what if the report is already pre-generated and the email just says "Your weekly summary is ready"? One click, no waiting. Could also work as an in-app notification for users who are already in the dashboard.
Olga — you're spot on about pre-generating to remove friction. I actually considered that approach first.
But here's why I'm leaning toward on-demand generation instead-
My approach is inspired from The YouTube Bell Model.
YouTube doesn't notify everyone about every new video. They only notify users who have explicitly clicked the bell icon — because those users have shown clear intent. Everyone else? They can find the video when they visit the channel.
My thinking is the same:
Pre-generating for everyone = burning LLM calls on users who might never open the email
On-demand generation = only serious users will click, and those users actually want the summary
It's not just about cost savings — it's about respecting user intent. If someone clicks "Generate AI Summary," I know they actually care about what happened this week. If they don't, I'm not wasting resources on something they'll ignore.
There's also a second benefit I didn't mention earlier: this approach naturally helps me identify serious users on the platform. The users who click the button are the ones who genuinely care about their monitoring data. Those are the users I should focus on — for feedback, testimonials, and future feature development.
So instead of a loading spinner, I'm thinking:
like something in this way
"Your summary is being generated. and its just a fetch call to LLM i know their are some tradeoffs but in long run it can be helpful in two ways
1) reducing LLM calls
2) Calculating serious users on my platform
i know for now in growing stage i can implement the pre generated template but in long run i think I might have to change my approach.
Curious what you think — does the YouTube model make sense in this context?
↳ Replying to Md_haseeb
The YouTube bell analogy totally makes sense, Haseeb. And I had to make the same choices for one of my products: how to make people come back to the app without wasting AI tokens on those who won't. I actually set up a similar mechanic. Users can sign up to receive generated results at the day and time convenient for them. So only people who actively want it get it. Same idea, different implementation. I think your approach of using the click as a signal to identify serious users is smart. You're getting two things from one action: the summary delivery and the user intent data.
Md_haseeb, ranking by impact and confidence level is the stronger design, it turns the report into a decision tool instead of a summary. The action recommendation is the part most tools skip, and that is where the real time saving lives. Olga's pre-generated angle is worth testing too, removing the wait removes a reason to close the email before reading it. Curious how you plan to calibrate confidence level, on fixed severity rules or something that learns from what users actually click.
I like the idea of evolving it over time.
Initially, I'm leaning toward evidence-based confidence rather than learning from clicks. For example, confidence would increase when multiple monitoring signals point to the same conclusion (incident history, response time patterns, monitor health, deployment timing if available, etc.), and decrease when the evidence is ambiguous.
I think user interactions are still valuable, but I'd rather use them to improve the product experience than directly influence the AI's confidence. I don't want a frequently clicked explanation to automatically become "more correct."
The long-term goal is exactly what you mentioned: make the weekly report a decision tool, not just an AI-generated summary.
↳ Replying to Md_haseeb
Md_haseeb, evidence based confidence over click based learning is the safer default, especially early. Letting clicks influence correctness would let a popular but wrong explanation look more trustworthy over time, which is hard to walk back once people trust the label. Using interaction data to improve the product instead of the confidence score itself keeps the incentives cleaner. If two signals disagree, incident history says one thing and deployment timing says another, does the system pick the stronger evidence source, or drop to a lower confidence tier until a person looks at it?
↳ Replying to Stacy Wycoff
I’d drop to a lower confidence tier when the evidence genuinely conflicts rather than forcing a conclusion.
The plan is to consider the reliability and relevance of each evidence source, but if incident history and deployment timing point in different directions without enough evidence to resolve the conflict, Vigil should surface both signals and explicitly state the uncertainty.
I’d rather have it say “the cause is unclear” than turn a weak correlation into a confident explanation. As I add more deployment, GitHub, and infrastructure context, the goal is to make those evidence relationships stronger without hiding ambiguity.
Goal is simple connect more and less vigil has to guess....
Md_haseeb, dropping to a lower tier and surfacing both signals instead of forcing a single story is the more honest choice, even if it reads as less satisfying in the moment. FounderFlow ran into the same fork with contested insights, pick the stronger source or admit the system is not sure yet. We landed on your answer too, a real conflict should show up as Needs Review, not get quietly resolved behind the scenes. As you add more evidence sources, does the resolution logic get more complex, or does having more signals to triangulate against actually make disagreements rarer?
Md_haseeb, this whole thread has been one of the sharper product conversations I have had here. The needs review approach you landed on for conflicting evidence is close to where FounderFlow ended up on the same problem. Since you have already helped me think through that harder, I want to ask for something in return. Would you be open to taking a real look at FounderFlow and telling me honestly where the confidence logic breaks down from your side? You clearly think about this the right way, and that kind of honest read is worth more to me than a generic demo call.
Sign in to comment or upvote.