Prompted, Not Proven – Part 3: Decide What Looks Right First
Prompted, Not Proven – Part 3: Decide What Looks Right First
This is Part 3 of the Prompted, Not Proven series in The Statement on what’s real, and what isn’t, in AI for finance content. If you missed the first two, start here: Part 1: Most “AI for Finance” Content Has Never Met a Real Client and Part 2: The Last 30% Is the Job.
In Part 1, I offered a simple test for every AI tip, template, and model being presented at conferences or on social media: has the person sharing it ever put it in front of a real client? In Part 2, I argued that as AI does more of the work, the remaining 20–30% becomes the job we retain: deciding what deserves to be automated, interpreting what AI outputs, and being confident enough to put our name – and responsibility – on the results.
Part 1 also ended with a second question that I want to come back to:
“If AI produces extremely similar outputs for everyone who asks, what separates one professional from another?”
My short answer: what we expect before we review what AI gives us.
More specifically, it’s a habit I’ve started practicing deliberately and now encourage the analysts, managers, and VPs I work with to adopt. Most of us can look at an AI output and ask,
“Is this right?”
I’ve stopped starting there. Instead, before I open anything, I form a basic hypothesis. I ask myself what I’d EXPECT to see before it arrives. Then I validate what AI gives me against that expectation.
It sounds like a small change in thinking, but I don’t think it is. And the reasons why have far more to do with how our brains work than with how AI works.
I'm Not a Psychologist, But I Had a Hunch
I feel like almost every dinner conversation I’m now having with friends, family, and colleagues somehow ends up with AI being a topic of discussion. On the one hand, we’re mesmerized by how good AI is with some of the challenges we pose to it. On the other hand, we’re dumbfounded at just how wrong some AI results are. Because of this back-and-forth, it’s hard to know what to believe or not.
When I started doing this, I assumed what I was fighting was confirmation bias. If something hands us a result that looks convincing, we tend to trust it without validating it thoroughly.
It turns out I was close, but not quite right. And the more I read, the more I realized there isn’t just one bias at work when we review AI output. There are at least four; they’re all well-documented, and AI can trigger every one of them at the same time.
I’ll walk through each one, because I think understanding them is what makes the case for independent human judgment better than any of the warnings I continue to give you.
Bias #1: Automation Bias
Automation bias is our tendency to trust what an automated system tells us and to discount information that contradicts it. Researchers have studied it for decades with pilots, air traffic controllers, and doctors using decision-support software. The pattern is remarkably consistent: once the system offers an answer, people stop looking as hard. It’s the same reason why finance professionals tend to trust results that come out of FP&A software more than they do out of manual Excel.
Automation bias shows up in two ways, and I think finance professionals are exposed to both.
The first is acting on something the system got wrong. Let me offer a very basic example: Say AI writes your variance commentary and attributes a revenue miss to lower volume. It sounds rational, so it goes into the board deck. But the real driver was price, and now leadership is spending 20 minutes debating the wrong problem.
The second is less obvious, and in my experience, more dangerous. It’s missing what the system never flagged in the first place. In Part 2 of this series, I described a tool flagging a negative $48,000 in revenue that the Controller cleared in seconds. That’s the easy case. What concerns me more is the $48,000 the tool didn’t flag. When we rely on AI to tell us where to look, we may gradually stop looking everywhere else.
This is where independent judgment matters so much. If I’ve already formed a view of what the numbers should look like, AI’s answer becomes one input I check against mine. It stops being the final word.
Bias #2: Anchoring
In a well-known 1974 experiment, Amos Tversky and Daniel Kahneman spun a wheel of fortune in front of participants. The wheel was rigged to stop on either 10 or 65. They then asked participants to estimate the percentage of African countries in the United Nations.
The wheel had nothing to do with the question. It didn’t matter. People who saw 10 gave a median estimate of 25%. People who saw 65 gave a median estimate of 45%. A random number moved their judgment by 20 points.
If you’ve ever sat through a budget negotiation, you already know this instinctively. Whoever puts the first number on the table usually has a big say in where the conversation ends up. Chris Voss brought this idea to a wide audience in his bestseller Never Split the Difference, which draws on his years as an FBI hostage negotiator.
Now think about what happens when AI goes first. It tells you Q3 revenue will come in at $4.9M. You do what feels like an independent sanity check and land at $4.7M. Close enough, right? Not so fast. Your “independent” check started at $4.9M and worked its way down. Had you started from scratch, you might have landed at $4.2M.
Once you’ve seen an anchor, you can’t un-see it. The only reliable defense I know of is to set my own number before AI gives me one.
Bias #3: Processing Fluency
Our brains treat ease as a signal of truth. When information is easy to read and process, we’re more inclined to believe it. Just look at social media, where we see short, easy-to-digest concepts. Or YouTube videos that oversimplify complex matters, so simple that even kids think they understand.
In one study, people rated identical statements as more likely to be true when they were printed in colors that were easier to read. In another, people judged rhyming sayings as more accurate than the same idea without the rhyme. “Woes unite foes” was rated as more true than “Woes unite enemies.” Same meaning. Better rhythm. Sounds crazy, right? It’s not.
If a rhyme can do that, what do you think a perfectly formatted three-statement model does to us? This past week at the Association for Financial Professionals in Las Vegas, I presented a three-statement model created entirely by Claude. It was perfectly formatted and looked right on every page, but the numbers were wrong.
This is exactly what I described in Part 1. AI outputs look finished. The formatting is clean, the formulas calculate, no conditional formatting flashes red, and the commentary is written in confident, complete sentences. AI rarely sounds unsure, even when it should be.
Now compare that to a first-year analyst’s draft. A few hard-codes, inconsistent font sizes, and maybe a note that says “need to confirm with ops.” Many of us instinctively scrutinize the messy version and relax around the polished one.
But that’s backwards. The analyst told you exactly where the uncertainty was. The AI didn’t.
When I’ve decided in advance what the answer should look like, the polish stops being persuasive. I’m comparing numbers to numbers, and the formatting doesn’t get to win me over.
There’s a flip side to this. GoDaddy CEO Aman Bhutani recently told Business Insider that when he can tell quickly that a document was written by AI, he doesn’t read it. His tells: language that’s a little too perfect, documents that run long, and short writing that’s suspiciously punchy.
I agree with the bigger point he’s making. Writing is how many of us figure out what we actually think, and modeling works the same way. Building the drivers and the forecast is how you learn the business.
But look at what he’s reacting to: the polish. Fluency pushes many of us to trust a document because it reads well, and Bhutani rejects a document for that very same reason. Whether we trust it or toss it, the formatting is still seen to matter.
In finance, someone has to review the forecast. And “this looks like AI” tells me nothing about whether $4.9M is right. A clean model AI built can be correct…or it could be wrong. A messy one an analyst built by hand can be correct…or it could be wrong.
What settles it is comparing the numbers to what I expected before I opened the file.
Bias #4: Hindsight Bias
This is the one that makes the other three so hard to catch.
In the 1970s, psychologist Baruch Fischhoff showed that once people learn an outcome, they overestimate how likely they were to have predicted it. Knowing the answer subtly rewrites our memory of what we expected. We genuinely believe we saw it coming. And we experience this ALL the time.
Here’s how that plays out with AI. You open the forecast, see $4.9M, and think, “Yes, that’s about what I figured.” Did you figure that? Or did seeing $4.9M convince you that you had?
Honestly, you can’t know. Once the output is in front of you, your true prior expectation is gone, and no amount of discipline brings it back.
That’s why I’ve become such a stickler about one thing: the expectation has to be written down or vocalized before I look. I want a number, a range, and the reasons behind it. If it only lives in my head, hindsight will clear it up for me.
When all Four Show Up Together
Any single one of these biases can trip up a good professional. But AI stacks them. When we review output-first, here’s the sequence:
- Automation bias makes us trust it.
- Anchoring sets the starting point for our check.
- Fluency makes it feel right.
- Hindsight convinces us we expected it all along.
By the time we finally ask “is this right?”, all four have already shown up. What feels like a thoughtful, careful, human-in-the-loop review may be closer to a rubber stamp.
Accountants Figured This Out Decades Ago
As a CPA, I probably should’ve acknowledged this sooner. But, then again, I was never an auditor. Here’s the analogy I’m getting at:
The accounting profession already has a formal version of this habit. It’s built into the auditing standards for substantive analytical procedures. When an auditor tests a revenue or expense balance this way, the process looks like this:
- Develop an independent expectation of what the balance should be, using data and relationships the auditor trusts.
- Decide in advance how large a difference is acceptable.
- Compare that expectation to the recorded amount.
- Investigate anything outside the threshold and get it explained.
Notice the order. The expectation and the threshold come before the auditor looks at the number being tested. That’s deliberate. The profession learned long ago that reviewing a number after you’ve already seen it isn’t truly independent.
Which raises the key question. I suspect many of us may be giving AI outputs less skepticism than an auditor gives a client’s general ledger that doesn’t jive. We’d never accept “it looked reasonable” as audit evidence. So why are we accepting it from ourselves?
Here's What This Looks Like in Practice
Say you’ve asked Claude or Copilot to build a Q3 revenue forecast. Before you open it, you spend five minutes writing down what you’d expect.
Last year’s Q3 revenue was $3.9M. Pricing went up 4%, which gets you to about $4.06M. You picked up one new customer worth roughly $250K for the quarter and lost one worth about $100K. So your expectation is about $4.2M, and you’ll accept anything within 5%, or roughly $4.0M to $4.4M.
Now you open the output.
Scenario A: AI says $4.9M. That’s about 17% above your expectation and well outside your range. So you dig in, and you find that a $680K annual renewal for your largest customer was booked in both July and September. Without your expectation, $4.9M would have looked perfectly reasonable. The model looked good, the commentary was confident, and red conditional formatting never showed up.
Scenario B: AI says $4.25M. That’s inside your range, so you’re done, right?
Not quite. When you check the drivers, you find AI got there by applying a 9% price increase and ignoring both customer changes entirely. That’s two wrong assumptions producing one right-looking total. You only caught it because you wrote down why you expected $4.2M, and AI’s reasons didn’t match yours.
That’s the power of setting expectations first.
Where My Original Hunch Comes Back In
At the beginning of this, I shared how I thought this was about confirmation bias. Confirmation bias does belong in this conversation, just not where I first put it.
The moment we form an expectation, we’ve exposed ourselves to a different bias. Confirmation bias is our tendency to notice evidence that supports what we already believe and explain away evidence that doesn’t. If I treat my expectation as something to defend, I’ll find reasons AI must be wrong every time it disagrees with me.
That’s just a different way of not thinking.
So I try to hold my expectations loosely, forming a hypothesis around what I think I should see. Sometimes AI will be right and I’ll be wrong. I’d call either a good outcome, because I’ve still learned something I didn’t know, and my next expectation will be sharper for it.
So What Separates One Professional From Another?
In Part 1, I shared that I’d seen two models posted publicly by different people that looked almost identical to the ones I had Claude build for me. Same layout, same formatting, even the same fictional company and vendor names (just so you know, they’re Meridian and Apex – so if you see those names, they’re likely made by Claude).
I believe that’s where many routine finance deliverables are heading. The outputs are going to converge. We’re already seeing it in the way Anthropic, OpenAI, and Microsoft are enabling skills, plugins, and connectors. They’re becoming extremely similar.
What won’t converge are our expectations, because they come from sources AI can’t easily combine: your actual data, your conversations with the business, the customer you know won’t renew next quarter, and the years you’ve spent watching models fail to represent reality because of nuance.
GoDaddy’s Bhutani made a related point in that same interview I referred to above: AI only knows what’s already on the internet, and its massive trove of training data; but AI doesn’t know all that we know. That’s exactly why our expectations about results are worth writing down or vocalizing. They’re built from what AI hasn’t seen.
At conferences and workshops, I keep hearing that better prompts are the answer to better results. Yes, good prompts matter, but I don’t think they’ll separate anyone for long. The professionals who stand apart will be the ones who can explain why AI’s answer is off, because they’ve already formed and proved their own hypothesis. And when AI agrees with them, they can explain why. They are willing to be the stop-gap, preventing bad answers from going to the Director of FP&A, no matter how polished the AI version looks.
This takes skill. But it also takes a bit of courage.
When I started this series, I wanted to separate what’s real from what isn’t in AI for finance. Three issues later, I think it comes down to this: anyone can prompt and get good-looking results. But that doesn’t mean they’ll hold up in front of an audience of skeptics who know more than the machine.
Thank you for coming along with me on this one.
Questions for You
- “Before you open an AI-generated forecast, do you write down what you expect to see?”
- “Which of the four biases do you see most often in your own reviews?”






