AI Guides · Part 4 of 5

What AI gets wrong (and how to catch it)

The failures are real and specific. So are the habits that catch them. Trust it the right amount.

Read time
5 minutes
Level
Total beginner
Mood
Calm, promise
Panic required
None

Every guide in this series says “read it before you send it.” This one explains why. The failures are real, they’re specific, and once you’ve seen the pattern you’ll catch them without thinking about it.

Trust it like a talented new hire. A lot, and never blindly.

Part 01

What making things up looks like

The technical word is “hallucination,” but here’s the practical version: sometimes the tool fills a gap in its knowledge with something that sounds right. A statistic with a decimal point. A regulation with a plausible number. A book that doesn’t exist, with a convincing title.

The dangerous part isn’t the wrongness. It’s the delivery. Wrong answers arrive in the same confident, well-organized tone as right ones. There’s no stammer, no “I think,” no tell.

You ask“What permits do I need to sell food at farmers markets in my county?”

What can happenFour permits listed. Three are right. One was renamed two years ago, and the fee quoted is from a different state. All four sound equally official.

Part 02

Why it happens (the 60-second version)

Remember guide 1: these tools work by predicting what words should come next. When they know the answer, the most likely next words are the true ones. When they don’t, the most likely-sounding words still come out, just as fluently. The machine doesn’t get that nervous feeling a person gets when they’re bluffing. It doesn’t know it’s bluffing.

Newer tools are better at saying “I’m not sure,” and most can now search the web and show sources. That helps a lot. It moves the job from “hope it’s right” to “click the link,” which is a job you can actually do.

Part 03

The catching kit

Four habits, each under thirty seconds:

  • Ask for sources, then click oneOne click separates research from rumor. Dead or vague links mean the claim is a guess.
  • Make it argue against itself“What’s the strongest case that this is wrong?” is a shockingly effective question.
  • Ask twice, worded differentlyIf the two answers disagree, you found a soft spot. Slow down there.
  • Say where the stakes areTell it “this involves real money, be careful and cite sources.” It genuinely changes the answer quality.

Part 04

Where the real risk lives

You don’t need to double-check everything. You need to know the three neighborhoods where errors cost money:

Numbers it did math on.

Laws, taxes, and anything with a deadline.

Facts about your own business it was never told.

Everything else (drafts, summaries, brainstorms, tone) is low-stakes by nature, because you’re the editor before anything leaves the building. That’s why the drafting work in guide 3 is safe to start with today.

Part 05

Is my stuff private?

The other trust question, and it deserves a straight answer. Depending on the tool and your settings, chats can be used to improve the product. Every major tool has a privacy or data-controls page where you can turn that off, and the paid business tiers are generally stricter by default.

The working rule: treat a chat like email to a competent outside vendor. Fine for drafts, summaries, and most day-to-day work. Wrong for passwords, full client lists, and anything under an NDA. If you wouldn’t email it to a contractor, don’t paste it into a chat.

Field noteThe settings page for this takes two minutes to find and set once. Search the tool’s name plus “data controls.”

Where to start

The 30-second habit

  1. Before anything AI-drafted leaves your hands, read it once as the recipient would.
  2. For any fact or number that matters, ask “how do you know?” and click one source.
  3. When the stakes are money, law, or health, a human expert stays in the loop. Always.

That’s the whole discipline. It costs half a minute per task and it’s the difference between using these tools like a pro and getting burned once and swearing off the whole thing.

Spot the pattern

The six things to remember

Fluent isn’t true

Wrong answers arrive in perfect grammar. Tone proves nothing.

Click a source

One click separates research from rumor.

Make it argue

“What’s the strongest case that I’m wrong?”

Ask twice

Different wording. If the answers differ, slow down.

Guard secrets

No passwords, client lists, or NDA material in chats.

The stakes rule

Money, law, health: a human expert stays in the loop.

So no, you can’t trust it blindly. You also couldn’t trust the best employee you ever hired blindly on their first week. Same posture: give real work, check what matters, and the trust gets earned fast.

One guide left, and it’s the one closest to home: what all of this can do for your website.

So what are you working on?

Tell me what you're building and where it's stuck. I'll tell you honestly whether I'm the right person for it.