Why I Don't Trust AI to Read Deprecation Notices (And Neither Should You)

Last month, I was integrating a third-party API at work and asked Claude for a quick example of how to handle authentication. It gave me code using an endpoint that looked official, clean, well-documented. I almost shipped it. Then I checked the vendor's changelog and realized it had been deprecated two years ago. The code would have failed in production. That moment stuck with me, not because Claude is bad, but because I'd almost trusted it completely on something that mattered.

I stumbled across this benchmark recently and it crystallized something I've been feeling: we're treating AI models like they have stable, reliable knowledge about the state of the world. They don't. Not even close.

What Actually Happened in This Benchmark

The experiment was elegantly simple: ask seven different AI models about the deprecation status of Cloudflare API features. Each model got asked the same thirteen questions about features in different states, fully removed, deprecated but still working, or current. Straightforward yes/no territory, right?

Wrong. Six out of seven models contradicted themselves when asked the same question three times. The consistency scores ranged from 0.636 to 0.855. One model gave answers that varied so wildly that the noise floor was nearly as big as the actual measurements. We're not talking about edge cases here, we're talking about basic factual questions about what does and doesn't work.

But here's what really caught my attention: the two questions with the worst consistency weren't about completely removed features. They were about partial deprecation-endpoints that still exist but where specific operations no longer work. One example: you can still update DNS records through an API, but you can't change the record type anymore. The rest works fine.

Models would contradict themselves on these all day long. Sometimes they'd say it works, sometimes they'd say it doesn't. And this matters because there's a huge difference between saying "this is gone" and saying "this mostly works but not for everything." The first sends you searching for alternatives you don't need. The second gives you code that crashes in production.

The Deeper Problem: Non-Determinism Isn't a Bug, It's the Feature

Here's what really bothered me: the benchmark was supposed to run with deterministic results. The Kaggle SDK has a temperature parameter set to 0. But in the actual implementation, the temperature parameter gets dropped for models that don't explicitly support it. So vendors' default temperature settings kick in. Every run is slightly different. Nobody's in control.

I've been burned by non-determinism before in my own code. Usually I notice it immediately and fix it. But with these models, we're not fixing it, we're accepting it as normal. We're benchmarking in conditions where we can't actually measure anything reliably.

The researcher ran each model three times and scored based on the most common answer. That's practical, sure, but it's also a band-aid over something systemic. When your best model still gives different answers 31% of the time, something's genuinely broken.

What This Means When You're Actually Shipping Code

I use AI models every day in my development work. They're fast, they're often right, and they're convenient. But I've stopped treating them like documentation. I treat them like a junior developer who's smart but hasn't read the manual yet.

Here's my workflow now:

• Get the AI suggestion
• Check the official docs (yes, actually read them)
• If it's about API compatibility or deprecation, verify it's current
• Test it locally before committing

It's slower. It's also how I avoid shipping broken code into production.

The real issue isn't that models are bad at this. It's that we're asking models for consistency on knowledge that requires real-time updates and careful semantic distinctions. A feature being "deprecated but live" is a fuzzy state. Models hate fuzzy. They want to collapse it into categories. They'll do it differently every time you ask.

My Ask for the Broader Community

If you're building tools that rely on AI for factual accuracy, especially anything touching API documentation, deprecation schedules, or version compatibility, I'd strongly encourage you to think about this benchmark. Don't blindly trust the output. Build verification layers. Make your system aware that the AI might contradict itself tomorrow.

And if you're running your own benchmarks on these models? Duplicate at least one test case, every time. Run it twice. That noise floor matters more than you think.

What's your experience with AI-generated code and outdated information? Have you caught this happening in production?

Source: This post was inspired by "Six of seven models contradicted themselves on the same deprecation question" by Dev.to. Read the original article