I spent an embarrassing amount of time last year debugging why my blog posts weren't appearing in Google search results. I had a beautiful sitemap. I had a robots.txt file. I thought I was doing everything right. Turns out, I was conflating three completely different concerns into one mental model, which meant every decision I made was slightly wrong.
The real wake-up call came when I started seeing AI crawlers hitting my site aggressively, and I had no idea how to tell them what to actually read versus what was just internal cruft. That's when I realized I'd never really understood the distinction between these three files, and honestly, most articles online blur the boundaries in ways that make the problem worse, not better.
The Three Responsibilities (They're Not Interchangeable)
I think of this differently now. robots.txt is a bouncer. It sits at /robots.txt and decides who gets through the door. You tell it: "Googlebot, come in. GPTBot, stay out. Everyone else, you can browse but not the /admin/ section." It's a voluntary policy, malicious crawlers ignore it, but the legitimate ones (Google, Bing, the major AI companies) respect it because it's standardized in RFC 9309.
The key thing I was missing: robots.txt doesn't remove anything from search results. It just prevents crawling. If Google already indexed a page, blocking it in robots.txt doesn't un-index it. You need noindex meta tags for that. And here's the frustrating part, robots.txt is publicly readable. Anyone can see exactly what you're trying to hide, so it's useless for actual security. Authentication is what protects private content.
sitemap.xml is the map. It's a structured list of URLs you think are worth crawling, with optional metadata about when they were last updated and how often they change. It lives at /sitemap.xml and is a suggestion to search engines, not a command. Google will index whatever it wants based on quality, internal linking, and dozens of other signals. I used to think putting a URL in the sitemap guaranteed indexing. It doesn't. A page with zero internal links pointing to it is a weak candidate no matter what the sitemap says.
llms.txt is the brief. This is the new kid on the block (2024), and it's explicitly designed for AI assistants, not search engines. It's a single markdown file where you curate the most important pages with clean descriptions. Instead of making an LLM crawl your entire site and synthesize information, you just hand it a reading list. This actually solves a real problem I've had, when someone asks ChatGPT about my work, it sometimes pulls outdated blog posts instead of my current documentation.
What This Means in Practice
Here's where I've changed my approach. I now maintain these files as separate concerns, each with its own update cycle.
My robots.txt is a policy statement. I use it to block internal tools, API endpoints, and admin panels. I explicitly block certain AI training crawlers because I want to control how my content is used for training. But I'm realistic, this isn't airtight. It's politeness enforcement, not security.
My sitemap.xml is automatically generated. I'm tired of manually maintaining it and lying about lastmod dates (which search engines can tell you're doing). I have a script that runs weekly, pulls all published content with accurate timestamps, and regenerates the file. I don't list every variation or paginated version, just the canonical URLs.
My llms.txt is where I actually think strategically about how I present myself to AI. I have sections for documentation, recent blog posts, and my services. I link to markdown versions where possible because LLMs ingest plain text better than HTML. I keep it short enough that an AI with token limitations can actually read the whole thing.
The Mistake Everyone Makes
The biggest trap I see (and fell into) is trying to use these files as a unified content policy. People ask, "Should I disallow this in robots.txt or omit it from the sitemap?" The answer is usually: they're solving different problems. A page can be crawlable (allowed in robots.txt), discoverable (in sitemap.xml), and still not recommended for AI assistants (absent from llms.txt).
Also, blocking a crawler in robots.txt doesn't block it perfectly. OpenAI's GPTBot is one thing, but ChatGPT can still cite your site through its browsing feature and other AI products use different bot names. If you want real control over AI usage, you need licensing or legal terms, robots.txt is just a signal.
Next Steps
My advice: audit your three files separately. Check that your robots.txt policy matches your actual security needs (spoiler: it probably doesn't, use auth instead). Verify your sitemap is generated automatically and not manually maintained. And if you care about AI representation, actually write an llms.txt file instead of hoping crawlers figure out what matters on your site.
What's your setup? Are you maintaining these files intentionally, or have they been copy-pasted into your site and forgotten?
Source: This post was inspired by "robots.txt vs llms.txt vs sitemap.xml: what each is for" by Dev.to. Read the original article
