Image source: Rawpixel — Data Center collection (royalty-free)
For two decades, robots.txt was the only doorman your website needed. Then ChatGPT arrived, and the front door turned into a hallway with a dozen new entrances, each one belonging to an AI crawler with its own agenda. Some want to train a model on your content. Many want to summarize your page inside a chat answer. Some want both. And almost none of them existed when your current robots.txt was written. llms.txt and robots.txt for B2B sites.
In 2026, the rules have changed again. There’s a new file in the mix, llms.txt, and a much longer list of bots to think about. If you run a B2B site and you haven’t reviewed your crawling configuration in the last year, you’re almost certainly leaving visibility, revenue, or both on the table.
Here’s the field guide every site owner should have on the desk.
What robots.txt Actually Does (and Doesn’t)
Robots.txt is a plain-text file that lives at the root of your website and tells well-behaved crawlers where they can and can’t go. It’s been around since 1994 and is still the foundation of crawl control. The file works by matching a bot’s name to specific allow or disallow rules for different parts of your site. So when a crawler shows up, the first thing it checks is whether you’ve given it permission to be there.
This system is good at a few important things. It can keep entire bots out of your training pipeline, focus crawl budget on the pages you actually want indexed, and protect staging, admin, and gated content paths from being scraped. What it can’t do is force compliance it’s a request, not a wall, which means bad-faith bots will ignore it entirely. It also can’t tell AI which of your pages are the most important, and it has no influence on how your brand gets summarized inside an AI answer.
That last gap is exactly what the new file was created to fill.
What llms.txt Is, and What It Isn’t
Llms.txt is a proposed standard written in plain Markdown, also placed at your site’s root. Think of it as a curated menu of your best, AI-friendly content — a way to say to a language model, “If you’re going to read about us, read these pages first.” Instead of trying to control where bots go, it points them toward what matters most: your core service pages, your most accurate company description, your case studies, your documentation.
The key thing to understand is that llms.txt does not block anything. It’s a navigation aid, not an access policy. Robots.txt is the bouncer at the door. Llms.txt is the concierge handing out a guided tour. They do different jobs, and a modern B2B site needs both.
It’s also worth being honest about the early data. A recent OtterlyAI study found that AI agents still mostly crawl standard web pages and don’t lean heavily on llms.txt yet. That said, publishing one is cheap insurance, adoption is accelerating, and the brands that have one in place will be ready when it does become a primary signal.
The Bots You Actually Need to Know
Not every AI crawler does the same thing. Knowing which ones do what is the difference between making smart decisions and blocking the bots that send you traffic.
GPTBot is OpenAI’s training crawler. When it visits your site, the content it reads can end up shaping future versions of ChatGPT. Blocking it stops your material from training a model that might one day quote your competitors instead of you.
OAI-SearchBot is a different OpenAI crawler, this one powers the live search results that appear inside ChatGPT. If you block it, you remove yourself from those answers entirely. That’s a significant visibility loss in 2026.
ChatGPT-User is the bot that fetches a page in real time when a ChatGPT user clicks or asks about a specific link. It’s on-demand, not bulk crawling.
ClaudeBot is Anthropic’s crawler for Claude. It plays a similar role to GPTBot but feeds the Claude ecosystem.
Google-Extended is Google’s opt-out signal specifically for Gemini and Vertex AI training. Importantly, it does not affect your regular Google Search rankings — blocking it only removes you from Gemini’s training data.
PerplexityBot powers Perplexity’s answer engine, and for many B2B sites it has quietly become one of the better sources of referral traffic. Blocking it usually costs more than it protects.
CCBot belongs to Common Crawl, an open dataset that many AI labs use as a starting point. Blocking CCBot has downstream effects, because anyone training on Common Crawl will no longer see your pages.
Bytespider is ByteDance’s crawler, known for being aggressive and for having a mixed reputation when it comes to respecting site rules.
For a deeper bot-by-bot breakdown, Contently’s AI crawler guide is a useful reference.
The Business Tradeoff: Block or Allow?
This is where strategy beats reflex. Many marketing teams blocked every AI bot back in 2023 to “protect their content,” then quietly opened them back up when they realized AI search was actually driving qualified traffic. The decision isn’t one-size-fits-all, and it shouldn’t be a one-time choice.
There are real reasons to allow AI bots. Your brand gets cited inside AI answers, which is increasingly the new top of the funnel. Referral traffic from Perplexity, ChatGPT, and Microsoft Copilot is growing every quarter. And more buyers are starting their product research inside a language model now, not on Google.
On the other side, there are real reasons to block them too. If you sell proprietary research, paid reports, or licensed data, training crawlers represent an obvious risk. If your content is your product, like a publisher, an education company, or a paywalled SaaS knowledge base, letting AI vacuum it up undercuts your business model. And some companies would simply rather have AI quote their competitors than train on their own intellectual property.
For most B2B sites in 2026, the right answer lands in the middle: allow the answer bots, block the training bots. Let Perplexity and OAI-SearchBot read you so you show up in real-time answers, but tell GPTBot, ClaudeBot, and CCBot to stay out of your training data. You get the visibility without giving away the IP.
What to Configure on Your Site
Once you’ve made your bot-by-bot decisions, the work itself is straightforward. Your developer or marketing team can update your robots.txt to allow the search and answer bots you want and disallow the training crawlers you don’t. The file should also continue to protect things like your admin paths, staging environments, and any sections of the site that aren’t meant for public crawling.
Then publish a llms.txt file at the root of your site. Keep it simple: a short company description, links to your most important pages, and grouped sections for resources, case studies, and documentation. Don’t try to point AI at every page, point it at the ones you’d hand a journalist if they were writing a story about you. That’s the standard to aim for.
After both files are live, verify them by simply loading the URLs in a browser. If they appear, the crawlers will find them.
Image source: MDN Web Docs — Dealing with files (royalty-free, CC license)
A 30-Minute Quarterly Checklist
Crawling configuration used to be a one-and-done task you handed off to a developer. In 2026, it needs to be a quarterly marketing decision. Every quarter, audit your current robots.txt and confirm it still reflects your stance, because most sites haven’t touched theirs in two years. Decide your position on each major bot using the breakdown above. Update your robots.txt to allow the answer bots and block the training crawlers. Publish or refresh your llms.txt so it points AI to your most important, evergreen pages. Test both URLs in a browser to confirm they load. And then put a recurring reminder on the calendar, because new bots are launching faster than the standards committees can keep up with them.
The Bottom Line
The brands showing up inside AI answers aren’t doing magic. They’re doing two simple things: keeping a clean robots.txt that lets the right bots in, and publishing a llms.txt that hands those bots a guided tour through their best content. Everyone else is still treating AI crawling like a 2019 SEO problem, and quietly losing share of voice while they wait.
Spend thirty minutes on both this week, and you’ll be ahead of 90% of your competitors before the next quarter starts.
About the Author
Isaac Miranda is the owner of T.R.O. Agency (since 2010) and a digital marketing specialist focused on human-first creative, video, content creation, social media, SEO, Generative Engine Optimization (GEO), and website development. He helps brands grow visibility and trust through clear messaging, strong storytelling, and consistent execution.
For all of your AI marketing needs, contact troagency.com or call 1-800-983-1213.