Ask an AI agent to summarize an article. The page loads. The words get read. The publisher never sees a visit.
That gap in the ledger used to be invisible. Now it shows up in server logs, and publishers have stopped pretending it doesn’t matter.
Three things collided at once, and the people building agents that read the web for a living are the ones absorbing the fallout.
Why Publishers Are Closing the Gates in 2026
Automated traffic now makes up a large and rising share of requests to most sites. Every one of those requests costs the operator money, whether or not anything gets used afterward.
Major AI companies started publishing the user agent strings their crawlers use. That single change made selective blocking possible for the first time — a site could finally tell one bot from another instead of blocking everything or nothing.
Then large publishers began signing licensing deals. That move settled a question that had been open for years: the data has a price, and taking it without paying is now a negotiating position rather than an accepted norm.
The cost pressure runs both directions. Teams running distributed agents already route requests through infrastructure like ProxyWing to spread load across regions and avoid hammering a single IP, since a lot of the web renders differently depending on which market a request comes from. That’s a sensible way to be a lighter, more polite visitor. It is not a way around a block someone put there on purpose, and treating it as one turns a manageable technical problem into a legal one.
What a Robots.txt File Actually Says Now
A robots.txt file used to answer one question: crawler, yes or no. It now has to answer three, and the standard was never built for the third one.
| Visitor type | What it does | Typical publisher response |
|---|---|---|
| Search crawler | Indexes pages, sends readers back | Usually allowed |
| Training crawler | Collects pages to train a model, no visit generated | Increasingly blocked |
| User-initiated agent | Fetches one page because a person asked a question | Split — no consensus |
That third row is where most of the friction lives.
Is a User’s AI Agent the Same as a Training Crawler?
Arguably not, but nothing standardizes the distinction, and publishers answer it differently from each other.
A crawler pulling millions of pages to build a training set and an agent fetching a single page because someone asked it a direct question are different activities with different economics. From the server’s point of view, they can look identical.
Some publishers treat a user-initiated fetch like a browser visit and let it through. Others block anything that isn’t a human holding a mouse, reasoning that a summary still replaces the click. Both positions hold up under scrutiny. Until something changes that, an agent builder is working across a patchwork where identical code gets welcomed on one domain and refused on the next.
How Should AI Agent Builders Handle Blocked Content?
A few habits separate agents that stay welcome from ones that get shut out fast.
- Use the official API first, every time. A documented endpoint with a rate limit survives a redesign. A scraper built against page structure doesn’t.
- Read robots.txt at runtime and honor it. Treat it as the clearest statement of intent a site can give, not a formality to route around.
- Identify the agent in the user agent string, with a contact address. Operators who can email a builder usually do that before they block one.
- Cache aggressively. Never fetch the same resource twice — it cuts cost and footprint in the same move.
- Back off on a 429 instead of retrying. Retry storms are what convert a tolerated agent into a permanently blocked one.
- Keep personal data out of the pipeline entirely. That’s both the ethical line and the one carrying regulatory weight.
Modern agent frameworks increasingly package this kind of judgment directly into how the agent runs. Reusable, folder-based agent skill definitions built around a SKILL.md file can encode exactly these fetch-and-respect rules once, so every agent built on top of that skill inherits the same manners instead of each builder reinventing them from scratch.
Rate limiting arrives quietly at scale, too. Requests from a single address get throttled, and the failure mode isn’t an error — it’s a partial result set that a summarization or comparison pipeline treats as a finding instead of a fault.
Where Is Web Access for AI Agents Heading?
Two paths are visible from here.
One is licensing at the top: the largest publishers strike deals, and everyone smaller gets nothing. That concentrates the training corpus in the hands of whoever can afford the contracts.
The other is a technical settlement — machine-readable permissions and per-request compensation that would let a small publisher say yes on their own terms instead of only being able to say no.
Neither is close to finished. In the meantime, the open web is getting less open specifically to automated readers, right as the most interesting software being built is an automated reader.
Build as though every site depended on will eventually say no, because some of them will. Partnerships and official APIs where the data actually matters. Caching so a temporary block doesn’t become an outage. Honest attribution so that when a publisher checks their logs and finds an agent, the conversation starts from a reasonable place.
The teams that get shut out are rarely the ones taking the most data. They’re the ones nobody could identify and nobody could reach.
FAQs
Q. Is scraping for AI training legal?
It depends on jurisdiction, on what gets taken, and on whether personal data or a contract term is involved. Treat it as contested, not settled.
Q. Does robots.txt legally bind an agent?
It isn’t a law by itself, but ignoring it is the first thing anyone points to in a dispute, and it can strengthen a claim that access was unauthorized.
Q. Is a user-initiated agent fetch different from a crawler?
Arguably yes, and no standard expresses that difference cleanly yet. Publishers currently answer it differently from one another.
Q. What’s the most common technical mistake agent builders make?
Retrying on a 429 instead of backing off. It turns a soft limit into a hard block within hours.
