Ask an AI browser agent to check one competitor’s price, and it will happily load a 2.6 MB page, run every script and maybe grab a screenshot. Then it passes a few hundred tokens of useful text to the model. The price itself sits in a sliver of HTML.
Teams building data feeds for LLMs budget the same way everyone else does, in pages instead of bytes. Then the first proxy invoice lands at three times the estimate. That’s usually when someone goes looking for rotating residential proxies unlimited bandwidth and hopes for a flat fee. Most rotating residential pools, including that one, are still sold per GB, though non-expiring traffic at least softens the month-end cliff. The cheaper fix usually sits upstream: fetch fewer bytes in the first place.
Why AI Scrapers Are the Heaviest Visitors on the Web
AI has changed who does the scraping. Wikipedia’s scraping entry notes that by the mid-2020s, scrapers run by LLM vendors had become a major source of web traffic. Retrieval pipelines, price-intelligence feeds and agent tools now hit the same pages that search crawlers used to.
Agents make it worse. A classic scraper requests one document and moves on. A browser agent renders the whole page, clicks, scrolls, and renders again. Research agents like Manus can chain dozens of those steps into a single task, so every “quick lookup” becomes a full page load, over and over.
The Page Count Is the Wrong Number
Run the numbers on a common job. Say a retailer wants competitor prices from 2 million product pages, checked daily. Those pages are heavy with zoomable galleries and widgets, so call it about 2 MB per fully loaded page. That’s 4 TB a day, or roughly 120 TB by month’s end.
On per-GB billing, the bill climbs fast. MarsProxies’ published residential tiers run from $4.99/GB down to $1.65/GB on its largest plan. At even that lowest rate, 120 TB comes to almost $200,000 a month. Every product photo, tracking pixel, and chat widget gets billed, even though the job only needed a price and an “in stock” label.
Now fetch only the HTML. At around 18 KB per page, the same job uses about 36 GB a day, or roughly 1.1 TB a month. That’s a bill closer to $1,800 at the same rate. Real product pages often carry more HTML than that median, so treat the gap as a direction, not a quote.
How Heavy Pages Have Gotten
Pretty heavy. HTTP Archive tracks this every year. The 2024 Web Almanac chapter on page weight put the median desktop page at 2,652 KB in October 2024, with mobile at 2,311 KB.
Put a million pages through headless Chrome in Playwright, and that’s roughly 2.6 TB per run. Run it daily for a month, and you’re near 80 TB.
The useful part is tiny by comparison. The same chapter found the median desktop homepage carried 18 KB of HTML, against 1,054 KB of images and 613 KB of JavaScript, spread across 71 requests. If your data lives in the HTML, a full render downloads more than 100 times what you need.
What the Model Actually Needs
This is where AI changes the math. An LLM reads text. It can’t use a hero image or a carousel script unless you’re running a vision model, and screenshots for vision agents only add bytes.
So the bytes you pay a proxy to carry are mostly ones the model never sees. Converting fetched pages into clean text or Markdown before they reach the model cuts both bills at once: bandwidth on the way in, and tokens on the way to the LLM.
Cutting Bytes Before They Cost You
Skip the browser when you can. Python’s requests or a Scrapy spider pulls a single document. A browser pulls whatever the page asks for, which means 71 requests on the median desktop page.
Block what you won’t read. Single-page apps built with React and similar frameworks often won’t show data until JavaScript runs. Use Playwright for those, but intercept routes and drop images and fonts before they download. On the median desktop homepage, those two make up nearly half the page’s weight.
Ask for compression. Mozilla’s docs on HTTP compression note that compression can shrink some documents by up to 70%. A server only compresses when the client sends an Accept-Encoding header, and plenty of homemade scrapers never send one.
Don’t refetch what hasn’t changed. For slow-moving catalogs, send ETag and If-Modified-Since headers. An unchanged page returns a 304 with no body, so you pay almost nothing.
Heavy, AI-Shaped Traffic Gets Blocked, or Poisoned
Bandwidth has a second cost that never shows up on an invoice. Wikipedia’s article on web scraping lists the usual defenses: monitoring for excess traffic, CAPTCHAs, IP blocks, and robots.txt rules.
One defense on that list matters a lot for AI teams: tarpits that feed nonsense data to bots ignoring robots.txt, deliberately poisoning their datasets. A pipeline that grabs everything at full weight is exactly the kind of traffic that gets flagged. If it lands in a tarpit, the problem isn’t empty rows. It’s wrong rows flowing into your model’s answers.
Infrastructure is shifting too. In July 2025, Cloudflare began blocking AI crawlers by default for new customers and launched Pay Per Crawl, which lets publishers charge AI companies for access. Publishers now balance blocking crawlers against chasing visibility in AI search, and their settings can change overnight.
The sensible response is to collect less and collect openly:
- Use lighter requests at a reasonable pace.
- Respect robots.txt and crawl-rate rules.
- Identify your crawler where you can.
- Rethink the plan when a site blocks you or asks for payment.
Rotating IPs to get around an explicit block carries real legal risk. In Craigslist v. 3Taps, a US court treated a cease-and-desist letter combined with IP blocking as enough to support a computer-fraud claim. Personal data is its own issue. France’s privacy regulator, CNIL, has said publicly available data is still personal data, so check GDPR and local rules before scraping anything about people.
Run a Pilot Before You Pick a Plan
The cheapest insurance is a small test:
- Sample. Grab 1,000 pages that look like your real target list.
- Measure. Log bytes per page, broken down by content type, and the tokens each page produces after cleanup.
- Multiply. Scale those numbers by your real volume and refresh rate.
Then add headroom. The Almanac found median desktop page weight grew 8.6% from October 2023 to October 2024, so today’s estimate will likely run tight next year.
Pick the proxy plan and rendering setup from that number, not a guess, and the invoice conversation mostly disappears. The open question is what happens when more sites start charging per crawl. At that point, the cheapest byte may be the one you never request at all.
Related: 6 Best Proxies for AI Agents in 2026 (Ranked & Compared)
