Technical SEO in 2026: Is Your Website Invisible to AI Search?

Most technical SEO problems are invisible until they cost you something. A product category quietly drops out of Google’s index. A redesign ships with a stray noindex tag. A JavaScript framework that looks perfect in the browser serves search bots an almost empty page.

None of that shows up on the homepage. It shows up weeks later in traffic reports, if anyone’s looking.

There’s also a second audience now, and it behaves differently from Googlebot. AI assistants like ChatGPT, Claude, and Perplexity send their own crawlers to build answers, and plenty of sites are invisible to them without knowing it. Search performance used to mean rankings and clicks. Increasingly, it also means whether an AI system can read your page at all, and whether it cites you.

A good Technical SEO agency earns its fee by finding the problems that never announce themselves. Here’s what that work covers today, old fundamentals and new AI-era checks included.

Crawling: can bots actually get through?

Everything starts here. If a crawler can’t reach a page, nothing else you do for that page matters.

The usual culprits are dull: broken internal links, redirect chains, accidental Disallow rules in robots.txt, orphan pages nothing links to, and faceted navigation that spawns thousands of near-duplicate URLs. On big sites, that last one can eat a crawler’s attention so badly that important pages get visited rarely.

AI crawlers make this more pressing, not less. When Vercel analyzed traffic across its network in late 2024, it found that ChatGPT’s and Claude’s crawlers spent over 34% of their requests on 404 pages. Messy URL structures and dead links waste their visits even more than Googlebot’s.

The fix starts with server logs. Search Console’s crawl stats show what Google does on your site, and logs show what every bot does.

Your robots.txt now has a lot more readers

For years, robots.txt mostly mattered for Googlebot and Bingbot. That’s changed. AI companies now run several distinct bots each, and they don’t all do the same job.

The split matters. Blocking a training bot doesn’t block the matching search bot. A site that blocks only GPTBot and ClaudeBot is opting out of model training, but it can still show up and get cited in ChatGPT and Claude answers through their search crawlers. Google says Google-Extended has no effect on inclusion in Google Search.

So decide deliberately. A publisher who wants citations but not training might use something like this:

User-agent: GPTBot

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: OAI-SearchBot

Allow: /

User-agent: Claude-SearchBot

Allow: /

One caveat: rules for user-triggered fetchers behave differently. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says its user fetcher generally ignores it. Anthropic says all three of its bots honor robots.txt.

Your CDN may have decided for you

Some sites block AI crawlers without anyone on the team choosing to. Cloudflare, which sits in front of a large share of the web, started blocking known AI crawlers by default for new domains on July 1, 2025. It also launched a “pay per crawl” option so publishers can charge AI companies for access.

Cloudflare’s reasoning was blunt. It measured how often each company crawls compared with how often it sends visitors back. In June 2025, Google crawled about 14 times per referral, while Anthropic’s ratio was 73,000 to one. (Anthropic makes Claude, the assistant writing this, so take that figure as reported rather than endorsed.)

Whether you block or allow is a business call, not a technical one. But it should be a call someone actually makes. Check your CDN and firewall settings, not just robots.txt, because those layers can block bots before robots.txt ever matters.

JavaScript: what Google sees, AI crawlers may miss

This is probably the biggest blind spot in technical SEO right now.

Google renders JavaScript. It fetches the page, queues it, and later runs it in a headless browser to see what users see. The main AI crawlers don’t. Vercel’s study with MERJ found that none of the major AI crawlers it measured executed JavaScript, including OpenAI’s crawlers and Anthropic’s ClaudeBot. GPTBot did download JavaScript files about 11.5% of the time, but it never ran them.

The practical result is that a single-page app loading product specs, prices or article text through client-side calls can rank fine on Google and still look like an empty shell to ChatGPT or Claude.

Testing takes a minute. Fetch a page the way a bot would:

curl -s -A “GPTBot” https://example.com/page | grep “your key sentence”

If your main content doesn’t appear in the raw HTML, AI crawlers probably can’t see it either. Server-side rendering or static generation for core content fixes it. Keep client-side scripts for extras like chat widgets and view counters.

Speed and Core Web Vitals, updated

Page speed still matters, though the metrics changed. Google replaced First Input Delay with Interaction to Next Paint as a Core Web Vital on March 12, 2024. INP measures how quickly a page responds visually after someone taps or clicks, across the whole visit, not just the first interaction.

Google has also said good Core Web Vitals don’t guarantee good rankings, and that obsessing over tiny score gains isn’t worth it. Treat them as a user-experience floor. Heavy images, render-blocking scripts and bloated third-party tags are the usual offenders. A technical audit should find them, and a developer should fix them.

There’s a quieter benefit for AI too. AI crawlers tend to use short timeouts, so a page that’s slow to return its HTML risks being skipped altogether.

Mobile is the default now

Google finished moving to mobile-first indexing in 2024, which means it judges sites mainly on their mobile version. If something exists on desktop but not on mobile, such as content, links or structured data, assume Google doesn’t see it.

Check for parity. Hidden tabs, trimmed mobile menus and “lite” mobile templates are classic places where content quietly disappears.

Architecture, indexing and duplicates

Site structure decides how easily people and bots understand what matters. Shallow, logical hierarchies help. That might mean category, then subcategory, then product, with sensible internal links between related pages. Pages buried six clicks deep tend to get crawled less and ranked less.

Indexing is where small mistakes cause large losses. Common problems include:

• noindex tags left over from a staging site
• canonical tags pointing at the wrong URL, or contradicting the sitemap
• several URLs serving the same content, through parameters, trailing slashes, or HTTP and HTTPS versions
• sitemaps listing pages that redirect or return errors

Each one sends search engines mixed signals about which page is the “real” one. A technical audit lines these up and makes them agree.

Structured data still helps, and llms.txt is unproven

Schema markup describes your content in a format machines can parse, such as products, prices, reviews, articles, events, and organizations. Correct markup can make pages eligible for rich results in Google, and it gives any system reading the page clearer facts to work with.

You may also hear about llms.txt, a proposed file that summarizes a site for AI models. Treat it with caution. Google has said it doesn’t use it, and there’s little public evidence the major AI assistants rely on it. It’s cheap to add, but don’t expect it to fix anything a crawl problem is causing.

Clean HTML, clear headings, and accurate structured data do much more of the work. AI systems also repeat what they read with real confidence, and one journalist showed how a single fake web page could fool chatbots within a day. Accurate, clearly marked-up facts on your own pages are your best defense against being misquoted.

Monitoring never really ends

Technical SEO isn’t a one-time fix. Sites change every week as new pages ship, old ones vanish, plugins update, and developers push code. The canonical tag that was right in January can be wrong by June.

Ongoing monitoring should cover crawl errors and index coverage in Search Console, Core Web Vitals trends, server logs, including AI bot activity, and alerts for sudden robots.txt or noindex changes.

Watch where traffic comes from as well. AI assistants have huge audiences, and ChatGPT alone has passed a billion users. Referrals from AI answers are becoming a channel worth measuring, even while they’re small.

FAQ

Q. Should I block AI crawlers?
That depends on your goals. Blocking training bots like GPTBot and ClaudeBot keeps your content out of model training, while allowing search bots like OAI-SearchBot and Claude-SearchBot keeps you eligible for citations in AI answers.

Q. Does blocking Google-Extended hurt my Google rankings?
No. Google says Google-Extended doesn’t affect inclusion in Google Search. It controls whether content can be used for Gemini.

Q. Do AI crawlers read JavaScript?
Testing shows the major ones don’t execute it. Put your key content in server-rendered HTML.

Q. What replaced FID in Core Web Vitals?
Interaction to Next Paint (INP), since March 12, 2024.

Q. Is llms.txt worth adding?
It’s harmless, but unproven. Fix crawlability, rendering and structured data first.

Bottom Line 

Technical SEO used to mean making one search engine’s job easier. Now there are several kinds of bots reading your site for different reasons, and each one sees a different version of it. The sites that do well will be the ones where someone checks what each of those bots actually gets.

Related: Why AI Crawlers Are Breaking JavaScript SEO in 2026