The technical layer: crawlers, schema, and the speed of getting cited

The mention is the signal. The technical layer decides if you ever get one.

Most of Visigent's research lives one level up from the code. It is about narrative, authority, entity strength, and the simple truth that AI recommends, it does not rank. A model does not return ten blue links and let the user sort them out. It composes an answer and names a handful of sources. The mention is the signal, and the mention is binary. You are in the answer or you are invisible.

This report goes underneath that. Before an engine can decide whether to name you, three mechanical things have to happen. A crawler has to reach your page and be allowed to read it. A parser has to understand what the page is about with as little ambiguity as possible. And the page has to render fast enough and cleanly enough that the content is actually available to the model. None of this is glamorous. All of it is upstream of the citation.

The trap is treating the technical layer like classic SEO and assuming the two share a playbook. They overlap, but they diverge in ways that quietly cost brands citations. AI crawlers are not one monolith. They split into training bots and retrieval bots, and those serve completely different functions. Schema behaves differently for AI answers than it does for rich snippets. And page speed, long treated as a ranking nicety, turns out to correlate with citation volume at a magnitude that should change how teams prioritize engineering work.

The content strategy gets you considered. The technical layer determines whether the engine can act on the consideration. One without the other is wasted spend.

There is a second reason the technical layer gets neglected. It does not show up in the dashboards marketers watch. A content team can see a blog post's word count, its keyword targets, its backlink profile. It cannot see, without looking, that the page renders in 1.4 seconds, that its critical content loads only after a JavaScript framework hydrates, or that a robots.txt line written two years ago is quietly turning away the exact crawler that feeds ChatGPT. These are invisible failures. The post looks finished. The engine simply never receives it in a usable form.

That invisibility is why the technical layer rewards teams that measure it directly. The brands winning AI citations are not always the ones with the most content. They are frequently the ones whose pages are fastest, cleanest, and most legible to a machine operating under a time budget. The work is unglamorous and it compounds.

Three things move the needle: who you let crawl you, how legibly you describe yourself, and how fast you deliver the bytes. We will take each in turn.

Who is actually crawling you, and why it is not who you think

The instinct is to picture "the AI bots" as a single swarm. The data says otherwise, and the composition matters because allowing or blocking the wrong agent has opposite consequences.

The volume is no longer a rounding error. In June 2026, Cloudflare's Matthew Prince reported that bots generate 57.5% of HTML web traffic, the first time automated requests have held the majority over humans (Cloudflare). AI crawlers alone made up 20.3% of verified bot traffic in May 2026, with AI-search bots adding another 6.5% (Cloudflare). The machines reading your site now outnumber the people. Whether they can read it, and whether you let them, is a question every brand has to answer on purpose rather than by default.

Start with raw share. In May 2026, Googlebot accounted for 27.26% of AI-adjacent bot HTTP requests, making it the single largest crawler reaching the average site (Cloudflare). That is the most important fact in this chapter. The crawler most people think of as legacy search is also the primary pipe into Google's AI Overviews, which now touch the majority of informational queries. Block Googlebot to spite AI and you delete yourself from both search and AI in one move.

Behind Google, the named AI crawlers are smaller and more volatile than the headlines suggest. GPTBot sat at 11.48% of AI bot requests in May 2026, with ClaudeBot at 9.73% (Cloudflare). That order had flipped from April, when ClaudeBot led 11.69% to GPTBot's 9.84%. The takeaway is not the leaderboard. It is that no single AI crawler dominates the way Google dominated search, and the rankings swing month to month.

The figure that reframes everything is purpose. In May 2026, Cloudflare attributed 51.8% of AI crawler requests to training, 35.7% to mixed training-plus-retrieval, and only 9.3% to search-only purpose. More than half of the bots hitting your servers are there to ingest your content into a model, not to fetch it in response to a live user question. Those are different transactions with different value to you.

• Training crawlers (GPTBot, Google-Extended, ClaudeBot, CCBot, Bytespider, Applebot-Extended) consume your content to build future model weights. They send you nothing in return today.
• Retrieval and search crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, BraveBot) fetch your pages to answer a question a real person just asked. These are the bots that produce citations and clicks.
• Googlebot is the exception that does both jobs with one user agent, which is exactly why it cannot be blocked safely.

This split is the single most useful mental model in crawler management. It means blocking AI is not one decision. It is two.

The crawl-to-refer gap, and why "block the AI bots" is a trap

There is a real grievance underneath the crawler debate, and it is worth stating plainly because it explains why so many teams reach for a blanket block.

The economics are lopsided. Cloudflare's tracking of crawl-to-refer ratios shows how few visitors AI engines return for the content they consume. In early 2026, ClaudeBot crawled roughly 23,951 pages for every single referral it sent back, while GPTBot sat near 1,276:1 (Cloudflare). For comparison, DuckDuckGo runs near parity at about 1.5:1, and Google's classic search hovers around 5:1. By mid-2026 the AI gaps had narrowed, with one Cloudflare reading putting Anthropic near 4,580:1 and OpenAI near 848:1, but the imbalance is structural. AI engines take far more than they give back in raw traffic.

That feels like a reason to block everything. It is not, and the reason is the thesis of this entire corpus.

If AI is becoming the place buyers form opinions and shortlists, then a citation is not a traffic source. It is a brand placement at the moment of decision. You do not measure it in clicks.

When 67% of B2B buyers now start research with AI, the value of being named in the answer is not the click that may or may not follow. It is being the brand the buyer carries into the next step. Blocking the retrieval crawlers to protest the traffic gap removes you from that moment entirely. You win the principle and lose the market.

The disciplined position is surgical, not blanket:

1. Always allow the retrieval and search crawlers. OAI-SearchBot, PerplexityBot, Claude-SearchBot, and BraveBot are how you get cited. Blocking them is self-removal from AI answers.
2. Decide training separately, on its own merits. Blocking GPTBot, Google-Extended, ClaudeBot, and CCBot stops your content feeding future model training. Critically, blocking training crawlers does not harm your search or AI-answer visibility (the search bots are separate agents). This is a content-rights decision, not a visibility decision.
3. Never block Googlebot. It is one user agent for search and AI Overviews. There is no clean way to remove it from AI without removing it from search.
4. Name the agents that lack a clean identifier. xAI's Grok crawlers do not always present a standard bot string. If you are configuring allow rules, add them explicitly rather than assuming a wildcard covers them.

A defensible robots.txt for a brand that wants maximum AI visibility allows every retrieval crawler, makes a deliberate yes-or-no call on each training crawler, and leaves Googlebot untouched. The most common self-inflicted wound in GEO is a well-meaning blanket Disallow that took the search bots down with the training bots.

llms.txt: a good idea that the engines still ignore

No technical-layer report is complete without addressing llms.txt, because it is the single most over-recommended tactic in GEO right now. The pitch is appealing. Place a markdown file at your root that describes your content for language models, and you supposedly hand the engines a clean map of your site.

Adoption among publishers is real and growing. An SE Ranking study of 300,000 domains found a 10.13% adoption rate, roughly one site in ten, after eighteen months of industry discussion. By April 2026, names like Anthropic, Stripe, Zapier, Cloudflare, Vercel, and Hugging Face had shipped one. The standard is no longer fringe.

The problem is the demand side. The crawlers do not read it. Analysis shows GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended overwhelmingly skip /llms.txt and crawl HTML directly. ALLMO's analysis of more than 94,000 cited URLs found no measurable citation uplift associated with llms.txt adoption. Anthropic has confirmed Claude's products respect llms.txt directives in some retrieval workflows, which is the strongest signal any major lab has given, but no major engine has confirmed it as a ranking or citation input.

llms.txt is a bet on a future standard, not a present-day citation lever. Ship one if it is cheap. Do not let it crowd out work that is proven to move citations.

The practical guidance is unchanged from a year ago. If a developer can add the file in an hour, there is little downside and a modest option value if adoption accelerates. But any team treating llms.txt as a priority over schema, performance, or crawler hygiene is optimizing for a signal the engines are not yet reading. The bytes that get cited are still the HTML.

Schema: legibility, not magic, and why stacking beats stuffing

Schema is one input to the technical layer, not the whole of it, and Visigent's schema playbook covers the implementation in depth. Here the point is narrower: structured data is how you remove ambiguity about what a page is and who you are, and the engines reward legibility.

The headline finding is consistent across studies. Pages running 3–4 complementary schema types get cited roughly 2x more often than pages with a single schema type (Search Engine Land). The operative word is complementary. The lift comes from layering types that describe different facets of the same page, not from cramming one page full of every type in the vocabulary.

• Organization defines the brand entity. Its sameAs array, linking to Wikipedia, Wikidata, LinkedIn, and Crunchbase, is the single most important property for entity disambiguation. It tells the engine which "Acme" you are.
• Article or BlogPosting carries author, publisher, and publication dates, which feed freshness and authority signals.
• Person establishes author credentials and expertise, the structured backbone of E-E-A-T for AI.
• BreadcrumbList gives the engine site-structure context.

A page that carries Organization, Article, Person, and BreadcrumbList together is describing itself from four angles. That convergence is what the 2x figure rewards.

Two cautions keep teams honest. First, schema must match visible content. Google's official guidance is explicit, and marking up claims that do not appear on the page is a manipulation risk, not an optimization. Second, more schema is not strictly better. SE Ranking found FAQPage schema correlates negatively with ChatGPT citations specifically, 3.6 citations with the markup versus 4.2 without. The FAQ content format still helps. The schema wrapper can hurt for ChatGPT-focused pages. Stack complementary types with intent. Do not stuff.

Schema is no longer a technical add-on. It is a core signal that helps AI decide which sources are most reliable. — WordStream, 2025

The mechanics are settled: JSON-LD format, @id properties to connect entities across the site rather than forcing everything into one block, validation through Google's Rich Results Test. The strategy is the part teams miss. Schema is the place you tell the engine, without ambiguity, exactly what you are and what you know.

Speed and rendering: the citations you lose before the model ever reads you

The most underrated finding in the technical layer is about milliseconds, and it should reorder some engineering backlogs.

SE Ranking's analysis of 129,000 domains found a startlingly clean relationship between load speed and citation volume. Pages with First Contentful Paint under 0.4 seconds averaged 6.7 citations. Pages over 1.13 seconds averaged 2.1. That is roughly a 3x gap tied to a metric most marketing teams delegate to engineering and forget. The likely mechanism is straightforward. Retrieval crawlers operate under time and resource budgets. A page that paints content instantly delivers usable text inside that budget. A slow page risks returning thin or empty content to the bot, and an engine cannot cite text it never received.

Speed is necessary but not sufficient. How the page renders matters as much as how fast.

• JavaScript rendering remains a gap. AI crawlers do not render client-side JavaScript as reliably as Googlebot. Content that only exists after a JS framework hydrates may be invisible to the bot. Use server-side rendering or static HTML for anything you need cited.
• Semantic HTML is the substrate. A clean H1-to-H2-to-H3 hierarchy, real lists, and real tables give the parser structure to work with. SE Ranking found pages organized into 120-to-180-word sections between headings earn 70% more ChatGPT citations than fragmented pages.
• Comparison tables punch above their weight. Digital Bloom found tables get cited 2.5x more than the same information in unstructured prose, because they map cleanly onto the comparative questions buyers ask AI.

The freshness signal sits alongside speed. SE Ranking found articles updated within the past three months average about 6 citations versus 3.6 for stale content, and SISTRIX measured 3.2x more citations for content within 30 days of publication. A fast, well-structured page that is also recent is the strongest technical profile you can offer a retrieval crawler.

Page speed is not a ranking nicety in AI search. It is the difference between a crawler receiving your content and receiving nothing. You cannot be cited for bytes that never arrived.

There is a counterintuitive wrinkle worth naming. SE Ranking's data showed that high Interaction to Next Paint, normally a red flag for poor responsiveness, correlated positively with citations. The likely explanation is not that slow interactivity helps. It is that genuinely rich, interactive, content-dense pages tend to carry the depth engines want to cite, even when they cost something in responsiveness. The lesson is not to chase a perfect Core Web Vitals scorecard for its own sake. It is to deliver substantive content fast on first paint, then let the page be as rich as the topic demands.

URL structure is the quiet finisher. SE Ranking found that broad, topic-descriptive URLs earn roughly 2x more citations than keyword-stuffed ones. A path like /crm-software-guide reads as a stable topic entity to an engine. A path like /best-crm-software-small-business-2025 reads as a dated marketing artifact. The engine is building a map of what your site is about, and clean URLs are legible coordinates on that map.

Put it together and the technical layer has a clear hierarchy. Let the right crawlers in. Render fast and server-side so they actually receive your content. Structure that content with semantic HTML and complementary schema so the parser understands it. Keep it fresh. Every one of those is upstream of the citation, and every one is something an execution-focused team can control this quarter.

Every statistic in this piece follows Visigent's research standards: named sources, dated figures, corrections on request to help@visigent.com.