GEO Handbook, Part 9: What AI crawlers can actually see

Your service page looks good in a browser. A crawler receives a security challenge, an empty HTML shell or a conflicting redirect. Even excellent copy cannot resolve that. This chapter explains how to inspect the technical path to the information.

1 — Four checks, four different outcomes#

Access, indexing, source selection and a later enquiry are distinct stages. HTTP status 200 only confirms a successful server response. The content tells you whether the intended page was delivered. An indexed article can still go unused. Technical quality establishes prerequisites; it cannot order a recommendation.

2 — Decide search access and model training separately#

OpenAI distinguishes OAI-SearchBot for search from GPTBot for potential model training. ChatGPT-User performs user-initiated fetches rather than ordinary automatic crawling. Anthropic documents different roles for Claude-SearchBot, ClaudeBot and Claude-User. Perplexity separates its search crawler PerplexityBot from Perplexity-User. Consult each provider’s current documentation for its precise treatment. [1][2][3]

Our working rule follows from this: first record which public content should be discoverable and which uses you intend to allow. Then inspect the relevant controls. A blanket block on all AI bots may have unintended effects. Disabling every protective measure for visibility would be equally misguided.

Google Search also documents a Search generative AI control in Search Console. Inspect the specific property and inherited settings. This control concerns applicable generative search features and is distinct from complete removal from Search or training controls. This describes the documentation, not a change made to your account. [4]

3 — Keep the important claim accessible#

Inspect the page both as a visitor and through the server response. Do title, service, location and contact details agree? Does essential content require a click, login or elaborate interaction? Publish key claims as real text. An infographic can explain them, but should not be their only carrier.

This does not mean all crawlers execute JavaScript equally. We recommend robust core content available from the server so that the page’s meaning depends less on a particular execution environment. Also inspect the visible page on mobile devices and with a keyboard.

4 — Resolve common conflicts carefully#

A bot name in a User-Agent does not prove who sent the request. For firewall or CDN exceptions, check the provider’s verification mechanisms. Apply narrowly scoped changes with a rollback path. A successful manual fetch alone does not establish access for the real bot.

5 — Structured data describe the visible page#

Google does not specify a special AI schema or additional AI text format as a prerequisite. Structured data must match visible information. A sound article therefore needs a clear title, accountable author, meaningful dates and appropriate sources. JSON-LD can describe that information; it cannot replace missing evidence. [5]

Robots.txt is not access security for confidential documents. Google explains that blocked URLs may still appear in search results. Private content needs appropriate access protection. A crawler must be able to read a noindex directive; blocking the page can prevent that. [6]

6 — Start with a representative page#

Choose an important service page, a guide and, where relevant, a language variant. Record the starting state, owner and expected outcome. After the fix, check the same URLs again. Only then apply the change to similar pages. Technical GEO becomes a traceable assignment with a controlled scope.

Connections within the course#

Sources and working method#

Checked on 25 October 2026. Inspection workflows and acceptance matrix are original working models. No new crawl measurement or documented client outcomes.

Glossary: terms and connections#