Gate one: the crawlers can reach your page
OAI-SearchBot has to be allowed in robots.txt and by your CDN or firewall. Blocking scrapers also blocks the answer crawlers, and that failure is silent.
Gate two: the page is indexed and eligible
Google requires a page to be indexed and eligible to be shown with a snippet. Eligibility for AI features is the same eligibility as classic search.
Gate three: the passage answers on its own
A passage gets quoted when it still makes sense with the rest of the page removed. Lead with the answer, in text form.
What Actually Happens When ChatGPT Answers from the Web
Ask ChatGPT a question about your market and it may answer in front of you, with a row of sources you can click. That is the question business owners now bring us. It is no longer only how do I rank, it is how do I get named.
The short answer is that you do not rank inside ChatGPT the way you rank in Google. You get selected, and selection runs on three gates: can the system fetch your page, is your page eligible to be shown, and does the passage on it answer the question cleanly enough to quote. All three are documented. None of them reward a trick.

ChatGPT decides per question whether to search the web, and the user can force a search manually from the interface. (OpenAI, October 2024) When it does search, the search model is a fine-tuned version of GPT-4o, post-trained with synthetic data that includes distillation from the o1 preview model, and it draws on third-party search providers as well as content provided directly by partners. (OpenAI)
Two things follow from that. First, the index underneath is rented, not yours, so the authority has to live on your own domain. Second, an answer is assembled at the moment it is asked, from pages the system can reach and passages it can parse. Your page is one candidate in that pool, and it is named only if the passage is usable on its own.
The interface also names its sources. Answers carry links, and a Sources control opens the list of references behind the response. (OpenAI) That list is the ranking result people are now competing for.
Google Runs the Same Systems You Already Know
Google's own guidance is blunt: the best practices for SEO remain relevant for AI features such as AI Overviews and AI Mode, and there are no additional requirements to appear in them. (Google Search Central)
To be eligible as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet, and there are no extra technical requirements beyond that. (Google Search Central)
What has changed is how many pages get a chance. Google describes a query fan-out technique, where multiple related searches run across subtopics while the response is being built, and the models identify more supporting web pages to display a wider and more diverse set of links than a classic result page would. (Google Search Central) AI Overviews themselves only appear when Google's systems decide the feature is additive to classic Search, which is why they often do not trigger. (Google Search Central)
A narrow subpage answering one precise question has a genuine shot at being cited, even when it is not your money page. That is the single biggest structural change this shift created, and it is why answer-first pages are worth publishing.
The Two Gates, Before Any Content Work
Nothing on the page matters if the two gates are closed. Both are technical, both are checkable in an afternoon, and both are the first thing we audit.
Gate One: Can the Crawlers Reach You
Google lists it plainly as a best practice for AI features: make sure crawling is allowed in robots.txt, and by any CDN or hosting infrastructure. (Google Search Central) Firewalls, bot managers and challenge pages that were set up to stop scrapers routinely stop the answer crawlers too, and they fail silently.
On the OpenAI side, OAI-SearchBot is the crawler for search. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges, and notes that it takes about 24 hours for its systems to adjust after a robots.txt change. (OpenAI)

| Crawler | What it does | What it controls |
|---|---|---|
| OAI-SearchBot | Surfaces websites in ChatGPT's search features | Whether you can appear in ChatGPT search answers at all |
| ChatGPT-User | Fetches a page when a user action triggers it, not an automatic crawl | Live page reads on request. OpenAI notes robots.txt rules may not apply here, and that this agent does not decide search appearance |
| GPTBot | Crawls content that may be used to train foundation models | Training use only. Blocking it does not remove you from search answers |
| Google's crawlers | Feed Google Search, which includes AI Overviews and AI Mode | Indexing and eligibility for Google's AI features |
Teams block GPTBot because they do not want training use, and assume that also covers search. It does not. GPTBot is a training decision. OAI-SearchBot is the search decision. They are set independently in the same robots.txt file, and confusing them is the fastest way to be invisible while believing you are protected.
Gate Two: Are You Indexed and Eligible
Eligibility for Google's AI features is the same eligibility as classic search. The page has to be indexed and eligible to be shown with a snippet, and Google is explicit that meeting the requirements, best practices and policies still does not guarantee crawling, indexing or serving. (Google Search Central) Treat inclusion as earned per page, never assumed for the domain.
What to Change on the Page
With both gates open, the work is structural. Each item below comes straight from published guidance rather than folklore.

- 1Lead with the answer.
Put the direct answer in the first lines, then the detail underneath. A passage gets quoted when it still makes sense with the rest of the page removed.
- 2Write headings the way the question is asked.
Queries arrive as questions, so headings phrased as questions give the system a clean match point and give readers a scan point.
- 3Keep important content in text form.
Google lists this as a best practice for AI features. Text locked inside images, carousels or client-rendered code is easy to miss and easy to misread.
- 4Let structured data match what is visible.
Google states there is no special schema.org markup needed to appear in AI features, and separately asks that structured data matches the visible text on the page. (Google Search Central) Honest markup that mirrors the page helps. Markup that promises something the page does not say hurts.
- 5Link your supporting pages internally.
Internal links are on Google's own best-practice list for AI features. Publish the narrow pages, the definition, the comparison, the step-by-step, the process, and link them to the pillar they support.
- 6Say who you are.
A named author, the business name, the place you serve, and first-hand detail that a competitor cannot honestly copy. Specifics are what an answer can attribute to you.
- 7Keep business facts current.
Google lists Business Profile and Merchant Center information among the things worth keeping up to date for AI features. Stale hours, stale service areas and stale categories are cheap to fix and common to find.
- 8Keep the page fast and readable.
Page experience is on Google's list too. A page that loads, renders and reads cleanly is one more system working in your favor.
How to Know It Is Working
Answer visibility is measurable, just not with a rank tracker.

- Referral traffic. Watch for chatgpt.com and other answer surfaces in your analytics channel reports. This measures visits, not citations, so it will always undercount how often you are named.
- Search Console. Google states that sites appearing in AI features are included in the overall search traffic in Search Console, reported in the Performance report. (Google Search Central) So impressions from those placements are already in the account you check for everything else.
- Manual sampling. Keep a written list of the questions your buyer actually asks. Run them in a fresh session each month and record whether you are named, and who is named instead. The pattern matters more than any single answer.
- Source checks. When you are cited, open the source link and confirm it lands on the page you intended. Systems often quote the page you forgot about.
The Mistakes That Keep Showing Up
- A CDN, bot manager or firewall quietly blocking the answer crawlers while everything looks fine in your own browser.
- Key claims living only inside images or carousels, which is exactly what the text-form best practice warns against.
- Long warm-up paragraphs before the answer, which is the opposite of how these passages get selected.
- Treating GPTBot and OAI-SearchBot as one setting. They are two decisions in one file.
- Expecting guaranteed inclusion. Google is explicit that indexing and serving are not guaranteed even when every requirement is met.
- Buying so-called AI text files or special markup packages. Google states you do not need to create new machine readable files, AI text files, or markup to appear in AI features.
Want This Run on Your Site?
We audit crawler access, page eligibility and answer-first structure in one pass, then fix the pages that should be getting named. See how our AEO and GEO optimization work is scoped, or start with a technical SEO audit if the gates may already be closed.
Get Your Free Proposalarrow_forwardFrequently Asked Questions
Does ChatGPT use Google or Bing for its answers?
Do I need special schema markup to appear in AI Overviews?
Will blocking GPTBot keep me out of ChatGPT answers?
How long after a robots.txt change?
Is answer engine optimization a separate service from SEO?
What does a local business do differently?

Co-Founder and Head of Search Architecture at Rank First Digital. Founded Mid-Hudson Web in 2000 and has spent 25+ years building websites, applications and search systems for New York businesses. Leads search strategy and technical SEO, and built the agency's AEO and GEO practice around being the source AI engines can safely quote.




