Executive Summary // How Sources Get Picked
No Special Markup Required
1

Gate one: the crawlers can reach your page

OAI-SearchBot has to be allowed in robots.txt and by your CDN or firewall. Blocking scrapers also blocks the answer crawlers, and that failure is silent.

2

Gate two: the page is indexed and eligible

Google requires a page to be indexed and eligible to be shown with a snippet. Eligibility for AI features is the same eligibility as classic search.

3

Gate three: the passage answers on its own

A passage gets quoted when it still makes sense with the rest of the page removed. Lead with the answer, in text form.

Section 01/The Mechanics

What Actually Happens When ChatGPT Answers from the Web

Ask ChatGPT a question about your market and it may answer in front of you, with a row of sources you can click. That is the question business owners now bring us. It is no longer only how do I rank, it is how do I get named.

The short answer is that you do not rank inside ChatGPT the way you rank in Google. You get selected, and selection runs on three gates: can the system fetch your page, is your page eligible to be shown, and does the passage on it answer the question cleanly enough to quote. All three are documented. None of them reward a trick.

A laptop showing a ChatGPT answer about SEO strategies, with three numbered source cards linking to rankfirstdigital.com, ahrefs.com and moz.com.
An answer with its sources beside it. That source list is the ranking result people are competing for.

ChatGPT decides per question whether to search the web, and the user can force a search manually from the interface. (OpenAI, October 2024) When it does search, the search model is a fine-tuned version of GPT-4o, post-trained with synthetic data that includes distillation from the o1 preview model, and it draws on third-party search providers as well as content provided directly by partners. (OpenAI)

Two things follow from that. First, the index underneath is rented, not yours, so the authority has to live on your own domain. Second, an answer is assembled at the moment it is asked, from pages the system can reach and passages it can parse. Your page is one candidate in that pool, and it is named only if the passage is usable on its own.

The interface also names its sources. Answers carry links, and a Sources control opens the list of references behind the response. (OpenAI) That list is the ranking result people are now competing for.

Section 02/Google's Systems

Google Runs the Same Systems You Already Know

Google's own guidance is blunt: the best practices for SEO remain relevant for AI features such as AI Overviews and AI Mode, and there are no additional requirements to appear in them. (Google Search Central)

To be eligible as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet, and there are no extra technical requirements beyond that. (Google Search Central)

What has changed is how many pages get a chance. Google describes a query fan-out technique, where multiple related searches run across subtopics while the response is being built, and the models identify more supporting web pages to display a wider and more diverse set of links than a classic result page would. (Google Search Central) AI Overviews themselves only appear when Google's systems decide the feature is additive to classic Search, which is why they often do not trigger. (Google Search Central)

Practical Read

A narrow subpage answering one precise question has a genuine shot at being cited, even when it is not your money page. That is the single biggest structural change this shift created, and it is why answer-first pages are worth publishing.

Section 03/Access

The Two Gates, Before Any Content Work

Nothing on the page matters if the two gates are closed. Both are technical, both are checkable in an afternoon, and both are the first thing we audit.

Gate One: Can the Crawlers Reach You

Google lists it plainly as a best practice for AI features: make sure crawling is allowed in robots.txt, and by any CDN or hosting infrastructure. (Google Search Central) Firewalls, bot managers and challenge pages that were set up to stop scrapers routinely stop the answer crawlers too, and they fail silently.

On the OpenAI side, OAI-SearchBot is the crawler for search. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges, and notes that it takes about 24 hours for its systems to adjust after a robots.txt change. (OpenAI)

Three OpenAI crawlers compared: OAI-SearchBot controls appearance in ChatGPT search answers, ChatGPT-User handles user triggered fetches, and GPTBot is for model training only.
Three crawlers, three different decisions. Only one of them removes you from answers.
CrawlerWhat it doesWhat it controls
OAI-SearchBotSurfaces websites in ChatGPT's search featuresWhether you can appear in ChatGPT search answers at all
ChatGPT-UserFetches a page when a user action triggers it, not an automatic crawlLive page reads on request. OpenAI notes robots.txt rules may not apply here, and that this agent does not decide search appearance
GPTBotCrawls content that may be used to train foundation modelsTraining use only. Blocking it does not remove you from search answers
Google's crawlersFeed Google Search, which includes AI Overviews and AI ModeIndexing and eligibility for Google's AI features
The Mix-Up We See Most

Teams block GPTBot because they do not want training use, and assume that also covers search. It does not. GPTBot is a training decision. OAI-SearchBot is the search decision. They are set independently in the same robots.txt file, and confusing them is the fastest way to be invisible while believing you are protected.

Gate Two: Are You Indexed and Eligible

Eligibility for Google's AI features is the same eligibility as classic search. The page has to be indexed and eligible to be shown with a snippet, and Google is explicit that meeting the requirements, best practices and policies still does not guarantee crawling, indexing or serving. (Google Search Central) Treat inclusion as earned per page, never assumed for the domain.

Section 04/On-Page Work

What to Change on the Page

With both gates open, the work is structural. Each item below comes straight from published guidance rather than folklore.

Wireframe of an answer-first page with a highlighted direct answer paragraph, a question heading, a short checklist, and matching FAQPage structured data.
Answer-first structure: the answer first, a question heading, a short list, and markup that matches the visible text.
  1. 1
    Lead with the answer.

    Put the direct answer in the first lines, then the detail underneath. A passage gets quoted when it still makes sense with the rest of the page removed.

  2. 2
    Write headings the way the question is asked.

    Queries arrive as questions, so headings phrased as questions give the system a clean match point and give readers a scan point.

  3. 3
    Keep important content in text form.

    Google lists this as a best practice for AI features. Text locked inside images, carousels or client-rendered code is easy to miss and easy to misread.

  4. 4
    Let structured data match what is visible.

    Google states there is no special schema.org markup needed to appear in AI features, and separately asks that structured data matches the visible text on the page. (Google Search Central) Honest markup that mirrors the page helps. Markup that promises something the page does not say hurts.

  5. 5
    Link your supporting pages internally.

    Internal links are on Google's own best-practice list for AI features. Publish the narrow pages, the definition, the comparison, the step-by-step, the process, and link them to the pillar they support.

  6. 6
    Say who you are.

    A named author, the business name, the place you serve, and first-hand detail that a competitor cannot honestly copy. Specifics are what an answer can attribute to you.

  7. 7
    Keep business facts current.

    Google lists Business Profile and Merchant Center information among the things worth keeping up to date for AI features. Stale hours, stale service areas and stale categories are cheap to fix and common to find.

  8. 8
    Keep the page fast and readable.

    Page experience is on Google's list too. A page that loads, renders and reads cleanly is one more system working in your favor.

Section 05/Measurement

How to Know It Is Working

Answer visibility is measurable, just not with a rank tracker.

An analytics traffic acquisition panel highlighting chatgpt.com referral traffic, above a Google Search Console performance panel labeled as including Google search and AI placements.
Referrals from chatgpt.com in analytics, and impressions Search Console already reports.
  • Referral traffic. Watch for chatgpt.com and other answer surfaces in your analytics channel reports. This measures visits, not citations, so it will always undercount how often you are named.
  • Search Console. Google states that sites appearing in AI features are included in the overall search traffic in Search Console, reported in the Performance report. (Google Search Central) So impressions from those placements are already in the account you check for everything else.
  • Manual sampling. Keep a written list of the questions your buyer actually asks. Run them in a fresh session each month and record whether you are named, and who is named instead. The pattern matters more than any single answer.
  • Source checks. When you are cited, open the source link and confirm it lands on the page you intended. Systems often quote the page you forgot about.
Section 06/Mistakes

The Mistakes That Keep Showing Up

  1. A CDN, bot manager or firewall quietly blocking the answer crawlers while everything looks fine in your own browser.
  2. Key claims living only inside images or carousels, which is exactly what the text-form best practice warns against.
  3. Long warm-up paragraphs before the answer, which is the opposite of how these passages get selected.
  4. Treating GPTBot and OAI-SearchBot as one setting. They are two decisions in one file.
  5. Expecting guaranteed inclusion. Google is explicit that indexing and serving are not guaranteed even when every requirement is met.
  6. Buying so-called AI text files or special markup packages. Google states you do not need to create new machine readable files, AI text files, or markup to appear in AI features.
Work With Us

Want This Run on Your Site?

We audit crawler access, page eligibility and answer-first structure in one pass, then fix the pages that should be getting named. See how our AEO and GEO optimization work is scoped, or start with a technical SEO audit if the gates may already be closed.

Get Your Free Proposalarrow_forward
Clear Answers

Frequently Asked Questions

Does ChatGPT use Google or Bing for its answers?
OpenAI describes it as third-party search providers plus content provided directly by partners. Treat the index layer as rented either way, and make your own domain carry the authority that needs to survive a platform change.
Do I need special schema markup to appear in AI Overviews?
No. Google states there is no special schema.org structured data to add for AI features. What it does ask for is structured data that matches the visible text on the page, plus the fundamentals that already governed search.
Will blocking GPTBot keep me out of ChatGPT answers?
No. GPTBot governs training use. OAI-SearchBot governs search appearance, and sites opted out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links.
How long after a robots.txt change?
OpenAI states it can take about 24 hours for its systems to adjust after a robots.txt update. Fix the CDN and firewall rules at the same time, since those failures do not show up in a robots.txt check.
Is answer engine optimization a separate service from SEO?
The fundamentals are the same, which is exactly Google's position. What changes is the discipline around them: answer-first structure, text-form content, pages written for narrow questions, and measurement that reads referrals and impressions rather than positions.
What does a local business do differently?
Less than you would expect, with two extras. Google names Business Profile information as something to keep current for AI features, and local questions are the ones where a named place, service area and real reviews give the answer something specific to quote.
Portrait of Steven Franzken, Co-Founder and Head of Search Architecture at Rank First Digital
Steven FranzkenCo-Founder & Head of Search Architecture
Book Consultarrow_forward

Co-Founder and Head of Search Architecture at Rank First Digital. Founded Mid-Hudson Web in 2000 and has spent 25+ years building websites, applications and search systems for New York businesses. Leads search strategy and technical SEO, and built the agency's AEO and GEO practice around being the source AI engines can safely quote.