Field notes · AI agents, Hermes & Claude Code By Julian Goldie

AI Search Optimisation Agency Playbook: 9 Steps I Run On Every Client Site (2026)

The exact robots.txt lines, meta tags, schema types and weekly loop an operator uses to get a site ready for Google AI Overviews, ChatGPT search, Perplexity and Copilot.

Get The AI Profit Stack Join AIPB →
  • 01 1,000+ done-for-you AI agent workflows
  • 02 5 live coaching calls / week with me
  • 03 7-day refund, cancel anytime
  • 04 3,400+ AI operators inside

An AI search optimisation agency earns its keep in the unglamorous layer: crawler access, snippet controls, entity data, useful pages and clean measurement across Google, ChatGPT, Perplexity and Copilot.

This is the operator version: the nine steps I would run on a client site, in order, with the exact lines, tags and schema types.

Each step names where an AI agent can take the grunt work off you, and where a human has to sign off.

If you want my team to run it with you, book a free SEO strategy session or go straight to the calendar: Book a free SEO strategy session.

The ground rule: AEO and GEO sit on top of ordinary SEO

Google is blunt about this in its guide to optimising for generative AI features: from Search's point of view, optimising for generative AI is still SEO.

Its AI features page adds that AI Overviews and AI Mode have no extra requirements beyond being indexed and eligible for a snippet.

Bing says the same about Copilot in its Webmaster Guidelines: grounding runs on the same crawling, indexing and ranking base as classic search.

So the engagement is not a separate magic track.

It is SEO done properly, plus a few platform-specific switches that most sites have never checked.

Those switches are where an operator finds quick wins, and they are steps 2 to 4 below.

Step 1: Scope the engagement and set up access the safe way

Before touching anything, agree what is in scope and who owns which account.

  • Search Console: read-only access at the audit stage, as Google's hiring page advises; upgrade only once the plan is signed.
  • Bing Webmaster Tools: add the agency as a user so they can see the AI Performance report.
  • ChatGPT Business Hub: the business admin claims the listing and then grants the agency access, per OpenAI's Business Hub help article; the agency never claims it under its own login.
  • CMS and DNS: named logins per person, never a shared password, and a written change log from day one.

The client keeps ownership of every account.

If the agency leaves, nothing should need to be recovered.

Step 2: Audit AI crawler access in robots.txt, the CDN and the WAF

This is where I find the most broken setups.

Each AI platform publishes separate bots for search and for training, and they are switched on and off independently.

PlatformSearch / answer bot to allowTraining bot (separate choice)Source
ChatGPT searchOAI-SearchBotGPTBotOpenAI crawlers
PerplexityPerplexityBotNone: PerplexityBot is not used for model trainingPerplexity crawlers
ClaudeClaude-SearchBotClaudeBotAnthropic crawlers
Google AI Overviews and AI ModeGooglebotGoogle-Extended (Gemini, not Search)Google common crawlers
CopilotBingbotNew training preference from February 2027Bing guidelines and blog

A starting robots.txt block for a client who wants AI search visibility looks like this.

# AI search crawlers: allow so pages can be found and cited
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Googlebot
Allow: /

# Training crawlers: a separate business decision
# User-agent: GPTBot
# Disallow: /
# User-agent: ClaudeBot
# Disallow: /
# User-agent: Google-Extended
# Disallow: /

Sitemap: https://www.example.co.uk/sitemap.xml

Whether to block the training bots is the client's call, not yours, so leave those lines commented until they decide.

Gotchas I check every time:

  • OpenAI says robots.txt changes take about 24 hours to register, so do not panic-test the same afternoon.
  • OpenAI and Perplexity both say to allow their published IP ranges as well; a CDN bot-fight mode can block them even when robots.txt allows them.
  • Blocking Google-Extended does not remove a site from Google Search and is not a ranking signal, per Google's crawler docs, so it is not an AI Overviews switch.
  • ChatGPT-User fetches pages on a user's request and is not what decides search inclusion; manage search through OAI-SearchBot.

📞 Want an operator to run this nine-step playbook on your site? Grab a free strategy call with my team and we will walk your robots.txt, snippet controls and entity setup with you, live. Prefer to pick a slot straight away? Book a free SEO strategy session. → Get the free strategy session

Step 3: Check snippet and preview controls page by page

Google's robots meta tag documentation says nosnippet also stops a page being used as a direct input for AI Overviews and AI Mode.

So a stray tag in a template can quietly remove a whole section from AI answers.

<meta name="robots" content="nosnippet">      <!-- removes the page from AI Overviews input -->
<meta name="robots" content="max-snippet:50">  <!-- limits how much text can be shown -->
<span data-nosnippet>Internal pricing note</span>  <!-- hides one passage only -->

Crawl the site, export every page that carries nosnippet, max-snippet or data-nosnippet, and ask the client which ones are deliberate.

On the Bing side, NOARCHIVE keeps content out of Copilot answers and NOCACHE limits Copilot to the URL, title and snippet.

Bing's October 2026 announcement replaces both with new AI Training and AI Grounding preferences from 1 February 2027, with a 1 March 2027 deadline for sites that use the old tags.

Put that date in the client calendar now.

Step 4: Make indexing and rendering boring

AI answers can only cite what the engines have indexed and can read.

  • JavaScript sites: Google's JavaScript SEO basics still recommends server-side or pre-rendering, because not every bot runs JavaScript.
  • Key facts in rendered HTML text, not inside images or tabs that load on click.
  • XML sitemaps with canonical URLs only and accurate lastmod dates.
  • IndexNow pings to Bing when pages are added, updated or removed, which Bing says keeps Copilot references fresher.
  • Duplicate URLs consolidated with redirects or canonicals, because Bing says duplicates lower its confidence when choosing a page to cite.

Step 5: Build the entity layer: Organization, LocalBusiness and sameAs

No special schema is needed for AI features, and Google says so plainly.

Structured data still helps engines work out who a business is, and Bing asks for clear, consistent entity names.

Use Organization markup on the home page, with sameAs pointing at the profiles the business really owns.

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Example Plumbing Ltd",
  "url": "https://www.example.co.uk/",
  "logo": "https://www.example.co.uk/logo.png",
  "sameAs": [
    "https://www.linkedin.com/company/example-plumbing",
    "https://www.youtube.com/@exampleplumbing"
  ]
}

For a shop or service business with a premises or a service area, add the right LocalBusiness subtype instead.

The markup must match the visible text on the page; marking up facts the page does not show is a fast way to have it ignored.

Then make the business name, address and phone identical on the site, the Business Profile and the Business Hub listing.

Step 6: Rework the money pages into non-commodity answers

Google's AI guide says unique, non-commodity content will likely matter more than anything else in it over the long run.

In practice I run every service page through four questions.

  • Does the first screen answer the main question directly, as Bing's guidelines ask?
  • Is there something only this business can say: a process, a photo of real work, a number from its own records?
  • Does the page stick to one topic and stand on its own without needing other pages for context?
  • Would a customer feel it was worth the click?

Skip the AI hacks.

Google says it ignores chunking content into tiny pieces, rewriting copy just for AI, and llms.txt files.

Google Search does not use llms.txt; it may matter to other services, so treat it as optional housekeeping, never as a ranking task.

🔥 Rather run the playbook yourself? Inside the AI Profit Boardroom you learn AI SEO yourself: step-by-step video trainings, the installable Agent OS, four live coaching calls a week and a daily Q&A. → Join the AI Profit Boardroom

Step 7: Earn mentions and links inside the spam policies

AI answers lean on what the wider web says about a business, so reputation work belongs in the plan.

Keep it inside Google's link spam policy: links bought to pass ranking credit, big link swaps and automated link building are all named there.

Paid placements are fine when they carry rel="sponsored", as Google's outbound link guide explains.

Goldie Agency describes its own approach as "Manual outreach, no PBNs", and that is the bar I would hold any provider to.

Google also says chasing inauthentic mentions is not as helpful as it looks, so skip fake reviews and forum spam completely.

Step 8: Measure AI visibility with the platforms' own reports

  • Google Search Console: the generative AI performance reports, rolled out to all sites on 31 August 2026, show impressions, pages, countries and devices inside AI features.
  • Bing Webmaster Tools: the AI Performance report shows total citations, cited pages and grounding queries across Copilot.
  • ChatGPT Business Hub: 28-day insights on mentions, website clicks and discovery topics once the claim and identity checks are approved.
  • Analytics: in our experience ChatGPT referrals usually show up with chatgpt.com as the source, so build a segment for it.

Report impressions and citations alongside clicks.

Bing itself notes that fewer clicks does not always mean less visibility.

Step 9: Run a weekly loop and a monthly report

  1. Monday: crawl, check robots.txt and meta-tag diffs, ping IndexNow for changed URLs.
  2. Tuesday to Thursday: one money page reworked, one entity fix, one outreach batch drafted for human approval.
  3. Friday: log every change with its date, so movement in the reports can be traced.
  4. Month end: AI impressions, citations and leads, plus what changed and what is next.

Where AI agents speed the work up

Agents are good at the repetitive half of this playbook.

  • Diffing robots.txt and meta tags across every URL each week.
  • Drafting Organization and LocalBusiness JSON-LD from the facts the client approves.
  • Turning a call transcript into a first-draft brief for a service page.
  • Building a list of relevant sites for outreach, with every email left as a draft.

They should not publish pages, send outreach or change robots.txt without a person approving it.

Google is clear that you answer for the work, whoever or whatever did it.

A worked example: a two-van plumbing firm

  1. Day 1: confirm OAI-SearchBot, PerplexityBot and Bingbot are allowed, and the CDN is not challenging them.
  2. Day 2: remove a nosnippet tag the old theme put on every service page.
  3. Day 3: add LocalBusiness markup for the service area and match the Business Profile details to the site.
  4. Week 2: rewrite the boiler repair page so the first lines answer cost factors, call-out areas and response times in the firm's own words.
  5. Week 3: claim the business in ChatGPT Business Hub if the firm is eligible, and invite the agency through it.
  6. Month 2: review AI impressions in Search Console and citations in Bing before deciding the next pages.

Red flags when you vet an agency to run this

  • A guaranteed #1 ranking or a guaranteed ChatGPT mention: Google, Bing and OpenAI all say placement cannot be promised.
  • A "special relationship" with Google or a priority submit service.
  • Private blog networks, link swaps or bulk directory links.
  • Hundreds of near-identical pages for every query variation, which Google treats as scaled content abuse.
  • No change log and no reason given for each recommendation.

Google's page on hiring an SEO now asks directly whether an agency's AEO or GEO advice matches its official guidance.

Ask that question in the first call and listen for documents, not secrets.

🎁 Two free tools for the audit week Download the 200+ AI SEO Prompts for briefs, schema drafts and page reviews, and grab the free link building book before you plan any outreach. → Learn AI SEO yourself in the AI Profit Boardroom

Who runs this at Goldie Agency

Julian Goldie's own published numbers: 1,000+ SEO and link building campaigns, 240+ clients and a 100% Job Success Score on Upwork, plus 425K+ YouTube subscribers.

His about page describes Goldie Agency as an AI search optimisation agency that tests AI SEO, GEO and AEO on its own sites every week before rolling it out to clients.

Nobody, including us, can guarantee a ranking or an AI citation.

What a good operator can promise is clean access, clear reporting and work that sits inside the platforms' rules.

📞 Hand the whole engagement to my team Book a free SEO strategy session and we will map steps 1 to 9 onto your site, with nothing promised that Google, OpenAI or Bing say nobody can promise. → Book the free strategy session

Also On Our Network

FAQ: AI Search Optimisation Agency Playbook

Which robots.txt lines does an AI search optimisation agency check first?

OAI-SearchBot for ChatGPT search, PerplexityBot for Perplexity, Claude-SearchBot for Claude, plus Googlebot and Bingbot. Training bots such as GPTBot, ClaudeBot and Google-Extended are a separate decision for the business owner.

Does blocking Google-Extended remove a site from AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. AI Overviews and AI Mode are controlled through Googlebot and snippet controls such as nosnippet.

What access should I give an agency at the start?

Read-only Search Console access during the audit, a user seat in Bing Webmaster Tools, and agency access through ChatGPT Business Hub granted by the business admin. Keep ownership of every account yourself.

Is llms.txt part of the playbook?

Only as optional housekeeping. Google says Google Search does not use llms.txt, so it is never treated as a ranking task here.

How do you measure results in AI search?

The generative AI performance reports in Search Console, the AI Performance report in Bing Webmaster Tools, Business Hub insights in ChatGPT, and a chatgpt.com referral segment in analytics.

📺 Video notes + links to the tools 👉

🎥 Learn how I make these videos 👉

🆓 Get a FREE AI Course + Community + 1,000 AI Agents 👉

About Julian

I'm Julian Goldie, an SEO entrepreneur, author and founder of the AI Profit Boardroom, which has 3,400+ members.

I help business owners scale with AI agents, automation and SEO.

  • I've built a 7-figure SEO and link building agency, Goldie Agency, with a team of 70+ SEO experts.
  • I've grown a YouTube channel to 400,000+ subscribers.
  • I wrote "Link Building Mastery".
  • My Udemy courses have taught over 29,000 students.

→ Get my best AI training inside the AI Profit Boardroom

Run steps 2 and 3 this week, log every change, and if you want an operator in the loop, book a free strategy session with the team that runs this playbook as an ai search optimisation agency.

§ Field reports

Real wins from inside the AI Profit Boardroom

See all 3,400+ members →
AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot

Scroll sideways →

⚑ Next step

Ready to run AI agents that actually do the work?

Join 3,400+ operators inside the AI Profit Boardroom. Get 1,000+ plug-and-play AI agent workflows, 5 coaching calls a week, and a community that holds you accountable.

Join The AI Agent Community →

7-day no-questions refund · Cancel anytime

Found this useful? Get more of this site in your Google.

One tap adds us as a preferred source — Google then shows you our posts more often in Search, Top Stories, and AI Overviews.

← Back to all posts