AI Visibility is becoming one of the most important factors in how experts, founders, and brands are discovered online.
A few days ago, I found myself wondering why some people seem to appear everywhere in ChatGPT, Perplexity, and other AI tools when you ask for experts in a particular field, while many highly capable professionals never show up at all.
At first, I assumed the answer was simple. Better content. More posts. More effort.
But the deeper I looked, the more I realised that a completely different game is unfolding behind the scenes.
As AI search changes the way people find information, new rules are emerging around visibility, authority, and trust. And the good news is that understanding these rules doesn’t require you to be a tech expert. It simply requires knowing what AI platforms are looking for — and how to position your expertise so it can be found, cited, and recommended.
Here’s what I learned.
I assumed it was a content thing. Better writing. More posts. More hustle.
I was wrong. And once you see why, you can’t unsee it.
Here’s everything I learned, broken down so anyone — not just tech people — can act on it today.
The Internet Has a New Librarian
Forget SEO for a second. Forget Google rankings. There’s a new game happening right now, and almost nobody is playing it on purpose.
When you ask ChatGPT, Perplexity, or Claude a question, those tools don’t just “know” things by magic. They send out little automated programs called crawlers — basically robot readers — that go visit websites, read what’s there, and bring information back.
Think of it like this: imagine you hired the world’s fastest intern. Their only job is to read every website on earth and come back with notes. That’s a crawler. Google has used these for 25 years. AI platforms now have their own versions.
Here’s the part that blew my mind: there isn’t just one AI crawler. There are 10 of them, spread across four companies, and they all do completely different jobs.
The 3 Types of Robots Reading Your Website Right Now
I had no idea this distinction existed until this week. Once I understood it, the whole “why am I invisible to AI” mystery made sense.
1. The Student — these bots read your content to train the AI’s brain. Examples: GPTBot, ClaudeBot, Google-Extended. Your knowledge gets absorbed, but you may never get credited.
2. The Researcher — these bots fetch live, current information the moment someone asks a question, and they cite you with a link. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot. This is the one that actually builds your authority.
3. The Courier — these only show up when a real person specifically asks the AI to go read your page right now. Examples: ChatGPT-User, Claude-User, Perplexity-User. Low traffic, high intent.
Most people lump all of this into “AI stuff” and move on. But if you don’t know which bot does what, you can’t control any of it.
The 5-Minute Fix
Here’s the wild part. Every website has a tiny, almost-always-ignored file called robots.txt. It’s like a bouncer standing at the door of your website, deciding who gets let in.
Most robots.txt files were written years before AI crawlers even existed. So by default, a lot of websites are accidentally blocking the exact bots that would make them visible to AI — without the owner ever knowing.
Lloyd Pilapil, founder of the AI marketing firm Pixelmojo, shared a story that says it all: his own company was completely invisible across all four major AI platforms — and the cause was a two-line rule in their robots.txt file blocking GPTBot and ClaudeBot. The fix took five minutes. The citations started appearing three weeks later [1].
Five minutes. Three weeks. That’s the entire ROI timeline on a problem most people don’t even know they have.
The Plot Twist: Sometimes Blocking Bots Gets You MORE Citations
This is the part that genuinely surprised me, and it flips conventional wisdom on its head.
In March 2026, Pixelmojo deliberately blocked 12 AI training bots — and their citations went UP, not down [2].
Why would blocking bots help you get cited more? Because training bots crawl your site constantly but almost never send anyone back to you. The data is wild: Anthropic’s training crawler has a 38,000-to-1 crawl-to-click ratio. That means 38,000 visits from the bot for every single human visitor it actually sends your way. Perplexity’s search bot, by comparison, runs at about 194-to-1 — far more generous [2].
So the team blocked the bots that take without giving, and kept the ones that cite with a link. Less server strain, same visibility, smarter strategy.
Then in May 2026, they refined it even further — selectively allowing specific training bots (the ones feeding AI platforms they actually wanted recognition from) while still blocking data brokers and aggressive scrapers [2].
The takeaway for the rest of us: your robots.txt file isn’t just a technical setting. It’s a strategy decision. And almost nobody is treating it that way yet.
Writing So AI Actually Wants to Quote You
Getting crawled is step one. Getting cited — meaning the AI actually mentions you by name in its answer — is the real prize.
Here’s the thing: AI doesn’t read the way humans do. It doesn’t enjoy your warm three-paragraph intro. It scans for clean, standalone chunks of information it can lift directly into its answer.
The fix is something researchers call structuring content “answer-first.” Put a clear question as your heading. Right below it, answer in 40 to 60 words, with no links cluttering the first part. Then go deeper with examples and data underneath.
This isn’t just a guess — it’s backed by real research. A study from Princeton and Georgia Tech, presented at the KDD 2024 conference, tested nine different optimization techniques across 10,000 real queries. The single biggest lifts? Adding statistics and citing sources — each one boosted AI visibility by 30 to 40% [3].
Translation: if your content reads like a Wikipedia answer instead of a personal essay, AI is far more likely to use it.
Why Any of This Actually Matters
Here’s the number that made me sit up: AI referral conversions are reportedly five times more valuable per session than traffic from a regular Google search [4]. People who land on your site after an AI specifically recommended you tend to already trust you before they even arrive.
That’s not a small SEO tweak. That’s a new front door to your entire personal brand.
Your Move This Week
- Go to
yourwebsite.com/robots.txtright now. See if AI bots are blocked. If you don’t know how to read it, paste it into ChatGPT and ask it to explain it to you in plain English. - Pick your single best article or LinkedIn post. Rewrite the opening as a direct, 40-60 word answer to the exact question your audience is asking.
Most people in your industry haven’t even heard of half of this yet. That’s not a threat — that’s your opening.
Sources:
[1] Lloyd Pilapil, “How to Get Cited by ChatGPT, Perplexity, and Claude,” Pixelmojo, April 2026 [2] Lloyd Pilapil, “We Blocked 12 AI Bots: What Worked, What Changed,” Pixelmojo, March–May 2026 [3] Princeton University / Georgia Institute of Technology, Generative Engine Optimization study, presented at KDD 2024 [4] “How to Track AI Citations Across 4 LLMs,” Pixelmojo, 2026
Author
Charu Mehrotra
Founder Womenlines
Follow Womenlines on Social Media
