Why user agents matter for SEO#
- Robots.txt rules target crawlers by user-agent token (
User-agent: Googlebot,User-agent: GPTBot). - Log analysis shows how often each crawler visits which URLs.
- Testing — you can switch your browser’s user agent in DevTools to see what a mobile visitor gets.
Common crawler tokens#
| Token | Owner / purpose |
|---|---|
| Googlebot | Google Search |
| Bingbot | Bing and Copilot search |
| Google-Extended | Controls use for Gemini training (doesn’t affect Search) |
| GPTBot / OAI-SearchBot | OpenAI training / ChatGPT search |
| ClaudeBot | Anthropic |
| PerplexityBot | Perplexity search |
| facebookexternalhit | Link previews on Facebook/WhatsApp |
User agents can be faked Anyone can claim to be Googlebot. Verify important bots with reverse DNS or published IP ranges before trusting or blocking them, and never serve search engines different content from users (that’s cloaking).
Related: Googlebot, robots.txt.