Let’s connect
SEO Library Technical SEO 2 min read

Robots.txt

In short

Robots.txt is a plain-text file at the root of a website that tells search engine crawlers which URLs they may or may not crawl. It controls crawling, not indexing.

Last reviewed · Iftikhar Qureshi

On this page

What is robots.txt?#

Robots.txt lives at yourdomain.com/robots.txt and follows the Robots Exclusion Protocol. Well-behaved bots like Googlebot read it before crawling and respect its rules. It’s a set of instructions, not a security measure — anyone can read the file, and malicious bots can ignore it.

Why it matters for SEO#

  • Keeps crawlers out of low-value areas such as admin screens, internal search results and endless filter combinations.
  • Helps large sites use crawl budget on pages that matter.
  • Points crawlers to your XML sitemap.

Example for a WordPress site#

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/wp-sitemap.xml

User-agent names the bot (* means all), Disallow blocks a path prefix, Allow creates an exception, and Sitemap lists sitemap URLs.

Crawling vs indexing#

Important Blocking a URL in robots.txt does not guarantee it stays out of Google. If other pages link to it, Google can index the URL without content. To keep a page out of search results, allow crawling and use a noindex meta tag instead.

Common mistakes#

  • Disallow: / left over from a staging site — blocks everything.
  • Blocking CSS and JavaScript files Google needs to render pages.
  • Using robots.txt to “hide” private content.
  • Combining a robots.txt block with noindex — Google can’t see the noindex if it can’t crawl the page.

Test changes with the robots.txt report in Search Console. Related: indexing, technical SEO checklist.

Share this term