The robots.txt file tells crawlers which URLs they may request. It is small, powerful and easy to get wrong: one bad line can stop Google crawling your whole site. Here is how to use it safely.

robots.txt controls crawling, not indexing; noindex controls indexing
robots.txt manages crawling. It is not a way to hide pages.

From my own work: In audits I regularly find staging-site rules like Disallow: / left on a live site after launch. It is the first file I check when a site suddenly loses traffic.

What robots.txt does (and doesn’t)

It sits at https://example.com/robots.txt and follows the Robots Exclusion Protocol. Google uses it to manage crawl traffic, not to keep pages out of results (Google intro). A blocked URL can still be indexed if other sites link to it. To keep a page out of Google, use noindex and allow crawling so Google can see that tag.

Syntax basics

User-agent: *          # which crawler the group applies to
Disallow: /private/    # do not crawl this path
Allow: /private/ok/    # exception inside a disallowed path
Sitemap: https://example.com/sitemap_index.xml

Rules are case-sensitive paths. Google supports * and $ wildcards and picks the most specific matching rule (details).

A safe WordPress example

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/sitemap_index.xml

Don’t block /wp-content/ or /wp-includes/: Google needs CSS and JS to render pages.

AI crawlers

You can allow or block AI crawlers by user agent, for example GPTBot, ClaudeBot, CCBot or Google-Extended (which controls use in Google’s AI models, not Search). Blocking them does not affect Google Search rankings. Decide based on whether you want to appear in AI answers; see answer engine optimisation.

Common mistakes

  • Disallow: / left from staging.
  • Blocking CSS/JS so Google can’t render.
  • Using robots.txt to “noindex” pages (unsupported).
  • Blocking pages that carry a noindex tag, so Google never sees it.
  • Wrong file location (must be at the root of each host and protocol).

Testing

Check the robots.txt report in Search Console, inspect important URLs and watch crawl stats after changes. For deeper crawl analysis, see crawl budget and server logs.

Want this done on your own server or site?

I am Saizul Amin, a web developer and technical SEO consultant. I set up and harden VPS servers, migrate WordPress sites, fix indexing problems and build automations. See my portfolio or get in touch.

FAQ

Does every site need robots.txt?

No, but a simple one with your sitemap is good practice.

Can robots.txt remove a page from Google?

No. Use noindex or remove the page; robots.txt only controls crawling.

Does blocking AI bots hurt SEO?

Blocking GPTBot or ClaudeBot does not affect Google Search; blocking Googlebot would.

Sources

NEW BLOGS. REAL STRATEGIES. REAL RESULTS.

Join our community and receive powerful SEO tips, web optimization guides, and growth strategies as soon as they’re published.

Don’t just read blogs — stay ahead of the competition.

We don’t spam! Read our privacy policy for more info.

Share.
Leave A Reply