"Robots.txt & XML Sitemaps: How Search Engines Crawl Your Site"

Two small files decide how search engines see your entire website: robots.txt and your XML sitemap. One tells crawlers where they may go; the other tells them what's worth visiting. Get them right and Google spends its time on your important pages. Get them wrong and you can vanish from search results overnight — it happens to real companies every year.

How Search Engines Crawl Your Site: Crawl Budget Basics

Googlebot doesn't crawl the entire web every day. For each site it allocates a crawl budget — roughly, how many URLs it will fetch in a given period, based on your site's authority, server speed and how often content changes.

On a 30-page portfolio, crawl budget is irrelevant. On a site with thousands of URLs — e-commerce filters, faceted navigation, calendar pages, session parameters — it matters a lot. If Googlebot burns its budget on /search?sort=price&page=847, it may take weeks to discover your new product pages.

Your two levers:

1. robots.txt — blocks crawlers from wasting time on low-value URL patterns.
2. XML sitemap — points them straight at the URLs you actually care about.

Robots.txt Syntax: The Four Directives That Matter

robots.txt lives at exactly one place: the root of your domain (https://example.com/robots.txt). It's plain text, case-sensitive on paths, and read top to bottom in groups started by User-agent lines.

• User-agent — which crawler the group applies to. * means all; Googlebot, Bingbot, GPTBot target specific bots. A bot uses the most specific group matching it and ignores the rest.
• Disallow — a path prefix the bot must not fetch. Disallow: /admin/ blocks everything under /admin/. An empty Disallow: means "allow everything."
• Allow — carves an exception out of a broader Disallow. When rules conflict, the longest (most specific) matching rule wins for Google.
• Sitemap — an absolute URL to your sitemap. It stands outside groups and applies globally; you can list several.

Google also supports two wildcards: * (any sequence of characters) and $ (end of URL). Disallow: /*.pdf$ blocks every PDF.

The Robots.txt Generator builds a valid file from checkboxes — common bots, standard exclusions, AI crawler opt-outs — so you don't fight the syntax by hand.

The classic disasters

Real-world robots.txt failures follow the same few patterns:

The Disallow: / accident is common enough that it should be part of every deployment checklist. A weekly pass with an SEO Audit catches it, along with missing meta tags and indexability problems, before the traffic graph does.

Robots.txt does NOT prevent indexing

This is the most misunderstood point in technical SEO. Disallow blocks crawling, not indexing. If other sites link to a disallowed URL, Google can still index it — showing the bare URL with the infamous "No information is available for this page" snippet.

Worse: if you add a noindex meta tag to a page and disallow it in robots.txt, Google can never crawl the page to see the noindex. The page stays indexed.

The rules:

• Want a page out of Google? Let it be crawled and serve <meta name="robots" content="noindex"> (or an X-Robots-Tag: noindex header). Once deindexed, you may optionally disallow it.
• Want to save crawl budget on pages that were never going to rank (cart, internal search, parameter chaos)? Use Disallow.
• Sensitive content? Neither. robots.txt is public — listing /secret-admin/ in it is an announcement, not a protection. Use authentication.

XML Sitemaps: Structure, lastmod, and When You Need One

An XML sitemap is a machine-readable list of the URLs you want indexed:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-07-08</lastmod>
  </url>
  <url>
    <loc>https://example.com/blog/robots-txt-guide</loc>
    <lastmod>2026-07-10</lastmod>
  </url>
</urlset>

What actually matters in 2026:

• <loc> — required. Absolute, canonical URLs only. Don't list redirects, 404s, or noindexed pages: a sitemap full of junk teaches Google to trust it less.
• <lastmod> — the one optional tag Google genuinely uses, if it's accurate. Setting lastmod to today on every URL at every build destroys its value. Update it only when content meaningfully changes.
• <priority> and <changefreq> — ignored by Google. Skip them.
• Limits — 50,000 URLs or 50 MB uncompressed per file. Beyond that, split into multiple sitemaps referenced by a sitemap index file.

Do you need one? If your site is under ~100 pages with clean internal linking, Google will find everything anyway. You do need one when the site is large, new (few backlinks pointing in), has deep pages far from the homepage, or content that changes often. Given how cheap it is, the pragmatic answer is: just have one. The Sitemap Generator produces a valid sitemap.xml from a list of URLs, with correct lastmod formatting.

Submit it in Google Search Console and declare it in robots.txt — the two mechanisms are complementary.

Keep the sitemap honest

A sitemap referencing dead URLs is worse than no sitemap. After any restructuring or migration, run the Broken Link Checker across the site: it flags 404s and redirect chains that should be removed from the sitemap or fixed at the source.

AI Crawlers: Opting Out of GPTBot & Friends

A new decision since 2023: do you let AI companies train on your content? These crawlers respect robots.txt and identify themselves:

Key nuances:

• Google-Extended controls AI training use only — blocking it does not affect your Google Search rankings.
• Blocking GPTBot stops training crawls but AI search products may use separate agents (e.g. OAI-SearchBot) — decide per use case: training opt-out vs. appearing in AI search answers.
• This is business strategy, not SEO hygiene. Publishers monetizing original content often block; SaaS docs teams often allow, because being cited in AI answers drives signups.

Generate Your Robots.txt Now

The Robots.txt Generator builds a clean, valid file in seconds — pick the bots, the paths to block, add your sitemap line, copy the result. No account required.