本文以英语撰写。
An XML sitemap lists preferred URLs for discovery, while robots.txt controls crawler access. Neither a sitemap entry nor a crawl rule guarantees that a page will be indexed.
Check one page across the discovery signals
Choose an important service URL and compare its internal links, sitemap entry, canonical tag and indexing directives. Check the response after any redirect. Use URL Inspection to investigate the engine's reported result for that exact page.
Key takeaways:
- sitemap.xml helps search engines discover important URLs
- robots.txt controls crawler access to URLs and directories
- A sitemap does not guarantee that a page will be crawled or indexed
- Blocking a URL with robots.txt does not necessarily prevent the URL from appearing in Google
- You can reference your XML sitemap inside robots.txt
- Important indexable pages should normally be crawlable and included in your sitemap
Check that your sitemap and canonical tags agree
Choose a sample of important URLs and compare the sitemap entry with the canonical tag on the destination page. Remove URLs that now redirect or are no longer intended for search.
If only your navigation is translated while the article stays in English, do not present those duplicate articles as complete language translations. Keep the preferred article URL consistent across internal links and the sitemap.
Use Search Console URL Inspection to investigate Google's reported canonical and indexing status, then follow our sitemap submission guide.
What Is the Difference Between Robots.txt and Sitemap.xml?
The main difference between robots.txt and sitemap.xml is their purpose.
Sitemap.xml helps with discovery. Robots.txt controls crawling.
| Feature | sitemap.xml | robots.txt |
|---|---|---|
| Main purpose | Helps search engines discover URLs | Controls crawler access |
| Typical location | example.com/sitemap.xml | example.com/robots.txt |
| Can encourage URL discovery? | Yes | Can point crawlers to a sitemap |
| Can block crawling? | No | Yes |
| Guarantees indexing? | No | No |
| Can prevent indexing? | No | Not reliably |
| Typical format | XML | Plain text |
| Should contain important URLs? | Yes | Usually rules rather than a URL inventory |
Think of your website as a building.
Your XML sitemap is the directory showing search engines where the important rooms are.
Your robots.txt file is the access policy telling crawlers which areas they should or should not enter.
They work together, but one cannot replace the other.
What Is an XML Sitemap?
An XML sitemap is a machine-readable file containing URLs that you want search engines such as Google to know about.
A basic XML sitemap might look like this:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
</url>
<url>
<loc>https://example.com/services/</loc>
</url>
<url>
<loc>https://example.com/blog/</loc>
</url>
</urlset>
The sitemap gives search engines a list of URLs they can discover and potentially crawl.
However, putting a URL in a sitemap does not guarantee that Google will crawl or index it.
A sitemap is a signal that helps search engines discover and understand your site's URLs. Google still decides which pages to crawl and index based on its own systems.
What Is a Sitemap Used For?
XML sitemaps can be particularly helpful for:
- Large websites with many URLs
- New websites with relatively few external links
- Websites with pages that are difficult to discover through internal links
- Ecommerce sites with large product catalogues
- News, image or video-heavy websites
- Sites that frequently add or update content
Even with a sitemap, you should maintain good internal linking.
A sitemap should support your site's crawlable structure rather than compensate for important pages that are effectively hidden from users and search engines.
What Is a Robots.txt File?
A robots.txt file is a plain-text file containing instructions for web crawlers.
It normally lives at the root of the website:
https://example.com/robots.txt
A simple robots.txt file could look like this:
User-agent: *
Disallow: /admin/
Disallow: /search/
Sitemap: https://example.com/sitemap.xml
In this example:
User-agent: *applies the following rules to crawlers generallyDisallow: /admin/tells applicable crawlers not to crawl URLs under/admin/Disallow: /search/blocks crawling under/search/Sitemap:tells crawlers where the XML sitemap can be found
This makes robots.txt primarily a crawler-management tool, not a page-indexing tool.
Sitemap.xml vs Robots.txt Example
Imagine an ecommerce website selling shoes.
Its XML sitemap might include:
https://example.com/
https://example.com/mens-shoes/
https://example.com/womens-shoes/
https://example.com/products/running-shoe/
https://example.com/blog/how-to-choose-running-shoes/
These are pages the business wants search engines to discover.
Its robots.txt might contain:
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /internal-search/
Sitemap: https://example.com/sitemap.xml
The two files are therefore communicating different things:
Sitemap.xml: "Here are important URLs on my website."
Robots.txt: "Here are areas crawlers should not access."
That is the fundamental difference between a sitemap and robots.txt.
Does Robots.txt Prevent a Page From Being Indexed?
Not necessarily.
This is one of the most important distinctions in technical SEO.
A robots.txt rule can prevent Googlebot from crawling a page, but that does not necessarily prevent Google from knowing that the URL exists.
For example, if other websites link to a blocked URL, Google may discover that URL through those links.
The result is that the URL can potentially appear in Google's search results even though Google cannot crawl the page content.
Robots.txt vs Noindex
If your objective is to prevent a normal HTML page from appearing in Google Search, robots.txt is generally not the right mechanism.
A noindex robots meta tag can be used instead:
<meta name="robots" content="noindex">
There is an important catch:
Google needs to be able to crawl the page to see the noindex directive.
Therefore, blocking the same page with robots.txt can prevent Google from accessing the noindex instruction.
This is why crawl control and index control should not be treated as the same thing.
XML Sitemap vs Robots.txt vs Noindex
These three SEO mechanisms are often confused.
| Tool | Main Purpose |
|---|---|
| XML sitemap | Helps search engines discover URLs |
| robots.txt | Controls whether crawlers can access URLs or paths |
| noindex | Tells supporting search engines not to index a crawled page |
A useful rule of thumb is:
Want Google to find an important page? Include it in your sitemap and link to it internally.
Want to control crawler access? Consider robots.txt.
Want an accessible page excluded from Google Search? Consider noindex.
Do not use robots.txt as a substitute for noindex.
Can You Put a Sitemap in Robots.txt?
Yes.
In fact, referencing your sitemap in robots.txt is a simple way to make its location available to crawlers that support the directive.
For example:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
The sitemap value should be a complete URL, including the protocol and hostname.
If you use a sitemap index, you can reference that instead:
Sitemap: https://example.com/sitemap_index.xml
A website can also reference multiple sitemap URLs where necessary.
How to Set Up an XML Sitemap
1. Generate Your Sitemap
Many modern content management systems can generate XML sitemaps automatically.
Depending on your setup, this may be handled by:
- Your CMS
- An SEO plugin
- Your ecommerce platform
- Your website framework
- A dedicated sitemap generator
Automatic generation is generally preferable to manually maintaining a large sitemap.
2. Include the Right URLs
Your sitemap should focus on the URLs you want search engines to discover and potentially index.
Generally, avoid filling your sitemap with:
- Redirecting URLs
- Broken or 404 URLs
- Duplicate URL variations
- URLs you deliberately do not want indexed
- Internal search results
- Non-canonical duplicates
Your sitemap should provide search engines with a clean list of useful URLs rather than every URL your server can generate.
3. Keep <lastmod> Accurate
If your sitemap uses the <lastmod> field, make sure it reflects meaningful page updates.
Do not change every URL's modification date every day simply to make content look fresh.
4. Submit the Sitemap to Google Search Console
You can submit your sitemap through the Sitemaps section of Google Search Console.
For example:
https://example.com/sitemap.xml
This helps Google discover the sitemap and gives you visibility into whether Google can process it.
5. Reference It in Robots.txt
You can also add:
Sitemap: https://example.com/sitemap.xml
to your robots.txt file.
How to Set Up Robots.txt Correctly
1. Put It in the Correct Location
For a normal website, robots.txt should be accessible at the root:
https://example.com/robots.txt
Putting a file at a URL such as:
https://example.com/blog/robots.txt
does not make it the robots.txt file for the whole example.com host.
2. Understand User-Agent
The User-agent directive identifies which crawler the rules apply to.
For example:
User-agent: *
Disallow: /private/
The * applies the group to crawlers generally.
3. Use Disallow Carefully
For example:
Disallow: /private/
tells applicable crawlers not to crawl URLs under that path.
Be extremely careful with:
Disallow: /
because it tells the applicable crawler not to crawl the entire site.
This is particularly worth checking after launching a staging website into production.
4. Use Allow Where Necessary
An Allow directive can permit crawling of a more specific path within a disallowed section when supported by the crawler.
For example:
User-agent: *
Disallow: /documents/
Allow: /documents/public/
5. Add Your Sitemap
Finish with the absolute sitemap URL where appropriate:
Sitemap: https://example.com/sitemap.xml
Should a URL Be in the Sitemap If It Is Blocked by Robots.txt?
For normal SEO pages, generally no.
Including an important URL in your sitemap suggests that you want search engines to discover it, while blocking it in robots.txt tells crawlers not to access it.
That creates conflicting intentions.
For pages you want indexed, a cleaner setup is generally to:
- Make the page crawlable
- Use a self-referencing canonical URL where appropriate
- Include it in your XML sitemap
- Link to it internally
Your sitemap should primarily contain canonical URLs that you actually want search engines to discover.
Do I Need Both Robots.txt and Sitemap.xml?
Many websites benefit from having both, but they are not mandatory simply because a website exists.
You May Benefit From an XML Sitemap If:
- Your website contains many pages
- Your site is new
- Some important pages have limited links pointing to them
- Your content changes frequently
- You manage large numbers of products, articles, images or videos
You May Need Robots.txt If:
- You need to manage crawler access
- Your website generates unnecessary crawlable URL patterns
- You have sections that crawlers should not access
- Your technical SEO strategy requires crawler-specific rules
A small, well-linked website may still be discoverable without an XML sitemap, but having a properly maintained sitemap is often straightforward and useful.
Common Sitemap and Robots.txt Mistakes
1. Blocking the Entire Website
This rule deserves particular attention:
User-agent: *
Disallow: /
It blocks applicable crawlers from crawling the entire host.
This sometimes happens when a development or staging configuration accidentally reaches production.
2. Using Robots.txt to Hide Sensitive Information
Do not treat robots.txt as a security mechanism.
The file is publicly accessible, and disallowed URLs can still potentially be discovered.
Sensitive or private information should be protected through proper authentication and access controls.
3. Blocking Pages That Use Noindex
If Google cannot crawl a page because robots.txt blocks it, Google cannot access a noindex meta tag on that page.
If your goal is deindexing, make sure the crawler can access the page long enough to process the indexing directive.
4. Putting Broken URLs in Your Sitemap
Regularly check that sitemap URLs resolve correctly and represent pages you actually want indexed.
5. Including Redirects in the Sitemap
Where practical, sitemap entries should point directly to the canonical destination rather than URLs that redirect elsewhere.
6. Including Non-Canonical URLs
A sitemap should generally contain your preferred canonical URLs rather than duplicate variations.
7. Assuming a Sitemap Guarantees Indexing
Submitting a sitemap helps with discovery.
It does not force Google to index every URL in it.
8. Assuming Robots.txt Guarantees Deindexing
Robots.txt controls crawling.
It should not be relied upon as a guarantee that a URL will disappear from search results.
How to Check Your Sitemap and Robots.txt
You can perform several basic checks yourself.
Check Robots.txt
Open:
https://yourdomain.com/robots.txt
Look for unexpected rules, particularly:
Disallow: /
Then check whether important sections of your site have accidentally been blocked.
Check Your Sitemap
Open your sitemap URL, which might be:
https://yourdomain.com/sitemap.xml
or:
https://yourdomain.com/sitemap_index.xml
Check whether it loads successfully and contains the correct canonical URLs.
Check Google Search Console
Google Search Console can help you:
- Submit and monitor sitemaps
- Inspect individual URLs
- Investigate indexing issues
- Identify URLs affected by robots.txt restrictions
- Understand how Google sees important pages
Use the URL Inspection tool when troubleshooting a specific URL.
Frequently Asked Questions
What is the difference between robots.txt and sitemap.xml?
Robots.txt controls crawler access, while sitemap.xml helps search engines discover URLs.
Robots.txt can tell crawlers not to access particular paths. An XML sitemap lists URLs you want search engines to know about.
Is sitemap.xml the same as robots.txt?
No.
They are separate files with different purposes. Sitemap.xml is primarily used for URL discovery, while robots.txt provides crawling rules.
Does robots.txt help SEO?
It can support technical SEO when used correctly by helping manage crawler access.
However, blocking important content by mistake can seriously harm crawling, so robots.txt should be configured carefully.
Does an XML sitemap improve SEO rankings?
An XML sitemap does not directly make a page rank higher simply because the URL appears in the file.
Its main benefit is helping search engines discover URLs and understand changes more efficiently.
The page still needs to be crawlable, indexable and sufficiently useful and relevant to compete in search results.
Can robots.txt block Google?
Yes.
Robots.txt can contain rules that prevent Googlebot from crawling particular paths or, if configured broadly enough, an entire site.
Can a page blocked by robots.txt still appear on Google?
Yes.
A blocked page's URL may still be indexed based on information Google discovers elsewhere, although Google cannot crawl the blocked page to understand its content normally.
Should sitemap.xml be listed in robots.txt?
It is useful to include the sitemap location in robots.txt.
For example:
Sitemap: https://example.com/sitemap.xml
This gives supporting crawlers another way to discover your sitemap.
Where should sitemap.xml and robots.txt be located?
Robots.txt normally needs to be at the root of the host it controls:
https://example.com/robots.txt
A sitemap can use different valid locations depending on its scope and configuration, although:
https://example.com/sitemap.xml
is a common setup.
Can I have multiple XML sitemaps?
Yes.
Large websites often use a sitemap index that references multiple individual sitemap files, such as separate sitemaps for products, articles, categories or other content types.
XML Sitemap vs Robots.txt: Final Comparison
If you remember only one thing from this guide, make it this:
XML sitemap = discovery.
Robots.txt = crawling control.
Noindex = indexing control.
They solve different technical SEO problems and should not be used interchangeably.
A healthy setup usually makes important SEO pages easy for search engines to discover and crawl while restricting unnecessary crawler access where there is a genuine reason to do so.
Need Help Checking Your Technical SEO?
A small mistake in robots.txt or your XML sitemap can affect how search engines discover and crawl large parts of your website.
Our SEO team can review your sitemap, robots.txt and technical SEO setup to identify crawling and indexing problems that may be limiting your organic visibility.