Robots.txt File

What is robots.txt?

A robots.txt file is a public text file at a site root that gives participating crawlers rules about which URL paths they may request. It helps manage crawling; it does not control who can access a page or guarantee whether a URL appears in search results.

Examples

User-agent: *
Disallow: /

This rule asks participating crawlers not to request URLs on the site. It is not a security barrier, and a blocked URL may still appear in search results without its page content.

A robots.txt file can set different crawl rules for different crawler user-agent groups. Syntax and rule handling can vary between crawlers.

Examples

User-agent: *
Disallow: /
Allow: /public/

This asks matching crawlers not to request the site, except for paths under /public/. Google applies the most specific matching rule; other crawlers may interpret rules differently.

How to make robots.txt file?

To create a robots.txt file for your website, follow these steps:

• Open a notepad and type User-agent: *
  This line tells all web search robots which rules to follow. We can also specify specific search robots name by replacing the *.
  For example, User-agent: Googlebot, this would apply the rules only to the Googlebot.
• Add a rule such as Disallow: /private/ to ask crawlers not to request matching paths. An empty Disallow: means no paths are disallowed for that group. Rules control crawling only; they do not restrict public access or guarantee removal from search results.
• Repeat steps for any additional pages or directories that want to allow or disallow on the website.
• Save the file as robots.txt and upload it to the root directory of website.
• Example of a complete robots.txt file:
  User-agent: *
Disallow: /private/
Allow: /public/

Uses of robots.txt file :

Use robots.txt to manage crawling for appropriate URL paths. Keep in mind its limitations:

• Managing crawl requests: Ask compliant crawlers to avoid URL paths that do not need to be crawled, such as certain duplicate or low-value areas.
• Do not use it for privacy: Protect private pages with authentication or access controls. Anyone can view robots.txt, and a disallowed URL may still be indexed.
• Know its indexing limits: A crawl rule is not a reliable way to remove a URL from search results. Use an appropriate indexing control, such as a noindex directive on a crawlable page, or restrict access when content must be private.
• Allow required resources: Avoid blocking files such as CSS or JavaScript when crawlers need them to understand the page.

Examples

Example 1
User-agent: * Disallow: / This asks participating crawlers not to crawl the site; it does not prevent access or guarantee that URLs stay out of search results.

Example 2 User-agent: *
Disallow: An empty Disallow rule allows crawling by crawlers that honor this robots.txt syntax.

Example 3 User-agent: BadBot
Disallow: / This asks a crawler identifying as BadBot not to crawl the site.

Example 4 User-agent: *
Disallow: /private/ This asks matching crawlers not to request URLs under /private/; it does not protect those URLs.

Example 5 User-agent: *
Disallow: /contact.html This asks matching crawlers not to request the path /contact.html. Paths are relative to the site root.

Read the Google Search Central robots.txt guide for crawler-specific behavior and indexing limitations.

Remember: robots.txt is public guidance for crawlers that choose to follow it. It is not access control, and a disallowed URL may still be listed in search results. Use authentication for private content.