How to Add a Custom Robots.txt File in Blogger {Updated}

Robots.txt file is crucial to index a website on search engines. It instructs how to crawl and index web pages. Blogger not only has a default robots.txt file but also allows you to customize it according to your needs, giving you the flexibility to adapt to changing SEO requirements. Using a custom robots.txt file is essential if you are serious about optimizing your site for search engines. In this tutorial, I will show you how to add custom robots.txt file in blogger.

What is a Robots.txt File?

A robots.txt file is a simple text file located in the root directory of a site. It contains the instructions for web crawlers. These instructions guide search engine bots on how to crawl and index a website. By using a custom robots.txt file on a Blogger site, you can restrict bots from accessing specific pages.

Key Components of a Robots.txt File

Understanding the following parts of a robots.txt file will help you make a custom robots.txt file for your site.

User-agent: Media partners-Google

This line is essential if you are running Google AdSense ads on your blog. It ensures that Google can deliver relevant ads to your site. It is best to leave this line intact even if you are not using AdSense.

User-agent: *

This line marked with an asterisk (*) applies to all robots. It serves as a catch-all for general instructions.

Disallow: /search

The above "Disallow" directive prevents search engines from indexing web pages that contain search query results. It can be beneficial if you don't want these pages to appear in search engines.

Allow: /

The above "Allow" directive instructs search engines to crawl and index your blog's homepage.

Sitemap: https://example.com/sitemap.xml

Including your sitemap in the robots.txt file helps search crawlers to locate and index your blog content easily.

How to Add a Custom Robots.txt File in Blogger

1. Go to your blogger dashboard.

2. In the left sidebar, click on the "Settings" tab.

3. Scroll down and find the "Crawlers and indexing" section.

4. Enable the "Custom Robots.txt" option.

5. After enabling the Custom robots.txt option, a text box will appear to paste your robots.txt file code.

6. Click on the "Save" to save your changes.

That's it! You have successfully added a custom robots.txt file to your blogger blog.

How to Check the Robots.txt File in Blogger

1. Open a browser such as Chrome or Firefox.

2. Enter your blog's URL in the address bar and put "/robots.txt” after the URL.

3. Press Enter to access your robots.txt file.

For example, if your blog's URL is "http://example.blogspot.com", then you have to type "http://example.blogspot.com/robots.txt" in the address bar and hit enter to access the robots.txt file.

How to Disallow Specific Content

If you want to prevent some posts or pages from being indexed by search engines, then follow the steps below:

Note: Be careful with Disallow directives; any misconfiguration can prevent search engines from indexing your entire site.

To disallow a specific post:

Disallow: /yyyy/mm/your-post-url.html

Replace "yyyy/mm/my-first-post.html" with the url slug of the post you want to block.

To disallow a particular page:

Disallow: /p/page-url.html

Replace "p/page-url.html" with the URL slug of the page you want to prevent from indexing.

Final Thoughts

Customizing the default Blogger robots.txt file is a straightforward process. However, always remember, robots.txt is a very sensitive file. A properly configured robots.txt file can improve your blog's visibility on search engines. And any misconfiguration can block search engine crawlers from accessing your website.

WikiPoka made a robots.txt file generator for blogger sites to simplify the process of creating a custom robots.txt file. So, don't forget to try this excellent tool.

If you found this article helpful, please consider sharing it with your fellow bloggers. Your support can help others who may be facing similar challenges. And if you have any questions regarding this tutorial, feel free to ask in the comments section.