Blogger is adding robots.txt by default recently

If you use Google Sitemaps for your Blogger and you are seeing a sudden increase in the number of links being blocks by robots.txt, don't panic as it is not anything you did wrong. It is all because Blogger is adding robots.txt by default recently. As you can see from the codes below, all pages under the /search directory is being disallowed meaning and pages under the /search directory would not appear included in any search result pages of major search engines (e.g. Google, Yahoo and Live.com).

Read more…

---

Some times back when blogger.com upgaraded to new Google blogger, new label feature was added in blogger blogs. Each lable has its own page. This robot file make sure that lable pages are not crawled and indexed by any search engine bot. This will certainly help search engines to avoid crawling millions of duplicate lable pages.

To check the robots.txt file of your blogger blog, just type the following URL in the adress bar of your browser.

http://www.yourblogname.blogspot.com/robots.txt
(replace yourblogname with your blog name).

Read more…