Comments on: Understanding robots and spiders

By: Anish K.S

Anish K.S — Fri, 30 Oct 2009 21:53:19 +0000

Your Blogspot allowing to upload robots.txt ?.

Een Rockstar — Sun, 25 Oct 2009 10:27:25 +0000

Creating a robot.txt File will help us a lot..It will force robot to crawl or not to crawl

Earn Money At Home — Tue, 06 Oct 2009 05:26:24 +0000

Wow! thank you for explanation. I never knew before how robots and spiders works…

I hope robots and spiders index my site… hehhehe

Kumar — Sun, 04 Jan 2009 10:19:28 +0000

Thats worth to read and bookmark.

Shafar — Sun, 04 Jan 2009 09:32:36 +0000

Useful info!

Admin — Wed, 10 Sep 2008 12:00:16 +0000

Giving which directory to search and which shouldn’t is very important in robots.txt File.

Admin — Wed, 10 Sep 2008 08:28:53 +0000

To restrict that you need to create a text File called robots.txt and upload it into root directory..

john — Wed, 10 Sep 2008 08:25:42 +0000

how to ristrict spider from crawling in one folder.
good work. keep it up.