Understanding robots and spiders
Search engine works by taking pieces of information on websites and storing it in a massive database.When a search is performed, the submitted search words are sent to database that returns a set of records,matching the user’s request,and generating a results page.
The information in the search engine database is collected by software called Robots and Spiders.The robots and spiders move throughout the web from hypertext links on web pages and return formatted information from these sites to the main search engine.
A search page contains around 10 results in any search engine as in below image.So the trick is to ensure that your site makes the results list!

Search engines share these commonalities:
- A link to a web site that contains the searched word
- A description of the site
- The full URL of the web site that been returned
If a robot or spider has not investigated your site, you will not be in the search database.A robot can be forced to visit your site by completing a registration form with a search engine.
Robots are always searching web sites.If a robot visits your site, many methods are available to ensure that the most amount of information about your site is exchanged with the robot.
Finally you need to create a robots.txt File to instruct search engine robots about what pages on your website should be crawled and what page shouldn’t be.This robots.txt file is to ensure that your admin or personal files not being able to searched by engines so that not being published to normal visitors or search users.Daniel explained how to create that robots.txt file in his blog.
Happy Blogging !
Got something to say? Comment here…
Related posts:































how to ristrict spider from crawling in one folder.
good work. keep it up.
To restrict that you need to create a text File called robots.txt and upload it into root directory..
Giving which directory to search and which shouldn’t is very important in robots.txt File.
Useful info!
Thats worth to read and bookmark.
Wow! thank you for explanation. I never knew before how robots and spiders works…
I hope robots and spiders index my site… hehhehe
Your Blogspot allowing to upload robots.txt ?.
Creating a robot.txt File will help us a lot..It will force robot to crawl or not to crawl