Understanding robots and spiders

September 10th, 2008 by Lax

Search engine works by taking pieces of information on websites and storing it in a massive database.When a search is performed, the submitted search words are sent to database that returns a set of records,matching the user’s request,and generating a results page.

The information in the search engine database is collected by software called Robots and Spiders.The robots and spiders move throughout the web from hypertext links on web pages and return formatted information from these sites to the main search engine.

A search page contains around 10 results in any search engine as in below image.So the trick is to ensure that your site makes the results list!

Search engines share these commonalities:

  • A link to a web site that contains the searched word
  • A description of the site
  • The full URL of the web site that been returned

If a robot or spider has not investigated your site, you will not be in the search database.A robot can be forced to visit your site by completing a registration form with a search engine.

Robots are always searching web sites.If a robot visits your site, many methods are available to ensure that the most amount of information about your site is exchanged with the robot.

Finally you need to create a robots.txt File to instruct search engine robots about what pages on your website should be crawled and what page shouldn’t be.This robots.txt file is to ensure that your admin or personal files not being able to searched by engines so that not being published to normal visitors or search users.Daniel explained how to create that robots.txt file in his blog.

Happy Blogging !

Got something to say? Comment here…

Related posts:

  1. What is Duplicate Content Problem and How to tackle it?
  2. Is your Blog Working Properly?
  3. SEO Copywriting Tips and Tricks

8 Responses to “Understanding robots and spiders”

Leave a Reply