Success on the Internet

Following is day 2 of our SEO tips mini course:

Normally a web page / web site is found by the act of a search engine crawler following a link from another page that’s currently listed in the search engine index. A crawler (or spider) is a virtual robot that crawls along, following links from one page to the next, archiving those pages into a search engine database, to be later analyzed for the purpose of ranking and indexing.

In the beginning of time for a crawler, the beginning point is suspected to be either the DMOZ and/or Yahoo directories, so if you can get listed in one or both of those directories, it is highly beneficial.

Your first goal in search engine optimization (SEO) is simply to make sure your new pages or new site is linked to another page that is already indexed by the major search engines. Once a page is indexed, it is revisited by the crawler or spider occasionally to see if anything is new.

The frequency of visits is determined mostly by the importance of the site and how often it changes content. Some sites are crawled monthly, some daily or even more frequently. If new pages you produce are linked from a page that is crawled daily, it will be found and subsequently indexed much quicker than if it were linked from a less important page.

This is one reason why blogs and RSS feeds are becoming so important and why Internet experts are becoming so excited about them. A blog by definition is something that is updated (actively added to) frequently, and the search engines absolutely love that. Of course bloggers that blog infrequently don’t reap those benefits because the crawlers become used to the update frequency after a while.

In a prior post I mentioned that I could get Yahoo to notice a new blog entry very quickly (within minutes of uploading it sometimes), and pages linked from those posts will be crawled soon thereafter.

RSS feeds are loosely related to blogs, because it is a mechanism whereby a page is constructed that point to multiple blog entries (or sometimes just multiple related pages on a site). If you have an RSS feed on your site, other sites have the opportunity to syndicate your feed to interested parties. The act of syndication benefits both parties, a subject for another post one day.

It is much easier at this point in time (late 2004) to get quickly listed in blog directories than it is to get listed in normal web directories. Therefore, one way to get a newly constructed web site indexed relatively is to put up a blog on the site, get the blog listed in the blog directories, link the home page of your main site to the blog, and soon enough the crawler will find its way to your entire site.

Now you have an idea how to get a new page found by the spiders. On my next post, I’ll discuss what factors influence how favorably your web pages are ranked by the search engines. See you then.