So How do Search engines work Anyway?

Everyone knows that when you’re going to look for something on the web, you first visit a search engine website, such as Google or Bing, and type in what you’re looking for. Ta-dah! You instantly have results, and likely the information that you were looking for. It all seems rather magical, but search engines work off of real world principles.

A search engine is basically the index you use to find information on the World Wide Web. The web itself is far too large, and individual websites too narrow to navigate without using tools to filter and organize information. There are two main types of search engines: crawler-based and human-based.

Crawler-based engines, such as Google, automatically create listings. If you make any changes to your website, a crawler-based engine will eventually find these changes, but it will affect how you are listed. Elements such as page titles and body text also affect your listings.

A crawler engine works by a spider (also called a crawler) computer program visiting a website. It reads the page and visits links within the website. The spider periodically returns to the website, usually somewhere between two weeks to a month, to look for any changes. Everything that the spider finds on the website goes into the search engine’s index. The index or catalog contains a copy of every webpage that the spider visits, and they are massive databases. Google’s index contains around 38 billion pages, as estimated on June 3rd, 2011 by World Wide Web Size (the actual number is constantly changing). If a webpage changes, then the new version is entered into the index. This can take some time, from when a webpage is “spidered” but not yet “indexed,” but until a webpage is indexed, it cannot be found by the search engine. This is why it is recommended for new businesses to build and list their websites before opening their doors to business; it gives search engines time to get your website listed online with them.

The last part of a crawler search engine is its software. The program sifts through millions and millions of webpages to find matches to your query, and then lists them from most relevant to least. This is the familiar results page that you see when using a search engine yourself.

All search engines have these components, but each one applies a different value to certain areas through its computer algorithms. That is why you can search for the same thing in one engine, and get an entirely different result from another.

When a spider visits a webpage, it records information such as how many times certain words are repeated, where the words are located, such as in the header or body of the webpage, and it weighs how relevant the page is to the subject, based on the surrounding words. This last portion is the hardest job for a search engine to do, and this is what makes a search engine effective, and thus, popular. HowStuffWorks has a great article on comparing some of the largest search engines’ spiders. The Google spider, Googlebot, was built to index every significant word on a page, leaving the articles “a,” “an” and “the.” The Lycos spider is said to use the words in the title, sub-headings and links, along with the 100 most frequently used words on the page, and each word in the first 20 lines of text. AltaVista’s spider indexes every single word on a page, including “a,” “an,” “the” and other insignificant words.

The actual method that each algorithm uses are actually trade secrets. WiseGEEK has this to say: “The algorithms that various search engines use are well protected, to prevent people from specifically creating pages to get better ranks, or at least to limit the degree to which they can do that. This difference is why different search engines yield different results for the same terms. Google might determine that one page is the best result for a search term, and Ask might determine that same page is not even in the top 50. This is all just based on how they value inbound and outbound links, the density of the keywords they find important, how they value different placement of words, and any number of smaller factors.”

A human-based directory, such as Mahalo, uses editors to write descriptions about websites, or depends on users to submit their own listings. Any search only lists exact matches in these descriptions to your query. If you change anything on your website, it will not affect your listing at all. The benefit of human-based directories is that they are evaluated by actual people, thus tend list websites of higher value and relevance to search users. However, there is a tendency for listings to go ‘stale’ due to a lack of updates, and search results are much smaller than with crawler-based engines.

Many modern search engines use a hybrid of these two types, according to Search Engine Watch, such as MSN Search blending results from human-powered LookSmart and crawler-based Inktomi for more obscure requests.

The cutting-edge in search engine technology is using concept-based algorithms, as opposed to keyword-based. This means that the search engine will suggest webpages that are related topically to your search term, as well as direct matches to your inquiry, by using statistical analysis (think Netflix’ widely-touted recommendations algorithm). Research is also being done in natural-language queries, that is, typing in a question exactly the way you would ask it to another person. The most popular natural-language search engine is Ask Jeeves, but it is presently restricted to only simple questions.

Search Engine Optimization (SEO) works to maximize the placement of your website for different search engine results, so you will get more ‘hits’ from visitors using that search engine. We’ll talk more about this in later articles, but this is a tool that most businesses will want to use to promote their business online.

Search engines are marvelous everyday tools of computer technology, and a key to making your business’ website successful.

How google works

 Janet Houck is the Senior Editor for the Small Business Online Toolkit
She has been a freelance editor and published author for almost a decade.
Read more about her on her SBwebtoolkit Bio page

Is It Time? Deciding if your business is ready for an online presence.

We live in exciting times, where more and more people are looking to start a new business and work for themselves. In both good and bad times, the American economy is driven by the success of small and medium-sized businesses. In this era of smartphones and “googling” for information on businesses, as opposed to using the Yellow Pages, the public expects businesses to have an online presence. The lack of an online presence can deter customers from your service and products, and limit the amount of advertisement exposure your business will have overall. Many people exclusively use the Internet when looking for business services, and without any online presence, you will be missing an ever-growing group of potential customers.

A website gives you greater control of your public image. Instead of being at the mercy of customer review websites such as Yelp, you will be able to communicate directly to potential and long-time customers, such as relaying up-to-date information, hours of operation and location of your physical storefront and answering commonly-asked questions. This also lends itself to a time savings for you, as customers can be referred to the website for information.

Mindie Burgoyne asks, “If you often wish you could communicate effectively with a broader range of clients / customers, publicly post answers to frequently asked questions, attract new customers, break in to new markets and take the market share from your competitors – then a website is a MUST!”

A website allows you to set yourself apart from your competition and to discuss the unique of your service or products (especially if your main competitors do not have a website), and you will be able to update information much faster and easier than with traditional media, such as magazines and newspapers. You will be able to direct search engine traffic towards your business, thus increasing your exposure. Additionally, a business website also allows you to establish yourself as an authority with your customers and within the industry, thus leading to greater credibility. This is especially true when it comes to businesses in specialized fields. It also gives you the opportunity to generate a mailing list of potential and existing customers by gathering website visitor statistics.

You will also gain the potential for online sales. Tim W. Knox advises business owners not to be too quick to dismiss the feasibility of offering products and services online. “Nowadays, there's very little that can't be sold over the internet. ...I'm not saying you should put all your efforts into selling your wares over the internet, though if your product lends itself to easy online sales, you should certainly be considering it. The point to be made here is that you should at the very least have a presence on the web so that customers, potential employees, business partners and perhaps even investors can quickly and easily find out more about your business and the products or services you have to offer.”

Although all businesses should have a web presence, there are a few situations where a website is not ideal. If you are currently at your capacity to serve your customers, and are not looking to expand, your web presence will be focused more on building trust and communicating with your current clientele. In this situation, you will be centered more on maintaining customer connections through social media, such as Twitter, Facebook and blogging. “I love blogging because I build relationships with new people and with potential customers and existing customers, but that’s not the actual business,” says LaShanda Henry, owner of Multiple Shades of You Online. If a business finds much of its clientele through foot traffic or word-of-mouth, they would want to center more on building and maintaining detailed online listings, such as Google Places.

It’s important to take time to examine your web presence goals before building a website. A website centered on building an online community is far different than a web storefront. As with any major business endeavor, it requires a good deal of forethought and pre-planning.

All businesses benefit from having a website to base their online presence. Whether you are focusing on doing business only locally or are intending to be worldwide through web commerce, your business will gain from reaching out into the online world.

Janet Houck is the Senior Editor for the Small Business Online Toolkit
She has been a freelance editor and published author for almost a decade.
Read more about her on her SBwebtoolkit Bio page