August 19th, 2010 by Dan

Have you ever really wondered how search engines can know what’s on the web? Let’s face it, if you take into account the billions of pages online and the constant introduction of new material that takes place, there’s an awful lot of information on the web. There’s an almost unimaginably vast amount, in fact.

If you’re  interested in getting your site onto the search engine results pages, you’ll want to learn as much as you can about the ways in which search engines investigate the Internet and your content there.

People Power?

So, do human employees of Google sit down to systematically read and appreciate all of it? They don’t. What the search engines have done, and continue to do, is devise software to travel the net. This type of travelling is called crawling, and the programs that move around the internet are robots, spiders or simply crawlers.

An SEO agency consists of professionals concerned with making your site as appealing as possible in the eyes of search engines. As an important part of this process, they’re occupied by learning about robots and how they operate. As a general rule, if you want to get the most out of your site in terms of sending the right signals to the robots it’s best to employ a good search engine optimisation company. How they work is a field of expertise in itself, and given that the search engines are loath to disclose their secrets to the general public. Figuring out the best ways to get results can be said to be an art form in itself.

Robots

Some basic facts about robots are now firmly established. It’s common knowledge that they like to read text. Flash, drop down menus, JavaScript and frames are not good because robots have problems seeing content in those forms. Experts from any good search engine optimisation company will tell you this and advise on ways around the problem if you really must feature these things on your site. One very popular method of pleasing robots as well as users is to have Flash content AND HTML text on pages, so that human users can follow the visually appealing features and robots have their own routes to follow.

Humans

It’s worth remembering that not all human users choose to view images on screen. HTML-only versions of pages are chosen by users wishing to increase the loading speed of pages. Appealing to robots helps you to get your content seen by these users at the same time.

The most important things for robots to be able to see are links. Links are the paths down which robots travel to view and send back information on your pages. If they can’t read a link, it may as not exist as far as search engine optimisation’s concerned. No wonder SEO experts are so concerned with robots’ ability to read!

With this in mind, a report from an SEO company will invariably advise you to beware of links that require human interaction beyond the click of a mouse button. If a good portion of your site’s available only to visitors who fill in an online form or answer questions you’d better be aware that search engines probably can’t index the pages beyond the form. Part of good SEO is planning. Design the site with SEO and robots in mind to avoid creating dead ends and missing out.

Robots are like people in that they pay close attention to titles. Like humans who judge books by their titles, robots can discern a lot about your site from the titles. For our purposes, the title referred to here is inside title tags of HTML on pages (in human terms, the text in the uppermost box on the screen). If robots see that this anchor text features key words from searches conducted by users, it will increase your chances of getting your site into a good position.

Conclusion

Though it’s good to be aware of how robots operate and why they like to read, if you want to really enhance for search engine optimization it’s useful to consult a respected SEO company. Robots send information back to search engines which use complex algorithms to determine how relevant your site is and how popular. There are a great many ways in which a website can be enhanced, and they’re changing all the time.

Remember, get indexed, give robots something to read!

Link to us

If you want to link to this blog, copy and paste the following HTML code to your website.