Getting Found Online Organically - Spiders, Crawlers and Indexing

Main Content


getting found online organically - spiders, crawlers and indexing

Good SEO optimization is not enough if you are just starting out as a creator wanting to get your content found on the Internet. To really start to get your websites and blog sites found organically on the Internet, you must have at least a baseline understanding of spiders and webpage indexing. The first thing to know is that, if a search engine has not internally indexed and stored information about your web page or other online content, searchers using that search engine just will not find you.

Organic Traffic Course

So how do you get indexed across search engines? Well, there are a few ways that this occurs. First, you can manually access search engines and ask them to index your pages - before they will do this, and a point to keep in mind, you will probably also need to submit to them your sitemap.xml file as well.


Next is where spiders, also known as crawlers, come into the picture. Google has a famous "crawler" called Googlebot whose job it is to go out and crawl websites around the Internet to get information about them for storage in their search engine database - the "Indexing" process referred to above. When you go into the Google Webmaster Toolkit and submit your sitemap.xml file and then ask Google to check out your webpages and index them for their search engine database - and you use the "Fetch as Google" tool to do this - it is the "Googlebot Crawler" that gets the order to go out and get these jobs done.

As the largest search engine on the Internet, the Googlebot crawler is kept extremely busy so don't expect this job to get done immediately - it often can take around 2-4 weeks before Googlebot gets around to actually processing your submission and crawl requests.

By the way, it is not just your webpage titles, keyword tags and meta-descriptions that get indexed when your website gets crawled. Your content on the page gets crawled and stored and listed within the search engine as well. This means that a top-down scan of your webpage text - or some of your webpage text is done, Image titles and alternate titles, anchor text and hyperlinks, etc. are also listed and stored to determine the "Authority and Content Value" of the webpage crawled. Depending on what the crawler finds on your page goes a long way to determining how your page will rank within the search engine itself when a searcher is looking to obtain information.

In this regard, here are a couple of important tips to keep in mind as you build out your pages.

  • First, make sure you have a few good keyword phrases near the top of your article, blogpost or page text that align with your page title, meta tags and meta description.
  • Second, it is a good idea to "Bold" these keyword phrases in your text on your page - this makes it easier for the crawlers to spot it as they pay extra attention to title lines within a page when crawling. Bolding this text will help show its importance and relevance to the page and your meta tag descriptions.
  • Third, use Anchor Text and hyperlinks as well - preferably within the first couple of paragraphs of your content as you just never know how deep down the page the spider search will go - so help it out in its job as much as you can. Do your best to get it to like your page.
  • Next, be sure to try to code your web pages according to the W3C compliance standards as this is something that these search spiders look at as well. Thee are many so called "web designers" out there in the world today building websites using templates of one form or another. Google and other search engines are aware of this trend and they have built coding tests into their spiders to separate the "men from the boys" so to speak. Basically - they penalize those that are building amateur websites and are rewarding those writing webpages professionally with higher rankings organically within their search engines.

Remember that you are competing against millions of other websites out there these days and everyone is using the same kinds of keywords, meta tag descriptions, page titles, etc. If you want to stand out, you have a much better chance of being found organically if your webpages are professionally constructed.

To view the W3C standards for webpage coding by the way, go to and start doing a little studying. Then, go back and take a look at the coding of your webpages and do what you can to improve them. A couple of non-professional coding approaches that come to mind include: Overuse of javascript callouts - such as you find frequently in WordPress widgets, older versions of the Adobe Flash player, building non-responsive/non-mobile compliant web-pages, not adding the final "/" in a web-domain address and slow-loading pages that could be caused by a number of things - like building "fat" web pages, insertion of large sized images and videos, non-closed tags within a page, etc.

I have built webpages both ways by the way. When I started out with my own site, I was in a hurry to get up and running so I bought a couple of templates, learned them and published them. I could never get any decent amount of traffic to come to these sites organically so as I found the time to rework them - basically this meant for me rewriting them "natively and from scratch", I started to see my organic search results start to improve. By the way, remember to ask search engines to recrawl revamped web pages once your rework is done - otherwise they might still sit there without activity for months and even years into the future.

Click here or image above to get the book on Amazon- as low as $2.99!

In summary, remember to do the things that keep search engine spiders happy. As you grow, be sure to put a robots.txt file on your site to:

  • Give additional instructions to the spiders crawling your online locations;
  • Clean up all hard and soft error 404's for missing webpages and links;
  • Minimize or eliminate javascript callouts;
  • Keep your sitemaps current;
  • Whenever you make significant changes to a page, look for webpage "meta-tag" duplications across 2 or more pages and fix them - this sometimes happens if you copy one page to be used for another and you forget to update the metatags in the newly created page;
  • Check to ensure your page loading speeds remain within acceptable parameters;
  • Check your web server activity logs from time to time to see if spiders are actively crawling you; and,
  • If you are technical enough or your webmaster is, run your own spiders against your own sites to see if they are working properly and are not "blocking" spiders unnecessarily so they have to quit.

Good luck in building your traffic to your site.

By Dan Grijzenhout - April 29, 2017

Click the following title to view my "Article Masterlist"



Affiliate Marketing Class

Affiliate Marketing Class

FREE: Take Chapter 1 of my Affiliate Marketing class titled - Home Business Strategy Development. Click Above Icon

How to Get Found on the Internet - 50 % Off Coupon!

Want more online traffic? Take my "How to Get Found on the Internet" class at 50% off! Click above photo icon.

Entrepreneurship and Retirement Class

Take my "Using Entrepreneurship to Prepare for Retirement" class for FREE! Click above icon to go to class.

Get My Books on Amazon

I now have 7 books published on Amazon - 2 of them available in paperback form. Click the picture icon above to check out my books on Amazon.