Help! It’s the attack of the robots! Well, the robots.txt. I’m talking about the bots & web crawlers from MSN, Yahoo, Google, and the other search engines that come through everything time and again to check on your website. I’m sure you’ve more than likely seen them in your stats. Maybe you even thought they screw up your stats and skew them. Well here’s the good news about the robots.
The more content you create, the more often the robots come back. The robots come back when new content is created so they can crawl through it and place your precious information in the right places in the search engines. So you are looking through your stats and noticing robots coming back more frequently, or looking back on past stats and seeing an abnormally high volume of hits from robots…. Great job! That means you created more than normal content and the robots are back.
Now when you create a lot of content not only do the robots come by more often, it means that you have more content that has been added to the search engines. More content for users to read, more content in search engines, these things more often than not pay off to you the creator of this content in the form of more visits to your site!
Now go attract those robots!