I have been working on WordPress sites for a couple of years now, but picked up a new account that has their site built in Drupal. While optimizing the website I ran across a nice little feature that controls the robots.txt file. If you’re not familiar with the importance of the robots.txt file, basically it tells the search engine which pages on your site are ok to index.

While going through my clients site today, I noticed that on each page the robots.txt file was set to no index, no follow. When Google, Yahoo or Bing come across this page, this little robots.txt file is telling them, “don’t index me and don’t follow me”! Well if you tell the search engines not to follow your website, then it will never be found during the search as you are telling them to remove the page from the search engines database!

Be sure to set up the robot.txt file TO follow and index each page of your website. Your be glad you did when the phones starts to ring as users are finding you on Google.

For you WordPress users, there is a plug-in that will create a robots.txt file for your website.

For more information on the robots.txt file, go to robotstxt.org.

About Ryan Perry

Ryan Perry has taken his 10+ years of business ownership and hands-on marketing skills and focused them on online marketing. In April of 2009, he started Simple Biz Support with an emphasis on Search Engine Optimization (SEO). Ryan is propelling local business websites to the top of Google, Yahoo and Bing resulting in increased market exposure and revenue for his clients using a variety of internet marketing tools including blogs, article submissions and video. Additionally, Ryan speaks and vlogs (video blogs) about internet marketing, educating business owners how to effectively use various SEO tools and techniques to promote their business on the internet. Ryan currently resides in Santa Rosa, CA. Connect with Ryan on Google+