Robots.txt is Pretty Damn Important, Yes
Posted on 09.30.07 by Widge @ 5:52 am

Just found this.

Technically, it's correct: you don't need a robots.txt for good SEO.

However, it's not that simple. Part of the problems I was having on my sites was that the search engine bots weren't just crawling my site, they were freaking pounding it into a fine powder. Oh sure, if you've got a big enough server, you can afford to let them run all over you–but I'm doing this crap on a budget.

If you're on a budget hosting service, or to put it another way, if you're using the cheapest hosting you feel you can get away with–you have to make sure you're not throwing away bandwidth or CPU cycles.

Look at your access logs. Are you getting hammered every couple of seconds by Googlebot? Or the Yahoo bot? Or any bot for that matter?

If you have a robots.txt, are the bots reading it and heeding it?

It's one thing if you've got flat HTML pages for your site, but even with wp-cache running, Wordpress can bog down if a bot is allowed to run rampant. And if your site is slow or can't be crawled properly because the bots have bogged it down, then yeah, that can affect your SEO.

Now you know. And knowing is half the battle.


Taggification: , , , , ,

Filed under: Fun With SEO
Comments:

« Hacking Quicktags: All Links Are Not Created Equal | Radiohead Blowing People's Minds With Variable Pricing » No Comments »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a comment

Line and paragraph breaks automatic, e-mail address never displayed, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

(required)

(required)



John Robinson is a writer of prose, poetry and comics who also writes under the pseudonym of Widgett Walls.

Widgett Walls is the director of Needcoffee.com who also writes under the pseudonym of John Robinson.

Don't ask.


This is my latest book. Short stories written especially for you, or at least someone who reminded me a lot of you at the time.

Read it for free here. Or if you like paper, buy it here.

Then tell all your friends about it. Or all your enemies. I'm not particular either way.


Want a translation? Try these:

They're not perfect, but they'll do in a pinch.







Syndication
Feedburner

Amigos
Sarah Brown
Catalyst
Dindrane
Doc Ezra
Tee Quillin
ScottC
Tibby's Bowl
JM Tuffley

Sites Which Distract Me From Writing
Boing Boing
Cringely
Defamer
Warren Ellis
Engadget
Fortean Times
Long Tail
Porphyre
Reason
Wired

Topics
General BS
Insomnia
Travel
Writing Fodder

Active Projects...
Dark Blue Monstropolis
Magnificent Desolation
Something Else
The Sunday Before You


Recent Entries
  • And Now That I've Upgraded to 2.6...
  • Don't Mind Me.
  • Amazon MAB Replacement?
  • My New Mascot
  • A Nice Coda to the Trip
  • The New Yorker Hotel Business Center
  • Blast From the Past
  • Crossposted From My StumbleUpon Blog
  • Update at Last
  • George Clooney Makes Small Films Profitable. Yes.
  • On the Other Side of the Flu
  • Piano and Trumpet For the Win
  • Gun, With Occasional Weightlessness
  • Tor Nørretranders on Permanent Reincarnation
  • Can I Get a Hell Yeah?

  • Wordpress Archives
    July 2008
    May 2008
    April 2008
    March 2008
    February 2008
    January 2008
    December 2007
    November 2007
    October 2007
    September 2007
    August 2007
    July 2007
    June 2007
    May 2007
    April 2007
    March 2007
    February 2007
    January 2007
    December 2006
    November 2006
    October 2006
    September 2006
    August 2006
    July 2006
    June 2006
    May 2006
    April 2006
    March 2006
    February 2006
    January 2006
    December 2005
    November 2005
    October 2005
    September 2005
    August 2005
    July 2005
    June 2005
    May 2005
    April 2005
    March 2005

    Credits and Copyright
    Proudly powered by WordPress. All content © 1997-present by John Robinson.
    Theme by Theron Parlin, but we've mangled it beyond all reason. So don't blame him.