I went out to find an image for Needcoffee from the archives we've already got on the site so I wouldn't have to grab it from elsewhere (remember, kids: recycle!) and realized that some of our archival pages weren't loading properly on the new server.

Brief explanation: on Version 3 of the site (pre-Wordpress) we used frames. We're not proud of it, but it made sense for 1998. The content portion of the site was an .htm file. The frame that went around it and contained the masthead and the menu/sidebar was an .html file. So you could have, for example, a DVD review of Drop Dead Fred and the review itself was ddfred.htm while the frame that held it was ddfred.html.

Again, it was the best we could muster at the time.

Anyway, while checking out the new server for this old image I wanted, I discovered that the .html versions of pages were coughing up a lung. "An error occurred while processing this directive." Lovely. But the .htm versions were fine. What to do?

Well, tear open a .html file and saw that AXS, the web tracking bit we used to use on the old site (and still use in the archives) wasn't working because it was pointing to an old CGI directory and the CGI directory wasn't kosher any longer.

So the first thing I tried to do was get the old CGI directory to work, but I couldn't seem to figure out how to do that. And I wanted to get the damn thinking working. So I decided, hey! Screw it, I'll just to a rewrite via the .htaccess and when people go to the .html version, I'll just send them to the direct review instead. That's actually better from an SEO perspective because back when we setup the whole framed system SEO wasn't even a twinkle in anybody's pants.

But somehow I botched it and the .htaccess blew up the site for a few minutes. I had a Redirect that never redirected anywhere, and it didn't matter that it was in my /html/dvd/ directory, the server read down through .htaccess, couldn't go any further and barfed.

Why did this happen, children? Because Uncle Widge fucked himself over. He got in a hurry and forgot the cardinal rule of screwing around with your site:

The Cardinal Rule of Screwing Around With Your Site:

That's right. When you forget the Cardinal Rule, Yahweh himself will laugh at you and your site will explode.

So. I finally got that corrected after a few minutes outage, and I was able to find some info on how to do this in .htaccess.

Not that I think anybody out there has the same setup as the old Needcoffee.com site, but still. You can futz with this to match your own site.

This is a variation on what I found at SEOBook, which was the best and closest to what I was trying to do.

Disclaimer:

So.

I basically have an .htaccess in the subdirectory where I want to make this change.

I added this to the top:

Options +FollowSymlinks
RewriteEngine on
RewriteRule ^(.+)\.html http://www.needcoffee.com/html/$1.htm [r=301,nc]

Here's what I can tell you. The Options and RewriteEngine statement make the thing work to begin with. The rule itself says for every .html hit, rewrite and resend them instead to .htm. Also, it gives a 301 redirect, which is good because search engines need to be able to find the content that's been "moved," or in this case, simply bypassed.

The one thing I don't understand is, the way I read this code, it looks like it should only work for the /html/ directory, but it instead works in all subdirectories. So that I'm clueless about. Just for the moment it works.

There? Aren't you sorry you asked now?

Update: Shade at That's My Stapler tries to help my inept ass out. I think I almost understand what he's saying…I can do coding, me.

WTF is a Frame Breaker?

Well, you know how when you search in Google Images, you get presented the site below a Google frame up top? The Frame Breaker breaks you out of that frame, natch, so you get served the site without the Google flavoring up top.

I always did wonder about that, so I flicked it on. That was around the 22nd or 23rd of October.

I decided to come back and check to see how it had affected my traffic on Google Images.

Now, let me state this up front. My understanding is that Google Images doesn't update very often. And I also understand that for the majority of my images, I haven't SEO'd them up worth a good goddamn because at the time I was putting them up, I had no idea why I should. (Of course, people still seem to find me and force me to do shit like this.)

So I don't have a great deal of traffic coming in anyway. But let's look.

Google Images spiked the day I turned it on, then went to a reasonable trickle.

As for Google Images.ca? I went from a trickle…to nothing.

Google Images.uk? From an erratic mess to…nothing.

And pretty much on down the line.

Did it do anything to my regular Google results? No. No discernable change.

In fact, if I just do "images" and pull that chart up, I spike, and then crater.

Fascinating. Now. One of two things is happening.

1. Either Google Images doesn't like the frame break and has something built in which makes me show up lower in the results because of it. Or…

2. Google Images needs the frame to show up as Google Images in my Analytics.

I don't know which. If I had a huge amount of Google Images traffic, I would be able to see if my Google hits went up an equivalent amount, so maybe Images traffic was being counted as regular traffic. Just a theory.

Or, if I had Analytics installed on my Version 3 archival part of the site, which has no Frame Breaker, I could see if it suffered, yes or no.

I've heard rumors that breaking the frame caused Google Images to not speak to you anymore, but never really saw that substantiated. And trying to Google terms like "frame breaker google images penalty" and the like didn't really get me anywhere.

Either way, I'm taking off the Frame Breaker. I'll see if it changes anything and if so, how quickly. And hopefully, I'll get Google Images out here to reindex my site with the SEO'd bits I do have.

If anybody has their own experience, I'd love to hear it.

So I went and added Daily Kicksplode to my thumbs up bits, stating up front that it was one of mine.

Boom. 1200 visitors arrived. Nice.

Now, before anybody gets all excited and starts bombarding Stumbleupon with their own sites, I would caution you as follows:

1. The "Don't Be a Dick" rule is a no-brainer. If you show up, register, and post your site and then leave, no one will ever pay attention to you, I wouldn't think. We never allow that crap on the Gabfest, the hit-and-run promotions nonsense, and I'm sure the Stumbleupon community won't put up with it either.

2. If you're going to use the service, then use it. And use it to your benefit. If you del.icio.us link something, Stumble it too. The people who run the sites, if they're smart, will appreciate you doing so.

This has certainly got me active again on Stumbleupon because it gives results, obviously. And I like what gives results.

I should have a report on Project Wonderful here in a couple of days.

Technically, it's correct: you don't need a robots.txt for good SEO.

However, it's not that simple. Part of the problems I was having on my sites was that the search engine bots weren't just crawling my site, they were freaking pounding it into a fine powder. Oh sure, if you've got a big enough server, you can afford to let them run all over you–but I'm doing this crap on a budget.

If you're on a budget hosting service, or to put it another way, if you're using the cheapest hosting you feel you can get away with–you have to make sure you're not throwing away bandwidth or CPU cycles.

Look at your access logs. Are you getting hammered every couple of seconds by Googlebot? Or the Yahoo bot? Or any bot for that matter?

If you have a robots.txt, are the bots reading it and heeding it?

It's one thing if you've got flat HTML pages for your site, but even with wp-cache running, WordPress can bog down if a bot is allowed to run rampant. And if your site is slow or can't be crawled properly because the bots have bogged it down, then yeah, that can affect your SEO.

Now you know. And knowing is half the battle.

An individual post can appear:

• As itself
• In a category (as many as it has categories assigned to it)
• In a tag page (as many as it has tags assigned to it)

So if I've got a review of a TV DVD set, it could appear in the categories: TV, DVD and Reviews. Plus if I've got five tags on it, well…you see the concern. Without even meaning to, I've got the same article showing up nine places.

What to do? Well, initially I went into my robots.txt file and just told the search engines to stay the hell away from everything but my individual posts:

Disallow: /tag/
Disallow: /category/

Then the pendulum swings back the other way–are all of my pages close enough to my front page that they get nabbed by Google and are considered important? Basically, if nobody externally is linking to a post, and it's too many hops/links away from the front page, it can be abandoned and left for dead. Or something equally dramatic.

So the happy medium is to have a post, split it up with the MORE tag, and then allow categories and individual posts in the robots.txt file, but then only have one category per post.

Ah, so: problem for me. I can have a post, as stated above, that's for a TV DVD set. Now I can see saying that DVD can be chucked because primarily it's a TV item, the media just happens to be DVD. Fine. But, well, it is a review. And what if somebody wants to browse all our reviews? Do I really want them to have to use the tag?

Then it struck me like a slice of provolone from the blue: "What a dumbass, just exclude the individual categories that you know are always going to be tied to something else." For example, we will always have Reviews AND TV. Or Reviews AND Movies. Just exclude that category. So I re-allowed /category/ and instead just did this:

Disallow: /category/reviews/
Disallow: /category/press/

Done deal. Now, granted, I have some posts with multiple categories that need to be cleaned up, but that can be done easily enough.

I wish there were a lot more SEO tools built in with WordPress, honestly. And maybe this exists among the fifty gajillion plugins and I haven't seen it, but a way of looking at all my posts and being able to check/uncheck categories en masse would be nice. Or even a plugin that went out, looked at how you've got your posts, robots.txt and such setup, and graded you for duplicate content. You know, you are at a 56% chance of being SOL because you've got too little content too many places. Something like that.

Anyway, What Have We Learned?

1. There are no absolute, hard and fast rules to SEO. And even that, being a hard and fast rule, is subject to scrutiny. Sure, you need to do stuff like use decent titles, decent URLs, and have your server, you know, actually online. There's some no brainers, but just because you find a post that says Your Robots.TXT Must Look Like This or You Are Doomed, well, have your grain of salt handy. Make sure what you're doing works for your individual site, because as I find, anyway, most SEO posts are for sites that aren't, shall we say, dealing in the trade of pop culture.

2. You can control a shitload of stuff about your site. I'm amazed at how many people actually don't have a robots.txt file. Or an .htaccess file (that they know of). I keep forgetting just how much power I have to shape what goes on on the site. It's a good idea to take five minutes a week and step back from the grind of posting and just go, "Right. Do I have my hatches battened down?"

3. Google Webmaster Tools are your friend. The robots.txt analyzer they provide has already saved me from fifteen really stupid things I could have done to cut my site off from the outside world. I highly recommend you do not make any changes to your robots.txt without running it through their first. And don't just check the Googlebot. Check the image-bot and check the media-bot (if you're running AdSense).

193.47.80.77 – - [14/Aug/2007:01:25:22 -0700] "GET /updates/tag/androids HTTP/1.0" 301 242 "-" "Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)"

Apparently there's been some other Exabots in the past, but nothing that's specifically "Exabot-Thumbnails." There's no info on a site to check for who the hell owns this bot, nor, from what I can tell, is it nabbing my robots.txt file. So…evil bot = .htaccess smackdown by IP address.

If anybody knows anything more about this, let me know.

For those who don't know (and probably don't care, if you're not a webmaster), there is a secondary set of search results you can get from Google. It's called Supplemental Results. It might as well be called "The Results That Aren't As Good As The Real Results." Nobody but nobody wants to be in them.

A couple of months ago, I noticed that a goodly number of Needcoffee's entries had wound up in the Supplemental Results. At first, it appeared that this was because we had a lot of duplicate content: tag pages, category pages, date pages–all with the same posts. All right, fair enough–I setup a robots.txt that kept the Googlebot from indexing pages that I didn't want, and kept single entries as indexable.

However, stuff continues to slide into Supplemental Results. Right now I was toying with internal links to try and get things under control, but basically Google has effectively blinded me to how well I'm doing. The name of the article should have been more properly called "Supplemental Goes Stealth."

This doesn't fix anything. In fact, it makes my job as a webmaster even more difficult.

It would be one thing if there was a webmaster tool that said, "Hey, Widge, here's what's wrong with your page and why it slid into Supplemental Hell." Then I would go and fix it. However, now I not only don't know why this is happening I can't even see it happening any longer. So the problem has just gotten a lot worse. Google's solution to the problem is simply to make it impossible to see the problem. But the problem hasn't gone away.

This, frankly, sucks. And this is me, Google enthusiast and defender, talking here. Why is Google doing this? I run AdSense on Needcoffee. Why would they make it harder for people to find pages on my site and thus harder to get at the ad revenue that I could potentially bring in? And this is not just my site–AdSense is all over the place, and this affects everybody's sites. It would be in Google's best interests, I would think, to provide us with the tools so we can make our sites work better with their search engine, so everybody wins. Again, I'm not one of this whiny assholes who thinks Google owes me this–they owe me jack crap. It's just hard to understand why they would respond to a problem by, instead of using their vaunted resources to throw at it, to make it look like it's gone away and hope nobody bitches.

Somebody help me understand how this is a good idea.

If you have verified the link and you are putting it on your site, do not put a nofollow on it. It's bad form.

Well, this threw me for a loop. And that's not a coding joke.

Back before I even started seriously screwing around with SEO, I ran across a few blog posts (and I won't even begin to fathom where they were) that seemed to know what they were talking about that explained the problem with bleeding PageRank. Basically, if you link out to a page, you give them a skoche of your PageRank. At the time, PageRank seemed like something you wanted to hang onto. And in my case, since I was trying to build a site properly, hang onto for dear life.

The way to get out of this was to stick a rel="nofollow" on your outbound links. This would tell the search engines that mojo leakage was not in order, thank you very much.

Okay, fine. Makes sense.

But here's where we learn that SEO is a lot like medical science. I shall explain.

One of my problems with doctors is that you get a lot of facts with your diagnosis but also a ton of opinion and interpretation. And also, they can't ever seem to agree on a lot of things. That's why red meat and caffeine are like Schrodinger's Diet–they exist in a "good for you/bad for you" state simultaneously, because nobody can make up their mind whether or not one or the other or both will kill you.

Same thing with SEO. Because it's a bunch of folks peering in through frosted glass at the inner workings of the search engines, they're guessing. And at the time, it seems like a lot of people were making honest guesses about nofollow and what it meant and what you should do about it.

But anyway, so this post at Black Hat really took the wind out of me, honestly. I've always considered myself a "black" hat SEO guy just because I only wear black, so I'm kind of one of them by default. But still–"bad form"? Fuck, the last thing I want to be accused of is bad form. I mean, if I'm going to be accused of being an asshole, I'd prefer it to be for something I meant to do.

So I started combing around. Here's this from Scobleizer. Which led me to this from Search Engine Journal:

Linking to someone with a NoFollow attribute is a sign of not trusting them. It's like reaching to shake someone's hand, but stopping to put on a pair of latex gloves.

Now, excluding for a moment that I might wear latex gloves when shaking somebody's hand because in my old age I'm finding I'm about three steps away from becoming Monk, still…FUCK. Now they tell me.

And upon reflection, since I am, after all, a Machiavellian bastard who wants to come out on top of everything (but at least I tell you this up front), I decided this makes sense. First of all, I don't want to look like an asshole unless I am an asshole. And I have plenty of other opportunities to be assholish that actually make sense. So. Second, I can't honestly tell you what PageRank does for me when it comes to Search Engine standings. I don't have the best PageRank in the world, and yet I seem to make out just fine. So. Fuck it.

I just went and did an uber-find-and-replace and all the nofollow shit should be gone on Needcoffee. Effective immediately. If you find anything that is, let me know. I'll fix it on here soon enough…I've got other stuff broken on here since the move.

Anyway, now that I've finally gotten a bunch of superfluous ne'er-do-wells gone from my access.log file, I can finally look at it and see what's happening moment by moment to try and address the problem. One thing I wanted to see, though, is what is really taxing the memory of my server space.

Trouble is, the access.log I get from Dreamhost (which I can only assume is the same sort you get from where you are) looks like this:

x.x.x.x – - [26/Apr/2007:00:36:50 -0700] "GET /wp-content/plugins/podpress/podpress_js.php HTTP/1.1" 200 2311 "http://www.needcoffee.com/2006/03/08/power-rangers-dino-thunder-vol-3-dvd-review/" "Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3"

Now, barring for a moment that someone is actually viewing a Power Rangers review and we must find them and stop them from breeding, imagine 10MB of that. That's how much I've got for a full day's access log, and that's after I've been working for a few days to optimize my robots.txt file.

Now if we want quick results to cut down on the really bad big files that we've got out there, what can we do? Especially when, like Needcoffee, we're looking at a site that's been around for ten years, and has scads and scads of pre-Wordpress material that hasn't been converted yet. That log is a mess.

Well, the obvious thing would be to sort the log file by size of the file being requested, and I've seen some sites promising perl scripts or whatever, but I thought there had to be an easier way.

And here it is.

1. Take your access.log and open it in a text editor. Now, granted, if you're looking to do a 10MB access log, Wordpad will cough up a lung so grab something like Editpad or the like, or just use a subset of the log.

2. Do a find and replace. You want to find a space, i.e. " " and replace it with a comma "," Since we don't care about any data that would get screwed up by doing this, go for it.

3. Save the file with the suffix of .csv

4. Open the file in Excel (or equivalent) as a text .csv file

5. This should put the info into a spreadsheet where you should have a column for size. On my version, it's column H. Sort by H and take a look.

In my case, once I get past the podcasts and such that are supposed to be large I find…wow, holy crap: there's a JPG on here that's 73KB that flat out doesn't need to be.

Also, prototype.js, which WordPress uses for the admin panels, is about that size as well. I wish somebody would create a stripped down, no FX, just want to get the shit done WordPress admin theme, for those of us who…well, just want to get the shit done.

Anyway, there you go. Enjoy.

If this is helpful, I may post more stuff like this as I find it.