Club Troppo

The updated copy of TinyMCE embedded in Wordpress 2.5 has bog ordinary indent and outdent buttons, of this approximate appearance: .

This button essentially just adds a few extra tags in the HTML that say “push this over to the right by 20 pixels”. That would be fine, except that it buggers up a bunch of other formatting rules set by the stylesheet.

The right thing to do is to use “semantic” markup. For quotes, one should use the blockquote tag. For lists, use the list tags. And so on: these are correctly laid out by the stylesheet which has been carefully tweaked by CSS gurus before I, or Wordpress, gets to mangle them.

If, like I do, you want to turn those buttons off in Wordpress, it’s a bit fiddly. The settings for the rich text editor are buried in wp-includes/js/tinymce/tiny_mce_config.php. Scroll down to the 128th line, which will look something like so:

$mce_buttons_2 = apply_filters('mce_buttons_2', array('formatselect', 'underline',

'justifyfull', 'forecolor', '|', 'pastetext', 'pasteword'...

This line tells the rich text editor what buttons to display. Delete or comment out this bit:

'|', 'outdent', 'indent',

And voila, the offending buttons go away.

General Whinge

This is essentially par for the course for Wordpress.org, of course. A shiny update which breaks things in stupid ways. They talk a lot about their you-beaut new interface (which I don’t think is much of an improvement on the old one), their fancy new uploading scheme (which is less convenient than the boring old one) and their exciting new … well as far as I can tell it was all about shiny things. Not so much about fixing boring bugs for the most part.

It annoys me because I have to bug Ken and Nick to let me upgrade to get access to security patches, which Wordpress.org don’t backport to older versions. Every time I hope it will be a smooth update. Every time I am made to look like a fool.

Jacques is advising us that we should upgrade to the latest version of Wordpress.   He may well be right, but Nicholas and I are nervous/remaining to be convinced. Apparently there are some potential security issues with the version we’re currently running.

Our concern is that just about every previous Wordpress upgrade we’ve installed has created more problems than it solved and led to seemingly endless headaches.  Moreover, we’re currently enjoying the only trouble-free run we’ve experienced with Troppo for more than 2 years, and we really don’t want to disturb it unless there’s a compelling reason to do so.

All advice/observations will be gratefully received.  What experiences have others had with upgrading to the latest version of Wordpress, which I gather is 2.5.1?  Did it create any headaches (especially for those running lots of plugins)?  Apparently version 2.5, which was only released a couple of weeks ago, itself introduced a new security flaw into the program, which 2.5.1 allegedly fixes - see what I mean?  We’re currently running version 2.3.3, which is working perfectly well and does everything we want.  Should we be overly worried about the security vulnerability?  Is this like “upgrading” from Windows XP to the markedly inferior Vista?  Or should we just cross our fingers and make the jump?

Edit: We’ve jumped the shark. In keeping with Murphy’s Law, my meticulous backing up beforehand has meant that the backups were unnecessary.

We’re running slowly because the database server our installation of Wordpress relies on has suddenly decided to go bananas. I’m trying to find out why but nothing really obvious is coming up. It’s got a solid 100% utilisation on one of our 4 CPUs but nothing appears on the process list. Any MySQL wonks out there who might have an idea of what in tapdancing buggery it’s up to?

Update: tried a few things to no avail. One of the scripts supplied by MySQL actually dropped the entire Club Troppo database and then choked, leaving it empty. Good thing I had a steaming fresh backup. Oh, but then Wordpress forgets a bunch of settings after being restored. Piddly stuff like which theme it’s using, or what plugins were enabled.

When you consider that the clowns who write this software secured $25 million in round B funding, the subprime crisis doesn’t seem like such a surprise any more.

In the end I had to manually kill MySQL and reboot the server.

On and off over the past few months I have received emails to say that our feeds don’t appear in aggregators like Google Reader or Bloglines. Or that they turn up late in big bunches. Or days in arrears.

Each time I would fire up my browser, navigate to the feed URL, confirm that the feed was feeding, and promptly blame Google or Bloglines; sometimes for variety I blamed Wordpress.

Then yesterday I got an email from James Andrewartha. He wrote:

http://clubtroppo.com.au/feed/ is sending a last-modified header of Thu, 01 Jan 1970 08:00:00 GMT which is messing up my bookmarks script. In other words, EPOCH FAIL!

James’s joke is a Unixy nerd thing — all computers running Unix, a Unix-derivative or a Unix-like OS measure time by counting the seconds elapsed since midnight, 1st of January 1970, known as the “beginning of the epoch”. The 8am GMT thing has to do with our server living in the Perth timezone.

Jokes aside, this was a big fat clue that I was wrong. Eagerly I scurried to my command line to test the claim:

(Continued)

A Troppo reader thoughtfully emailed me to point out that our clock was way out of whack. For some reason Wordpress had decided to set its time to GMT+22, which means that comments in the last day or so have all come to us (cue spooky theremin music) FROM THE FUTURE, duh duh daaa!Don’t be surprised if things show up in weird orders.

Update: Another helpful email, this time from our UCC liaison James Andrewartha. He has pointed out that our feeds are reporting that they were last modified on January 1st 1970. In his words, an epoch fail (that’s a nerd joke). It explains why so many feed readers have had trouble following our feed properly. When I get home today I will try and spelunk the Wordpress codebase to suss out what’s going wrong — and whether upgrading to the new release of Wordpress will fix it or not.

Our 50,000th non-spam comment has just been posted by Richard Phillips. Here’s a taste:

The thing about car locks and regulation demonstrates a huge legal issue, I think. It is, simply, that law that is written by legislators is almost inevitably too tight and does not leave enough wiggle room. We can see this if we compare the common law, and the way judges (with all their faults) are able to deal with just the issue in front of them, follow principles, and leave space for the law to develop.

For contrast we are up to 3,566 posts — and the famous Dr Gruen is just 3 posts away from overtaking the famous Prof. Parish as the leading poster at Club Troppo.

Our hosts at the UCC will be replacing a busted airconditioner tomorrow and warn that some machines may go offline. Make sure to lay in provisions of wit and wisdom to tide you over the gap in your normal reading.

An airconditioner failed in our host’s machine room today, causing the server to overheat and shut down. It seems to be fixed for now.

So far, every time I claim our server problems are licked, Troppo takes a dive. So in the spirit of contrariness, tonight’s work on migrating back to the previously worked-flawlessly-barring-power-or-network-failure configuration cannot, by definition, work. The site is broken forever, so it shouldn’t go down. Enjoy.

WA’s power supply is in deep shoite, so don’t be surprised if Club Troppo drops offline this afternoon.

Not long ago I thought our outage problems might be licked. I was wrong, and it seems like the problem is not ours alone.

Currently Club Troppo runs inside a virtual machine which runs on top of our physical server. The software we use to achieve this is called Xen. It uses a clever technique called paravirtualisation to give the advantages of virtual machines with less of a performance hit.

I and my UCC colleagues migrated Club Troppo to this arrangement to ease future problems for Club Troppo. A virtual machine is easier to migrate, backup and manage without hassle. Moving to virtual machine infrastructure meant that I would be able to have features such as hot failover, or to run test-only instances of Club Troppo to preview the latest exciting stupidities on new releases of Wordpress. It also meant that we could share some of the formidable computing power our physical server has with our hosts, the University Computer Club. Even on a very busy day, our server hardly breaks a sweat.

So much for the advantages. It turns out however that we picked the wrong approach, at the wrong time. A sudden shift in the software landscape left us with a buggy, unstable combination.

The problem is this. Xen’s paravirtualisation approach requires both the host operating system and the guest operating system to be modified. So to run Club Troppo we have a Xen-aware kernel running the in the virtual machine. It is specially modified to talk to the physical server’s Xen-aware kernel.

However, the company who support Xen development — XenSource — were acquired not long ago by a company called Citrix. Citrix makes their money in selling very expensive packages designed to help Windows to catch up with stuff Unix nerds have used for 20 years. They bought XenSource as part of a push into the increasingly lucrative virtual machine market. Since they don’t care about Linux, they stopped moving their code forwards to the latest releases.

The last set of XenSource official patches for Xen target the 2.6.18 Linux kernel. The current release we use is 2.6.22, the mainline is at 2.6.23 and 2.6.24 is just around the corner.

And here’s the rub: kernel version 2.6.18 is too old to support our server’s hardware. And third parties have given up on doing the messy gruntwork required to keep porting Xen to each new kernel release. So we can have virtualisation, or hardware support, but not both. The combination which was lashed together before christmas is unstable and tends to drop offline at a whim.

Today I’ve been chatting to UCC colleagues about our options. There are quite a few, but ultimately the sensible ones boil down to two: rolling back from virtualisation or jumping sideways to a different platform. We’re still trying to work out which is the better option, but in the meantime — please bear with us.

So James and I have just finished the third round of work on the troppo server to stop it going down. The original goal was to make the server more stable in future scenarios, which is why the past few days have been so frustrating. In any case, I remembered an xkcd comic which summed up our experiences so far. 

Something went horribly, terribly wrong last night. The server is all kinds of weird and frankly I am amazed it is working at all. It will let me log in but fails to reboot when told to.

Over the fold is a diagrammatic explanation of my feelings on the matter.

Update:  “Epic” is only world left to describe all the things that went wrong today, both in the first place and during the attempt to fix things. Today it was a mix of “why the **** did they do THAT?” and “GOD I HATE WORDPRESS episode 579,294″.

(Continued)

Just letting you all know that James Andrewartha and I will be taking the server down on Sunday, probably in the afternoon (we are, after all, students). In an ideal world the shutdown will last around in hour. ¹1. Exercise: It is left as a philosophical exercise for readers to determine whether we do, in fact, live in such a world. [↩]

The goal of the shutdown is twofold.

Firstly we’d like to upgrade the operating system to a later variant (the current Ubuntu distribution, to be precise). The purpose of this is to help us close to the leading edge of software development. In practice it won’t make a huge amount of difference, but when you fall too far behind active development it becomes harder to upgrade. We’ve also being having some stability issues with the current kernel release rebooting the server every few days, which is annoying but not show-stopping as the server reboots in 60 seconds or so.

Secondly we’ll migrate the scheme to use a virtual server setup so that the underlying hardware can share excess capacity with other users. Thanks to the generosity of donors Club Troppo has a wonderfully beefy server which makes this a pretty good decision. Not only does it make migrating Club Troppo to new servers easier, it makes it trivial to share resources with the University Computer Club who pay for our electricity and network connection. In future it would also make it possible for Club Troppo to have some degree of “failover” capability, though that’s some way off.

Update: The site will go down around 7.30pm WA Time.

Update 2: All done now.

As part of the multi-layered antispam measures in use here at Club Troppo, I installed a few months ago a program called “Bad Behaviour”. This program works against spam by examining requests made to the server for common spammer patterns, rather than (as our other tools do) examining the content of comments.

Today it decided to throw a wobbly and various people got 403 Forbidden messages. Apparently some Australian ISP has been added to a blacklist consulted by Bad Behaviour.

In any case I have deactivated it for now.