I strongly suspect that there's more to it than just search engine spidering.  While load times are now better than they were for the last week or so, I haven't yet experienced an occasion when I've attempted to open the site and had an experience that could remotely be described as "fine".  My most recent access is typical – it took 14.801 seconds for the front page to open, and even that appears to be with all the blogroll categories and other database-driven sidebar features disabled.

Blogs like Road to Surfdom and Larvatus Prodeo have databases just as large as Troppo's, get crawled by search engines at least as often, and operate on a Wordpress platform.  Yet they don't have our drastic speed and reliability problems.  However, I know LP DID have such problems up to a year or so ago.  It might be a good idea to email Mark Bahnisch (and perhaps Tim Dunlop), find out what techie gurus they used, and pick their brains.  For example, it could conceivably be that MySQL isn't up to the task with such large databases including lots of fields containing lots of entries each containing lots of characters (i.e. the posts and comments).

However, as mentioned in my previous email, you might be best advised to leave all this until you have successfully shifted to Perth.  We can limp along in the current form until then.  It would be best if you don't lose any more sleep or get any more stressed.