Finally some good news from Segpub regarding the timeouts and slow response times (3 minutes, 38 seconds for a plain HTML file?!) that have been sporadically plaguing my poor sites for months. From a mailing list email addressed to folks on my server:
The issues with [your server] unfortunately are related to search bots and a mix of other nasty traffic. We're working really hard to get ontop [sic] of these and are pretty close to having everything resolved and back to normal.
Nasty traffic? That's not a nice thing to call my Nonsense section! ;).
The problem is, because my sites slowed down to a crawl but still eventually loaded, they don't count these problems as downtime which means they won't be providing me any free months as stipulated in their uptime guarantee. Sneaky.
I've seen some runaway php-cgi processes on my VPS. Hooray for cacti with a custom load time script to graph and pick it up (and then I have to go kill -9 them by hand :<).
Then again, fail2ban is using 153meg (almost as much as mysqld, lots more than postgresql) with all the bots trying ssh brute force. Internet is a tough neighbourhood.