I spent most of yesterday working on the problem, which was an agonizingly slow process because the server was slow. The problem got in the way of its fix.

The server's running at its subjectively normal speed today. The fix is in progress; explaining that requires the warning

Danger! GeekSpeak ahead!

On the thread below I went on at mind-numbing length about the Microsoft Message Queue service, and that's where the problem lies. The first version of the software I installed after the server crashed had one teeny-tiny little bug: I left out the command to delete a queue when the software was done with it. Some small bug, eh? The brainfart was thinking that closing a queue also deleted it.

I fixed that in the second version of the software that I installed, but the bug left behind several hundred thousand orphaned queues, one for every table of contents and message post requested during that period. MSMQ maintains an index to look up queues, and as the database grew, performance dropped exponentially. Yesterday it turned into a sow, as Tom put it.

To get rid of those orphaned queues, I had to write some software to go through and delete them. I had quite cleverly given the queues names based on a seven-digit random number, so the program has to go through all 9,999,999 possible queue names and try to delete each one (in case you're wondering, the index is designed for fast lookup by name, and doesn't provide a query facility to just get a list of existing queues). Because the index is so big, in the early stages the process is agonizingly slow. Naturally, as it deletes queues, the process will get faster, probably exponentially faster. The program's running now, and it may well take several days before it completes. (It runs backward from 9,999,999 and as of this writing it just deleted queue # 9,950,067 [no, make that #9,949,915] after 18 hours of operation.) I set it to run at a low priority so that the site software can perform its normal operations in a timely manner.

So there you have it. Thinks are back to normal...until the next problem.