Sorry to be late to the party.

Daniel, I see you already found what Google can provide searching for "scramjet". That you didn't find what you're looking for illustrates the limitations of Google searches on the Zone. There are a bunch of technical reasons I won't bore you with...oh wait, this is GeekSpeak, you expect your eyes to glaze over...but more on that later.

I just now whipped up a simple Google search form that searches what Google has on the Zone. It's at http://www.habitablezone.com/Search.html. It was humbling to see that my name appears only 191 times, compared to Tom's 4,590. But who can match Tom's prolific output anyway? Who'd want to try?

One of my roomies just loaned me a book called "Google Hacks", and it has some interesting stuff in it. Maybe as I read deeper into it I'll find a way to improve searching using Google. If I come up with something, I'll post it here.

Steve, the fact there are technical limitations on Google searches would limit the usefulness of even the paid version. I appreciate the willingness to contribute, but I don't think that'd be a good use for the money.

OK, now the GeekSpeak...

The Earlier button isn't a link, as somebody correctly surmised. It triggers some script that knows which table of contents (TOC) page you're currently on, and subtracts 1 from that number (on earlier TOC pages there's also a "Later" button, and as you can guess, it adds 1 to the number). It then submits a request for that TOC page number to the server. The reason for this is that only the current front TOC page is stored as an HTML page. Earlier TOC pages are generated dynamically from a database. This means that there's nothing for Google to search earlier than the front page. The reason that it still manages to index so many HTML pages is that it searches that first, static, TOC page and follows its links to messages. Google searches (spiders, technically) every few weeks, so there's a moving window of opportunity. Older messages drop off, and new messages are added.

Another little technical roadblock is that the HabitableZone software stores messages in a format called XML, rather than the HTML that you view and that Google searches. There's one XML file for each thread, containing all the messages on that thread. When you post a message to an ongoing thread, the message is added to the XML file, and then the software regenerates the individual HTML pages for each message in the thread, automatically updating the thread TOC at the bottom of each page to reflect the new message.

It's true that this does result in an HTML page for each message, but the HTML pages aren't guaranteed to always be there. A feature I planned to add but didn't get around to is a "garbage collection" system that will delete the HTML versions when they get older than some fixed amount, say 60 days. This is necessary not so much to save space on the server's hard disk, which has plenty of room, but because the operating system begins to slow down if there are too many files in a folder. We've reached that point, I think, so I know I'd better add that feature soon.

Delete the files??!!! I hear you (all) shrieking. Hey, don't worry, that doesn't mean deleting the XML which contains the official message data. It only means deleting the derivative HTML. The system is already set up to automatically generate an HTML page if you (or a link) request a message ID number for which there's no HTML (technically, this is a 404 error handler; I use something similar in the image caching system on the Workbench site to go get a copy of an image if it isn't already stored on my server). You might notice a fractional delay when you request a message whose HTML page has been deleted, but then it shows up.

All that stuff affects the searchability of the site by an external search engine like Google. The kind of search engine we really need runs on the server, and knows how to read XML files directly. Justin mentioned in an email a while back that he was thinking of writing a searching engine in the Perl scripting language. That might work. Another option I can explore is to write an interface module for Microsoft's Index Server (built in to Windows 2000) that knows how to read the XML format and put it into a form that Index Server can deal with. The downside is, of course, that I or Justin have to spend some time doing it. We're both pretty busy with other things, and I guess the search feature keeps slipping down the priority list when other things keep coming up. I'd like to have a search engine, gawd knows nobody lets me forget that we need one, but I can't make any promises about when it'll happen.

In the meantime, I'll keep reading "Google Hacks" to see if I can come up with a way to make Google do all the work.