Nick, this isn't the first time I've noticed your gift for asking just the right questions or making the right observations to inspire the answer. Your suggestions/speculations point in the right direction. All I have to add is the observation that the problem isn't one of the organization of the data (storage), but rather the structure of access to it.
What I mean is that the data's always there, but it's the mechanism for getting at it that's dynamic and therefore inaccessible to Google (and other spiders). The answer, therefore, is to add a hidden link on the site's root page that a spider will follow to a script (to be) called "SpiderTrail.asp". SpiderTrail will prepare a series (chain) of simple pages it feeds to the spider consisting of links to the root post in all the threads in each forum. The HTML version doesn't have to be available; all that SpiderTrail has to do is read the ID# in the filename of the thread XML files. The 404 error handler will create the HTML pages in response to the spider's request for the root ID, and the spider will do the rest, following the thread TOC at the bottom of the page to find all the posts in a thread. I'll trust the spider to be smart enough to not get caught in a loop of circular references through the TOC on each page....in fact, interestingly enough, all those internal links will add to the "weight" of each message, causing them to rise in the Google rankings.
The mechanism you're thinking of for influencing a spider is a file in the site's root called robots.txt. Unfortunately, it doesn't provide for positive directions; it provides only for exclusion. In this case I'll block the spider from taking the public path to messages, because that will result in needless duplication of the messages currently in public view.
I don't think that this will be all that difficult to program. I should have time to tackle it this weekend...though it occurs to me that I have no effective way to test it, and so we might have to wait a few weeks for a mass of HabitableZone references to suddenly show up in Google. Or not, in which case it's back to the drawing board.
Thanks, Professor!
Geek Speak » in reply to re: Some thoughts on search engines and Google in particular
hmmmm...Hmmmm....HMMMMM....yes indeed!
The whole thread (40 posts)
- naive but hopeful query
- Some thoughts on search engines and Google in particular
- Not quite what I had in mind. . .
- re: Some thoughts on search engines and Google in particular
- re: Some thoughts on search engines and Google in particular
- another option
- Try this.
- Some thoughts on search engines and Google in particular
