I understand. Couple of years ago a lawyer hired me to create a database of California lawyers by scraping the Bar Association Web site's database engine. Turned out their terms of services took umbrage at that sort of thing, and they'd installed a "governor" to watch for lots of database requests in a short period, and cut off the offender. Cut me off, cut the project off at the knees. So yeah, I know whatcha mean.

With 10,000 pages to spider, SDF's going to have to balance the rate of requests against the calendar. If it were me, I might crank up the volume late at night on the theory that I'm less likely to inconvenience other users by tying up the site. I'd make the spider interruptible--recording where it left off--to let me break up the spidering into multiple sessions and pick up where I left off. It sounds like the old data's historical and static so that part should be easy, and he'd just need to check at the front of the list for a new page each day. Simple Access database on the backend's no sweat for VB--use it to record each page spidered and a simple SQL search tells it where it left off.