I understand the point you're trying to make, but a couple of quibbles, if I may...

100,000 years ago there were homo sapiens sapiens around who were, judging by their artifacts, already exhibiting undeniable signs of intelligence. If that arbitrary number were off by an order of magnitude, there were prehumans around a million years ago who would have shown the potential for intelligence, to the discerning eye.

Whether these hypothetical visitors would have had discerning eyes is up for debate, of course. At the risk of anthropomorphising, I would expect that one of the items on an explorer's checklist would be to look for signs of other intelligences. They'd have been drawn to Earth by its abundant life, and if they understand evolution at all the way we do, they'd have been alert for signs of the emergence of intelligence.

Maybe they did see the signs and then left. Science fiction writers have had a lot of fun imagining such visitors leaving behind "sentinels" and observers and monitors. That's a comfortable anthropomorphism; lack of curiosity would be far more alien (and is curiosity one of the fundamental aspects of intelligence, even broadly-defined? You'd think so, wouldn't you?)

I think that within Drake-Equationish parameters of your thought problem, there would have been somebody home within the last such statistical window of opportunity. Taking the low end of 100,000 years, that's about ten windows during human/hominid history.

Quite aside from Mars anomalists (and their even whackier brethren imagining artifacts in places as unlikely as Venus), there have been serious proposals to test the proposition of past visits by making a systematic search in our solar system for evidence. The logic seems fairly sound: While conditions on Earth would destroy most artifacts (that we can imagine using reasonable materials) very quickly in geological terms, not so vacuum environments like the asteroids or various moons; or in orbit in free space. It really doesn't sound any more speculative than listening for radio signals from deeper space.