Time and Fermi's Paradox
Ian Crawford estimated that it would take only 3.75 million years for a technologically advanced civilization capable of interstellar spaceflight to completely colonize the entire galaxy. That’s certainly in the same range as your rough estimate of 1-2 million years, and more importantly, a very, very short period compared to the history of life on Earth.
Where are they? Sci Am July 2000
(link is above, and hopefully visible!)