It's easy to fall for the idea that the Apollo program was born with Kennedy's 1961 speech, but that simply isn't so. According to this timeline, preliminary work on the technology that went into Apollo began many years before. For example, the preliminary design of the Saturn V's F-1 first stage engines began in 1958 and the thrust chamber was tested in 1959.

Even before Kennedy was elected (July 5, 1960), the House recommended that a manned lunar mission should be a high national priority. The Apollo project received its name a few weeks later. A lot of work was underway by the time Kennedy made his bold declaration. In the years to come, NASA's budget was as large as 5% of the entire federal budget. Adjusted for inflation, it would be over $30 billion a year in today's dollars. Keeping the percentage of the budget the same, NASA's budget would have to be about $120 billion this year instead of roughly $15 billion.

Back in the 1960s, NASA's unofficial motto and semiofficial operating policy was "Waste anything but time." They didn't care because they didn't have to care. Their "go fever" attitude was not only wasteful of money, it killed the Apollo 1 crew (and some other astronauts).

There's no willingness to give NASA anything like the budgets they had in the 1960s. Take a look at the news coverage and opinion pieces from both sides of the political spectrum about Bush's space proposal. Most of it has been very negative, in part because of the bogus cost projections (e.g. $1 trillion to Mars) being thrown around. NASA isn't going to get much more money, so they're going to have to do these things largely within the budget they're currently getting, so of course it's going to take longer than Apollo. Right now, I'll be surprised if the idea survives to the end of next year, but I'm feeling pretty pessimistic about things right now.