It would've been tough to hit much lower on the towers
From what I understand, the surrounding buildings are about 50 stories high. If true, it wouldn't been difficult to clear them and then dive the airliner to hit much lower on the towers. That's one of the reasons why most plans to replace the towers that I've read about involve shorter buildings - perhaps 4 55 story buildings instead of two 110 story towers.
The point about hitting later in the day is a good one. A lot of people hadn't arrived at work when the first plane struck.