In the long run, you are of course right. Still, if launch costs are low enough--a key point--it will make more economic sense in the short run to build stuff on Earth (where the infrastructure is already in place) and just ship it up. For many years, this will also be true of most components of lunar colonies, which will be shipped to the Moon. There are many examples of this "coals to Newcastle" approach on Earth. Steel buildings at iron mines in Minnesota are made from steel shipped from the East coast (or even overseas!), not smelted and forged on site, despite the access to raw materials.