A few dozen people would be enough to start a colony.
If they are screened for any genetic diseases, then a few dozen would be enough. The problem with small colony populations in the past was a lack of genetic understanding. If just a few of those people had serious genetic defects, then over generations those bad genes would accumulate in the general population. By weeding out any such genetic problems in the founding generation, their descendants wouldn't have to worry.