The endangered species act isn't based on science.
There are a lot of examples of populations of animals starting from a few individuals. The iguanas (both land and sea) of the Galapagos are thought to have originated from a single pregnant female washed out to sea on a raft of debris from South America. Most of the other fauna are thought to have arrived in a similiar fashion. Even the pre-Columbian Human population of the Americas is thought to have originated with a few family sized groups of nomad hunters following their prey across the land bridge.