The definitive fictional treatment of that has been done; I imagine you know the film "Outland", starring Sean Connery. Plotwise, it's pretty much a remake of the old Gary Cooper/Grace Kelly western "High Noon", but the social conditions it depicts in the minig camp immediately struck me as very believable.

It was one of the first of a more realistic series of films that started to depict space colonization based on models from our own history, instead of relying on some idealized, scrubbed, antiseptic vision of the future. By definition, the frontier will always be a wild place, law will be infrequent and probably corrupt, and it will be a world of great injustice, brutal exploitation as well as high opportunity. It will attract a lot of baddies, and a lot of victims. The frontier will be the outer edge, and no matter how the instituions of a society are organized, out on the boundary they are intermittent and spotty. In conditions like that, feudalsim seems to arise naturally as the social organization that works.
The high frontier will not be a libertarian utopia, or the United Federation of Planets.