Visual science fiction has tended to take the easy way out and use anthropomorphic aliens, mainly for reasons of cost. That was Roddenberry's explanation. You can see the effect of budget in the elaborateness of the prostheses. Low-budget operations tend to rely on a forehead prosthesis, some sort of bony brow ridge and maybe a funny nose to cue us that this is an alien. With a few extra bucks they add funny ears. Bigger budget productions have the elaborate body costumes, like Kosh on Babylon 5. Blockbusters like Alien go all the way and build "monsters" from the ground up. Notwithstanding even the latter, I'd agree that they generally end up being variations on a familiar terrestrial lifeform.

Novels try harder. Ursula K. LeGuin and CJ Cherryh come to mind right away as authors who concentrate on alien psychology and perceptions, and though they (the characters, not the authors) might be physically fairly routine humanoid forms, they (the authors) seem to leverage off that for contrast with truly alien minds.

As for sentient starships, that has been done, in both media. Anne McCaffrey wrote her first novel in the genre, "The Ship Who Sang", back in the '50s, about a sentient starship, female, who fell in love with her captain. The idea has cropped up regularly, more often as a backdrop than the centerpiece. Seem almost de rigeur to at least have a character refer to the "ship's AI".

And then in the visual branch, there's Andromeda. I was so charmed by their implementation, probably not least because the organic avatar ("the ship made flesh") of the AI was a charming young actress named Lexa Doig. She played Andromeda rather skillfully, I thought, as both the mature warrior starship of the AI, and the young and inexperienced avatar dealing for the first time with emotion. She too was inclined to fall in love with her captain, but we learned that that sort of thing was offlimits, adding a poignancy to it.

Matrioshka brains have shown up in science fiction novels by Charles Stross (title escapes me at the moment). The video's an hour long so I set it aside to watch later; I'll comment if moved to it.