And oh yeah,
Unless I misunderstood something. Peter was not time traveling. He was skipping around alternate universes. So, he was unknown in the universe where Walter never brought him back and Olivia didn't know him.
It can get confusing as hell and it also takes a lot of the fun out of it when you consider that having established that alternate universes exist, in some universe, they failed to send the kid to the future, they all perished and the observers won.