Specter doesn't play those games.
He's an old man, he's an independent-minded Republican (a dying breed), he's a former prosecutor, and he's been a Senator for a long, long time. He has a lot of respect for the law. He takes his Constitutional role as a legislator very seriously. He has no further political ambitions. He has nothing to gain from playing a serious matter like this for partisan purposes.
When you consider that some of the Administration's own people opposed the domestic spying and were fired for their opposition, the chances that Arlen is wrong are not great.