Every man needs a hobby
I remember Eisenhower was often criticized for playing golf during international crises, and I think Bush's reluctance to do so may stem from that. It's strictly a propaganda move.
Actually, being the US President is a tough job, and any recreation that helps him relax in a crisis is probably a benefit to both him and the nation, once these image issues are behind us. I'm sure Kennedy went sailing when he needed to relax and nobody cared. Nixon updated his enemies' list and Clinton conducted staff interviews...