I'm all for a man making a personal decision to partake in any activity, even if absent any moral standing... and I believe that the consequences of such action should be limited to the people it directly effected, in this case his wife and kids... however Spitzer made his career from busting the same prostitution rings he was frequenting, which opens him up to huge blackmail possibilities, or worse.
This shows an extreme lack of professional judgement, not just personal judgement and therefore makes him a candidate for removal from office. It has nothing to do with puritan values.