Swim trunks make sense. The dude can't swim in the nude, that's just wrong. Cartoon logic dictates characters put on more clothes to swim, after all (see Bugs Bunny and his old style full body swim suit).
But to always wear pants? Well... I dunno. Doesn't that just mean he actually does have something to cover up?