Weird, I've never gotten a negative reaction to wearing above-the-knee cargo shorts in warm weather. Of course they're not swimsuit short or anything, but they're not like those knee length half-jeans either.
Mind you, I wear loose-fitting short sleeve button shirts, but that's mostly about comfort, with a little awareness of how unflattering tight shirts would look with my waistline thrown in.