"Punching down" is subjective, and because of the mass pussification of audiences these days, people will always be offended by something a comedian says, no matter how much they refrain from "punching down".

Why not just let comedians do their thing? If you don't like a comedian, don't support them.

Or better yet, encourage audiences to grow some emotional resilience instead of babyproofing comedy.