I find it comical that he claims his critics are trying to take the nuance out of the situation and his speech by "silencing him" when he is removing the nuance himself.
It reminds me of this story about how you run a bar. Things are running like usual, and then a neo Nazi shows up. He's all respectful, he isn't preachy, he's keeping to himself, no reason to kick him out. You say, "ok, he's fine", and maybe he brings a friend or two. And they are absolutely ok as well. Just chill guys.
The thing is though, those friends bring friends, and maybe those friends bring friends too. Ok, these guys are kinda assholes, I gotta do something about this. Only, you can't, because your bar is now a Nazi Biker bar. You made exceptions for the "good guys" but you didn't think about how going down this path could affect your bar.
That's why you have to nip it in the bud immediately. This is what he's failing to see about his own "commentary". Sure, like he says, he's not directly causing violence in the trans community. He isn't calling for deaths and he isn't killing people. But he's making it A-OK for his viewers to know that trans people are lesser. They start watching comedians and podcasters in their recommended YouTube videos and TikToks that are friends with Chapelle and agree with what he's saying. Then you get deeper and you start watching misinformation filled videos about why trans people are actually bad instead of just criticizing them under the veil of comedy. After that, you get to those wanting to end the existence of trans people.
He has to think about his impact in a broader context than just what he's directly doing. People with strong convictions and beliefs with influence are often pipelines for belief systems. If you are speaking from a strong political standpoint, you have to consider that. Sure, many people don't navigate deep into these pipelines. Many people might actually be disgusted at the people further down the anti-trans rhetoric yet still listen to Chapelle as a bastion of truth. But indifference can quickly be construed as acceptance. And the more people feel comfortable shitting on trans people, the more likely action will happen against them.
Honestly, he is being intentionally ignorant to his own impact. How can you see hundreds of people, thousands of posts, dozens of articles, all aimed at your own conduct and just react "no, they must be wrong!" How many people does it take to make you just sit and think for one second that maybe these thousands and thousands of people aren't just tricked into trying to "cancel" you, but are actually trying to get you to understand that what you are doing can be harmful in the long run?
It's absurd.