Then say that. No need to throw the whole medium under the bus unless you want to Sturgeon's Law everything.
I mean there's been a pretty big resurgence of stuff like Jojo and similar older shounen trope-filled stuff like Megalo Box and Devilman, so I'm not sure why a casual fan wouldn't be thinking of that when they think of anime. If you're on a forum all day and see people deconstructing the seasonal charts, sure I can see the association to the more questionable elements of the medium, but that just goes back to Sturgeon's law.
I don't even like Berserk that much so I'm not gonna go out of my way to defend it, but it's a lot more than the Eclipse. I'm just talking about the general feel of the tone and Miyazaki pretty much straight up lifting panels from the manga for certain areas. The guy is a Berserk fanatic and it shows.
Ok, I'll stick with something like 'waifu anime' or 'anime titty game' to clarify better because yah, I'm totally aware that anime is a medium and not a genre. And that the genre has stuff like Utena, Totoro, as well as SAO. Idk, anime IS getting better with the resurgance of a lot of female focused stuff (Free! and Yuri on Ice, CCS reboot), as well as the older shounen stuff (AoT feels like older shounen, then there's Devilman and Jojo's like you said). Maybe some people use 'anime' as a derogative like that because it was mostly stuff like Haruhi or dating sim / light novel adaptations for a long time in the late 2000's to early 2010's. It's definitely changing for the better. I kinda agree with you that a bit of it is undeserved though, especially the hate some people have for 'anime avatars'.
I've heard from some Berserk fans that it does get better and more meaningful, I just made it to the Eclipse scene in the original anime and noped out because stuff like rape and sexualized violence are kinda triggering for me. But that's why I adore Dark Souls. Has all the parts I loved about Berserk (tone and design) with none of the parts I don't (demon rape).