Yes, bad HDR can make games worse. Madden '19 or '20 had really bad HDR at least on my TCL TV (and HDR heads, I know, some TVs have like legit HDR and others don't or something, just forgive me I'm not an expert), and it made the game look much, much worse... Like every outdoor game was being played overcast, or like the stadium lights weren't turned on. If you use replay mode they looked good, but while playing the game it looked terrible. I thought it was just the graphical style of the game, until I played it on my Xbox One S on my older TV that didn't support HDR and I was like "wait..... this looks way better wtf...." and then I turned off HDR on my 4K TV and played the game and it looked much better, like what I expected it to. I wasn't crazy in teh end and this ended up being a bug that they fixed.
Otherwise I have a really hard time noticing HDR. I remember when HDR came out and everybody was like "This is the future!" "this is more NExtGen than 4K!" And like... I felt like I was blind or something... Because like, ok sometimes it looks better to me, but honestly like....... I couldn't tell if it was *better* or just different. But I think I'm a broken person.
As a person who plays a lot of RDR2 I should probably turn it off. I think I have to turn it off system wide though, which is annoying to me, rather than game by game.