I only want my games to feel rewarding and have at least a minimum of challenge, so if a game creates a disonance between narrative hardship and actual challenge, I'll make sure to raise the difficulty.
For example, I'd say Hero mode is the way to play the GC/Wii Zelda games because enemies do too little damage on regular mode, but it's not like it improves the experience that much as I'd prefer if the enemies became more intelligent and faster. It just raises the stakes as much as the game would allow, at least unless you try to do a 3 heart run.
Thinking about this in "gamer mentaility" I recently saw a streamer starting Birth by Sleep for the first time on Critical knowing fully well that they won't even try to meld anything or use any of the other new battle mechanics until the final boss forces them to, and I pity them. That game is all about experimentation with new attacks and unlocking stat buffs by switching and maxing out commands as you play, but without them it just becomes a frustrating game of dying without knowing what hit you.
So... a difficulty setting by itself is not enough. It's also up to the player to use all of the tools at their disposal within that difficulty. If someone says that x game is too easy on normal or hard, it's probably because they have mastered the knowledge of everything that's "broken" in the game and how to exploit it, and thus can't have fun at those settings anymore. This is not what first time players will probably experience, so it can end up really unfair situations if you were to listen to those seasoned players that like to brag about their mastery.