It always depends on the implementation, and some of the frustrating aspects can also affect "proper" stealth games, but it's less likely to happen than with games where stealth is just a simplified optional mechanic. Implementing stealth means opening a whole new can of worms, there are so many things that can go wrong and either annoy the player or break immersion, and expose inner workings of the game.
For instance, you can expect to be less visible in dark places, only to find out that you're not. You can have enemies that are too smart, and once you're exposed, everyone will instantly home in on you, perhaps never losing sight of you again. You can have neverending respawn waves once the alarms have been turned on. You can have enemies that are too dumb, and won't notice dead bodies lying everywhere. Or maybe they will, but you'll be given no option to hide the bodies or dispose of them in any way. Or contextual takedown prompts will be poorly implemented, causing you to fail where you have succeeded before.
Perhaps enemies will be too easily distracted by throwing random objects, repeatedly. Or they will be in pursuit, but instantly lose sight of you when you enter tall grass and crouch. Or you will stealth kill one of them, while others standing mere feet away won't notice. Or you could have supposedly deaf enemies, but once you make a loud noise, they'll home in, not giving you the opportunity to quietly move out of the way. Or zones of detection will be made obvious by the fact that you can make any loud noise as soon as you step out of the zone, and enemies won't react, even though they should.
So, so many things that can go wrong, diminishing the overall experience, and for what? To mark that checkbox, and add more poorly handled complexity, as if more complex necessarily equals better.
I think that Quantum Break could've done a lot more with the puzzle aspects of time manipulation, rather than stealth.