It's not like they stealth levels are badly done. They are well-designed. The earlier-on stealth levels weren't hard, but they weren't easy, either. They felt just about right, basically.
It isn't a matter of difficulty, its more of an issue where it's just bad design and isn't fun.
Games that have done stealth well:
-Metal Gear Solid Series
-Splinter Cell Series
-Tenchu
-Apparently the Thief series, although I can't say.
-Arguably Assassin's Creed
Games that have fucked it up:
-Every other game out there
-Especially Beyond Good and Evil
The problem is that unless the entire game and the associated mechanics is built up around it, there's no such thing as "stealth action" - there's either stealth or action. What it amounts to is a situation where in the middle of the level you're asked to stand behind a box or crate for 2 min while a guard walks by and then wait there another 3 while you wait for that to line up with the other guard looking away. Sure, in some games like BG&E you can kind of bypass it by shooting the guards in the 'special area' but that doesn't always work and it doesn't make it any more fun or enjoyable. I mean, if they were out of ideas they might as well have gone all the way with it and just cut off the power to my controller for 30 seconds randomly or just made a big room you have to wait in for 5 min. before the door opens.
It's a broken mechanic in a lot of games, and I'd really just rather a puzzle or platforming area.