Yeah, that's not good. I'm pretty sure there are two difficult enemies there that sort of signal you aren't supposed to go any further, but you may not have recognized them as such. In either case, I didn't have the problem you did. I beat the dungeon, then went out, saved my game, and went back in again. Since this game is a bit more open than your average JRPG, and since it does take some cues from MMO stuff, you can expect that there will be circumstances where you can accidentally wander into something you aren't quite cut out to handle yet. Generally speaking, you can just run the fuck outta' there, but I guess in that one you couldn't.
And Cobra, if it makes you feel any better, I've been playing the game for over 112 hours now and I *still* haven't gone back to fight that dude because I'm still not sure I'm ready (the guy himself is one problem, but he spawns a ton of high level mobs which are pretty horrible too). So I can understand why you feel so bad about getting creamed, because you probably got really hardcore creamed like I did when I went in there.
For the most part, I can say that I don't remember any other area of the game being like that exactly, but there have been a couple points where I was at least half as frustrated as you because I got so close to winning a fight and died, or because there just wasn't a save point quite close enough to something and I felt like I lost too much time. I stopped playing the game a bit ago because there was a particular hunt that didn't have a save nearby, and I tried it and lost twice even though I got close both times, but it was just too time consuming to be anything other than frustrating. Of course, there *was* a save nearby, and I sort of knew it, but there was a challenge that had to be completed before it opened up, and the reason I got myself in trouble was mostly because I was being stubborn trying to beat this hunt before I beat the other challenge. I get like that sometimes, especially with this game. I tilt at windmills a bit too much just because I sort of find it fun when I actually do manage to overcome something that's technically too powerful for me (I've managed a few... it's always satisfying).
Anyway, I don't think there's too much beyond that one spot where you can really get into huge trouble like that, though it's possible in a few circumstances to wander into things. I've done so much, it's hard to remember anything specific now. I just know that it wasn't really an issue for me, so that's the best I can tell you. Just keep in mind that the world is still explorable for the whole game, so there are a lot of times when you might find something that's too rough because it's intended to be a later challenge. This isn't like other JRPGs that suddenly cut you off from things or have tons of stuff you can "miss". There are sort of a couple of the latter, but not really too much. Mostly there are a couple loot items you can just sell that actually have a use later (if you only have 1 of something, check online to see if it has a purpose before you off it, and it might help you to read a loot FAQ if you intend to unlock all the bazaar items and stuff.. it can be a little tricky).
Anyway, if you're in doubt about an area, stick to a path you know or think is safer and just try to get it done and find a save... once you know where a save is and that you have ready access, then go back. Generally delving really deep into dungeons and stuff will bring harder challenges... pretty much always. So be careful going deep into the Garamscythe Waterway, the Zertinan Caverns, the Sochen Cave Palace (I don't think you can get into too much trouble there without specifically looking for it later, though), Barheim Passage, either of the mines... a lot of the time stuff is locked off from you until later, but not always. The Waterway and Zertinan Caverns are good examples. I got into trouble I wasn't ready for in both of those places, though I was LOOKING for trouble in the former and was able to run away from the latter. Oh, and if enemies are super tough, take that as a warning that something even nastier is probably lying in wait. If you ever see a pair of guardians like were guarding that gate thing at the Stilshrine, just come back later. The game seems to mostly find ways to warn you if there's bad stuff ahead.