bethesda doesn't seem to be able to improve the core gameplay because the engine can't cope.

No, Bethesda can't improve because they keep catering for the lowest common denominator, engine has never had anything to do with it, it never has. They don't need a complex RPG system with a ton of flashy new things; New Vegas wasn't complex, it was fairly streamlined as far as RPGs go, what they need is better writers and better game designers that know how make interesting worlds, quests, characters and gameplay mechanics.

even if you fixed the writing and tossed out the awful procedural generation in favor of hand-crafted environments, at it heart it's still going to play like a stripped down borderlands 1

Because they've been dumbing down their games since forever, bring back more robust roleplay with more actions and consequences, fully fleshed out mechanics, get better writers. Just look at Fallout: London, despite the bugs everyone that has played it agrees it's the best "Bethesda game" since New Vegas, another game that wasn't actually made by Bethesda. I'll repeat: the problem was never the engine.