iframes are cruddy, but they are designed for this - the long era of iframe-based embedded advertisements caused vendors to double-down on those security policies.
Observable doesn't use WebWorkers, duktape, or a custom JavaScript interpreter. If you had sandboxed JavaScript that didn't need access to the DOM, those might be options, but if you want people to write JavaScript like usually, they don't do any good.
Considering the constraints I would experiment with a virtual machine. It can use your own set of instructions, like high level domain specific instructions in a json, or something lower level like a simple assembly.
You could use qemu in WebAssembly in a webworker and hope for no vm escape hack. Or a lua interpreter implemented in javascript.