It sounds like you're just reinventing either the JVM (runtime instruction translation), compilers (LLVM IR), or something in between (JIT interpreters).

The problem is that it's a hard problem to solve generally without expensive tradeoffs:

• interpreter like JVM - will always have performance overhead and can't easily target arch-specific optimizations like SIMD
• compiler - need a separate binary per arch, or have large binaries that can do multiple
• JIT - runtime cost to compiling optimizations

Each is fine and has a use case, but I really don't think we need a hardware agnostic layer, we just need languages that help alleviate issues with different architectures. For example, Rust's ownership model may help prevent bugs that out of order execution may expose. It could also allow programmers to specify more strict limits on types (e.g. non-zero numbers, for example), which could aid arch-specific optimizations).