There are indeed ways you could make it work, but then you add more hypothesis and thus the cost of the simulation hypothesis increases.

Optimizations are indeed necessary, but just like the player is something special in a game, the observer would need to have a special status in the universe. I don't like this idea because the history of science always moved in the direction of making us the observers less and less special.

Moreover if life spreads in the universe, the simulation would encounter a scaling issue with an exponential growth of the numbers of observers.

I agree with you that in the end we just don't know, it's fun to push ideas to their limits!