For example, if I’m writing a compiler for a new toy language

Ok, thinking about it (since I wrote a toy language not so long ago), this is probably a perfect example where unit tests make sense almost everywhere (even for prototyping, say parser).

I think it definitely depends what you're doing, writing unit tests for prototype graphics (engine) code is no fun (and I see no real benefit).

Code readability counts, but I don’t think it’s enough.

I think it depends, For general architecture, E2E or integration tests definitely make sense, for finer-grained code, I think documentation (Rust doc) of the functions in question should be enough to understand what they do (including some examples how to use them, could be tests, often examples (similar as in std rust) in the rust doc are enough IMHO, and otherwise the code itself is the documentation (being able to read code fast is a valuable skill I guess). That obviously doesn't apply for everything (think about highly theoretical computer science or math code), but yeah it depends...