While the exact details of this vulnerability are still investigated
(see here
if you want to catch up on the topic),
I wanted to share some of the thoughts I had regarding to what this incident
means for the wider open source ecosystem.
TL;DR: To summarize, these are the main points I found remarkable in this
entire development:
- A backdoor was snuck relatively openly into an open source project
- It was done by a somewhat trusted maintainer
- The target was not even xz itself, but rather sshd through an obscure chain
of dependencies
- Luckily, it was discovered within a few weeks before the backdoored version
was widely adopted
Obviously, there are many examples of security vulnerabilities occurring in open
source software.
But these are usually due to oversights or mistakes of most likely well-meaning
developers that end up enabling the possibility for critical exploits.
In the case of the xz backdoor however, it was obviously constructed with
malicious intent and high effort towards a precise target.
Does anybody know of another vulnerability ending up in a high-profile open
source project that is similar in that sense?
This was only possible because the malicious actor under the pseudonym Jia Tan
had direct write access to the xz repository as a maintainer.
I don't think it is too unreasonable that with enough time and effort,
anyone can get maintenance access to openly developed projects like xz.
That is part of the beauty of the democratic process in open source.
But what this incident shows is that for projects that are as widely used as
xz,
even changes coming from seemingly trusted maintainers should be properly
reviewed.
I don't mean to say that the original maintainer Lasse Collin
has any fault in this matter, or that he should have prevented it,
this is too much of a burden to expect from a single person.
Instead I think the large tech corporations should put more resources into
vetting these kind of open source projects that much of their infrastructure so
heavily relies on (in fact, this backdoor seems to mainly target servers).
Even just looking at the source code, the backdoor was very cleverly hidden in
testing binaries for the compression algorithm.
These things are always easy to say in hindsight, but I do believe that a
closer review of the build system shenanigans used to install the backdoor
would have at least raised some questions.
There was just too much luck involved in the discovery of the backdoor with
someone noticing ssh access taking 0.5 seconds longer than usual.
This isn't really news, but this incident again shows that just like a chain is
only as strong as its weakest link, a program is only as strong as its weakest
dependency.
The fact that the backdoor just hooks into the dynamic library loading process
and completely hijacks authorization functions of ssh from inside xz is pretty
scary.
Maybe this will encourage developers to be more careful and sparing with adding
dependencies.
However to be honest, up until recently I would have pretty blindly trusted xz
to be a very safe dependency due to its popularity and relatively simple
use-case.
By opening a backdoor into ssh servers, this is a very critical issue,
and there was clearly a lot of time and effort put into making it seem innocuous
and hard to detect.
I'm very glad that it got found and patched by the time it did,
but it does leave me wondering what else is out there.
It would be illusionary to think that such attack vectors always get found out
eventually.
But you can just as well make an exception to allow errors when -e is enabled with something like
command || true, or even some warning message.I feel like, while it does occur, allowing errors like this is more unusual than stopping the script in an error, so it's good to explicitly mark this case, therefore -e is still a reasonable default in most cases.