I felt the same way about webp when it came out.

In practice it doesn't really matter:

• if you're encoding the file you know how you're doing it.
• if you're receiving the file, you get the pixels you get no matter how it was encoded.
• if you're sending the image through some third party service, they're going to reencode and mangle it anyway so there's no point in worrying.

Also, it turned out that even if it's quite good, lossless webp is rarely seen in the wild because svg is more convenient.