I'm in the process of setting up this way too. I just switched over to lazy.nvim a few weeks ago, and I'd like to keep the lazy loading and configuration system.

I did a little experimenting to see if I could use lazy to lazily-load from Nix packages instead of downloading stuff itself. I couldn't work out. But with a version-controlled lock file lazy is declarative and reproducible by itself, so I suppose there's no need to mess with it.