29
Nix 3rd party repos and Binary packages
(feddit.nl)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 09 Aug 2023
29 points (100.0% liked)
Linux
47953 readers
1182 users here now
From Wikipedia, the free encyclopedia
Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).
Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.
Rules
- Posts must be relevant to operating systems running the Linux kernel. GNU/Linux or otherwise.
- No misinformation
- No NSFW content
- No hate speech, bigotry, etc
Related Communities
Community icon by Alpár-Etele Méder, licensed under CC BY 3.0
founded 5 years ago
MODERATORS
Sorry, I assumed you were an advanced user/the code was easier to understand. It's easy to get caught up in the imposter syndrome.
You mean
packageNamesToModify? You have to provide those. I don't know which packages could be built withmy-proprietary-codecand it's not feasible to figure that out automatically.Yes, certainly.
overrideAttrsallows you to "modify" the argument originally passed tomkDerivationsuch assrc,pname,version,buildInputsand many more. To understandoverrideAttrs, you must know what these commonmkDerivationarguments mean. From then on,overrideAttrsis trivial to understand.It's like writing a derivation but "updating" an existing derivation to be slightly different.
Oh absolutely. There's quite the learning curve, almost everyone knows that. It's just not an easy thing to fix and you don't tend to think about learning things you already know, so it's not constantly in people's minds.
I'ma let you in on a little secret: We're not actually a binary distribution. We're source-based.
It's integral to the functional approach. Every artefact is a more-or-less direct result of a realisation which is the direct result of an evaluation of Nix expressions. The latter is pure and the former we strive to make as pure as possible via sandboxing.
This allows us to treat artefacts as replaceable objects which means that instead of building locally, we can substitute the (theoretically) same artefact from somewhere else and that's the binary cache.
If you're at all interested, go ahead and do so. That'd make for a great first PR. JXL support is something you'd expect of ffmpeg in the near future (if not already), so this will happen anyways.
Not really. Looking at
genAttrs, it says that it'sgenAttrs :: [ String ] -> (String -> Any) -> AttrSet, and the example code isSo I'd expect a list of strings and the mapping function. However, from my first naive glance at the code you posted, there's no list of strings there, and rather directly a function definition that makes use of overrideAttrs, for which for me the documentation is unclear if it can only set attributes, create attributes or what else; also
buildInputsat first naive glance isn't an attribute of the packages, but rather ofstdenv(sorry if all of this is wrong), all of which isn't exactly intuitive if you've only worked with imperative languages.I do know that, that's why I worded it like I did. In fact I have created a single package myself before and as such, did read a bit about FOB and all that, how the cache works (not that I can fully recall everything or claim that I understood it all) with input-hashes etc. I tripped over some stuff back then when I found out that
whichis not part of the sandbox. The program however is a bit niche and there's another component it needs that I didn't write a derivation for so I also never wrote a pull request, plus the build process is somewhat ugly. So I wouldn't call myself a beginner exactly but I definitely have trouble with advanced usage of the language and the intricacies of the system.However, while source-based, I think it's a common understanding that the binary cache is a huge appeal, otherwise the discussion about the cost it wouldn't be had.
That's my bad. I didn't actually look at how the function works, so I got the arguments flipped. This is just a rough outline of how I'd imagine it to work; pseudo-code. 100% untested. Only there for the sake of argument.
packageNamesToModifyis the list of strings you're looking for here. As I said though, I can't know which packages are supposed to be changed. That's a piece of info you'd need to research and define yourself. This is just how you'd apply that knowledge through a mechanism.A function cannot "set" or "create" attributes. For better of for worse, there's no real meta-programming in Nix.
A function can only ever return a value. That value may be any of the primitive types such as booleans, strings, numbers, lists, attrsets etc. or even another function but it's always a value.
In the case of
overrideAttrs { ... }, the return value is a derivation. In the case ofgenAttrs, it's an attrset.That is correct. However note how I said that
overrideAttrsis about overriding arguments tomkDerivation? The canonical path tomkDerivationisstdenv.mkDerivation;)mkDerivationis the way you set arguments for the stdenv..overrideAttrsis how you "modify" the arguments to the stdenv of an existing package.I can absolutely see that. You'll get used to it though. When I was new to Nix, it took me quite a while to realise how you'd even do something like creating loops (spoiler: you don't).
I'd highly recommend familiarising yourself with basic functional programming concepts such as immutability, everything being a value (including functions, see lambdas), recursion, basic functional list comprehension (head/tail, map, filter, reduce) and perhaps even currying.
Nix is a great learning ground for the basics of functional programming as it's a pure expression language which is quite a bit more limited than an actual functional programming language.
That's what
buildInputsare for. AddwhichtobuildInputsand it's available inside the sandbox. The stdenv takes care of putting its binaries into$PATHand making its libraries discoverable.In the case of
which, you'd probably need it in order to execute its binary during the build process though, sonativeBuildInputsis more appropriate but that only truly matters for cross-compilation.Thanks for the explanation. I didn't make the connection that
buildInputsis an attribute itself as it is an attribute ofstdenvinstead of the function describing the derivation directly. Or at least I think that's where my confusion comes from.Small correction, it wasn't
which, but ratherenv, I had those mixed up. The "issue" is described here: https://github.com/NixOS/nixpkgs/issues/6227 What created more problems was thatpatchShebangswouldn't work here because it appeared in a configure script that was created and run during the actual build process (I think the build process is horrible, but here it is in case you're inclined: https://github.com/BSI-Bund/TaSK/tree/master/tlstesttool the stuff in the 3rdparty directory gets downloaded, configured and linked against in the main program's build phase so you have no opportunity to actually follow the solution in that issue, similar to what is described here https://github.com/NixOS/nixpkgs/issues/6227#issuecomment-73410536. I got it to work in the end and like to tell myself that it's elegant but the project's build process is just bad in my opinion.https://pastebin.com/0GwLk1wP if you want to see an example of my level of nix-fu. I have programming basics but the nix language can be confusing sometimes. I'd say I have a basic understanding of things but as said before the more intricate stuff still escapes me.
The problem there isn't that
envisn't available (it is, we have coreutils in stdenv) but rather that/usr/bin/env(that specific path) does not exist in the sandbox.Yikes, you don't want that.
I didn't read the issue in full but, since everything is pre-downloaded, you can always fix these scripts.
For the auto-generated configure script for example, you'd have to patch the build step which generates the configure script to put in
${coreutils}/bin/envrather than/usr/bin/env. Simple as that.3rd party repositories can be patched during their respective fetches. You have to inject them anyways since there's no way to download 3rd party repos during a build.
You have to differentiate between the Nix expression language and using Nixpkgs' frameworks to define packages here. These are two entirely different skillsets with only a slight dependence on the former by the latter.
If you take a look at your expression, it only required fairly basic Nix syntax: Simple function calls, declaring recursive attrsets, string interpolation and attribute access. Most packages are like this.
Figuring out which attributes need to be set how and what that means is an entirely different skillset. Defining
patchPhasein that attrset is trivial syntax-wise but figuring out what needs to be substituted using which stdenv "library" function is something else entirely.Looks pretty good btw. I'd look into whether the build could be coerced to link against Nixpkgs' versions of those libraries though instead of vendoring dependencies. Especially security-critical stuff like openssl. That'd probably also save you the trouble with the interpreter.
If you take a look at the build definition, there's this handy dandy
USE_3RDPARTYBUILDoption.I'd wager if you did
cmakeFlags = [ ... "-DUSE_3RDPARTYBUILD=OFF" ];andbuildInputs = [ asio zlib openssl ... ];(frompkgs, not your fetched sources) it'd just work. (Might needpkg-configinnativeBuildInputsbut I don't think it uses that and will instead discover those deps via cmake.)If you can get rid of the vendoring, feel free to submit that to Nixpkgs ;)