Nix pill 15: nix search paths

Welcome to the 15th Nix pill. In the previous 14th pill we have introduced the "override" pattern, useful for writing variants of derivations by passing different inputs.

Assuming you followed the previous posts, I hope you are now ready to understand nixpkgs. But we have to find nixpkgs in our system first! So this is the step: introducing some options and environment variables used by nix tools.

The NIX_PATH

The NIX_PATH environment variable is very important. It's very similar to the PATH environment variable. The syntax is similar, several paths are separated by a colon ":". Nix will then search for something in those paths from left to right.

Who uses NIX_PATH? The nix expressions! Yes, NIX_PATH is not of much use by the nix tools themselves, rather it's used when writing nix expressions.

In the shell for example, when you execute the command "ping", it's being searched in the PATH directories. The first one found is the one being used.

In nix it's exactly the same, however the syntax is different. Instead of just typing "ping" you have to type <ping>. Yes, I know... you are already thinking of <nixpkgs>.
However don't stop reading here, let's keep going.

What's NIX_PATH good for? Nix expressions may refer to an "abstract" path such as <nixpkgs>, and it's possible to override it from the command line.

For ease we will use nix-instantiate --eval to do our tests. I remind you, nix-instantiate is used to evaluate nix expressions and generate the .drv files. Here we are not interested in building derivations, so evaluation is enough. It can be used for one-shot expressions.

Fake it a little

It's useless from a nix view point, but I think it's useful for your own understanding. Let's use PATH itself as NIX_PATH, and try to locate ping (or another binary if you don't have it).

$ nix-instantiate --eval -E '<ping>'
error: file `ping' was not found in the Nix search path (add it using $NIX_PATH or -I)
$ NIX_PATH=$PATH nix-instantiate --eval -E '<ping>'
/bin/ping
$ nix-instantiate -I /bin --eval -E '<ping>'
/bin/ping

Great. At first attempt nix obviously said  could not be found anywhere in the search path. Note that the -I option accepts a single directory. Paths added with -I take precedence over NIX_PATH.

The NIX_PATH also accepts a different yet very handy syntax: "somename=somepath". That is, instead of searching inside a directory for a name, we specify exactly the value of that name.

$ NIX_PATH="ping=/bin/ping" nix-instantiate --eval -E '<ping>'
/bin/ping
$ NIX_PATH="ping=/bin/foo" nix-instantiate --eval -E '<ping>'
error: file `ping' was not found in the Nix search path (add it using $NIX_PATH or -I)

Note in the second case how Nix checks whether the path exists or not.

The path to repository

You are out of curiosity, right?

$ nix-instantiate --eval -E '<nixpkgs>'
/home/nix/.nix-defexpr/channels/nixpkgs
$ echo $NIX_PATH
nixpkgs=/home/nix/.nix-defexpr/channels/nixpkgs

You may have a different path, depending on how you added channels etc.. Anyway that's the whole point. The <nixpkgs> stranger that we used in our nix expressions, is referring to a path in the filesystem specified by NIX_PATH.
You can list that directory and realize it's simply a checkout of the nixpkgs repository at a specific commit (hint: .version-suffix).
The NIX_PATH variable is exported by nix.sh, and that's the reason why I always asked you to source nix.sh at the beginning of my posts.

You may wonder: then I can also specify a different nixpkgs path to, e.g., a git checkout of nixpkgs? Yes, you can and I encourage doing that. We'll talk about this in the next pill.

Let's define a path for our repository, then! Let's say all the default.nix, graphviz.nix etc. are under /home/nix/mypkgs:

$ export NIX_PATH=mypkgs=/home/nix/mypkgs:$NIX_PATH
$ nix-instantiate --eval '<mypkgs>'
{ graphviz = <code>; graphvizCore = <code>; hello = <code>; mkDerivation = <code>; }

As expected we got the set of our packages (well except the mkDerivation utility), that's our repository.

Until now we used nix-build directly in the directory of default.nix. However nix-build generally needs a .nix to be specified to the command line:

$ nix-build /home/nix/mypkgs -A graphviz
/nix/store/dhps2c1fnsfxssdri3yxbdiqmnhzg6yr-graphviz
$ nix-build '<mypkgs>' -A graphviz
/nix/store/dhps2c1fnsfxssdri3yxbdiqmnhzg6yr-graphviz

Yes, nix-build also accepts paths with angular brackets. We first evaluate the whole repository (default.nix) and then peek the graphviz attribute.

A big word about nix-env

The nix-env command is a little different than nix-instantiate and nix-build. Whereas nix-instantiate and nix-build require a starting nix expression, nix-env does not.

You may be crippled by this concept at the beginning, you may think nix-env uses NIX_PATH to find the nixpkgs repository. But that's not it.

The nix-env command uses ~/.nix-defexpr, which is also part of NIX_PATH by default, but that's only a coincidence. If you empty NIX_PATH, nix-env will still be able to find derivations because of ~/.nix-defexpr.

So if you run nix-env -i graphviz inside your repository, it will install the nixpkgs one. Same if you set NIX_PATH to point to your repository.

In order to specify an alternative to ~/.nix-defexpr it's possible to use the -f option:

$ nix-env -f '<mypkgs>' -i graphviz
warning: there are multiple derivations named `graphviz'; using the first one
replacing old `graphviz'
installing `graphviz'

Oh why did it say there's another derivation named graphviz? Because both graphviz and graphvizCore attributes in our repository have the name "graphviz" for the derivation:

$ nix-env -f '<mypkgs>' -qaP
graphviz      graphviz
graphvizCore  graphviz
hello         hello

By default nix-env parses all derivations and use the derivation names to interpret the command line. So in this case "graphviz" matched two derivations. Alternatively, like for nix-build, one can use -A to specify an attribute name instead of a derivation name:

$ nix-env -f '<mypkgs>' -i -A graphviz
replacing old `graphviz'
installing `graphviz'

This form, other than being more precise, it's also faster because nix-env does not have to parse all the derivations.

For completeness: you may install graphvizCore with -A, since without the -A switch it's ambiguous.

In summary, it may happen when playing with nix that nix-env peeks a different derivation than nix-build. In such case you probably specified NIX_PATH, but nix-env is instead looking into ~/.nix-defexpr.

Why is nix-env having this different behavior? I don't know specifically by myself either, but the answers could be:

• nix-env tries to be generic, thus it does not look for "nixpkgs" in NIX_PATH, rather it looks in ~/.nix-defexpr.
• nix-env is able to merge multiple trees in ~/.nix-defexpr by looking at all the possible derivations

It may also happen to you that you cannot match a derivation name when installing, because of the derivation name vs -A switch described above. Maybe nix-env wanted to be more friendly in this case for default user setups.

It may or may not make sense for you, or it's like that for historical reasons, but that's how it works currently, unless somebody comes up with a better idea.

Conclusion

The NIX_PATH variable is the search path used by nix when using the angular brackets syntax. It's possible to refer to "abstract" paths inside nix expressions and define the "concrete" path by means of NIX_PATH, or the usual -I flag in nix tools.

We've also explained some of the uncommon nix-env behaviors for newcomers. The nix-env tool does not use NIX_PATH to search for packages, but rather for ~/.nix-defexpr. Beware of that!

In general do not abuse NIX_PATH, when possible use relative paths when writing your own nix expressions. Of course, in the case of <nixpkgs> in our repository, that's a perfectly fine usage of NIX_PATH. Instead, inside our repository itself, refer to expressions with relative paths like ./hello.nix.

Next pill

...we will finally dive into nixpkgs. Most of the techniques we have developed in this series are already in nixpkgs, like mkDerivation, callPackage, override, etc., but of course better. With time, those base utilities get enhanced by the community with more features in order to handle more and more use cases and in a more general way.

Pill 16 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Nix pill 14: the override design pattern

Welcome to the 14th Nix pill. In the previous 13th pill we have introduced the callPackage pattern, used to simplify the composition of software in a repository.

The next design pattern is less necessary but useful in many cases and it's a good exercise to learn more about Nix.

About composability

Functional languages are known for being able to compose functions. In particular, you gain a lot from functions that are able to manipulate the original value into a new value having the same structure. So that in the end we're able to call multiple functions to have the desired modifications.

In Nix we mostly talk about functions that accept inputs in order to return derivations. In our world we want nice utility functions that are able to manipulate those structures. These utilities add some useful properties to the original value, and we must be able to apply more utilities on top of it.

For example let's say we have an initial derivation drv and we want it to be a drv with debugging information and also to apply some custom patches:

debugVersion (applyPatches [ ./patch1.patch ./patch2.patch ] drv)

The final result will be still the original derivation plus some changes. That's both interesting and very different from other packaging approaches, which is a consequence of using a functional language to describe packages.

Designing such utilities is not trivial in a functional language that is not statically typed, because understanding what can or cannot be composed is difficult. But we try to do the best.

The override pattern

In the pill 12 we introduced the inputs design pattern. We do not return a derivation picking dependencies directly from the repository, rather we declare the inputs and let the callers pass the necessary arguments.

In our repository we have a set of attributes that import the expressions of the packages and pass these arguments, getting back a derivation. Let's take for example the graphviz attribute:

graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpeg bzip2; };

If we wanted to produce a derivation of graphviz with a customized gd version, we would have to repeat most of the above plus specifying an alternative gd:

mygraphviz = import ./graphviz.nix {
  inherit mkDerivation fontconfig libjpeg bzip2;
  gd = customgd;
};

That's hard to maintain. Using callPackage it would be easier:

mygraphviz = callPackage ./graphviz.nix { gd = customgd; };

But we may still be diverging from the original graphviz in the repository.

We would like to avoid specifying the nix expression again, instead reuse the original graphviz attribute in the repository and add our overrides like this:

mygraphviz = graphviz.override { gd = customgd; };

The difference is obvious, as well as the advantages of this approach.

Note: that .override is not a "method" in the OO sense as you may think. Nix is a functional language. That .override is simply an attribute of a set.

The override implementation

I remind you, the graphviz attribute in the repository is the derivation returned by the function imported from graphviz.nix. We would like to add a further attribute named "override" to the returned set.

Let's start simple by first creating a function "makeOverridable" that takes a function and a set of original arguments to be passed to the function.

Contract: the wrapped function must return a set.

Let's write a lib.nix:

{
  makeOverridable = f: origArgs:
    let
      origRes = f origArgs;
    in
      origRes // { override = newArgs: f (origArgs // newArgs); };
}

So makeOverridable takes a function and a set of original arguments. It returns the original returned set, plus a new override attribute.
This override attribute is a function taking a set of new arguments, and returns the result of the original function called with the original arguments unified with the new arguments. What a mess.

Let's try it with nix-repl:

$ nix-repl
nix-repl> :l lib.nix
Added 1 variables.
nix-repl> f = { a, b }: { result = a+b; }
nix-repl> f { a = 3; b = 5; }
{ result = 8; }
nix-repl> res = makeOverridable f { a = 3; b = 5; }
nix-repl> res
{ override = «lambda»; result = 8; }
nix-repl> res.override { a = 10; }
{ result = 15; }

Note that the function f does not return the plain sum but a set, because of the contract. You didn't forget already, did you? :-)
The variable res is the result of the function call without any override. It's easy to see in the definition of makeOverridable. In addition you can see the new override attribute being a function.

Calling that .override with a set will invoke the original function with the overrides, as expected.
But: we can't override again! Because the returned set with result 15 does not have an override attribute!
That's bad, it breaks further compositions.

The solution is simple, the .override function should make the result overridable again:

rec {
  makeOverridable = f: origArgs:
    let
      origRes = f origArgs;
    in
      origRes // { override = newArgs: makeOverridable f (origArgs // newArgs); };
}

Please note the rec keyword. It's necessary so that we can refer to makeOverridable from makeOverridable itself.

Now let's try overriding twice:

nix-repl> :l lib.nix
Added 1 variables.
nix-repl> f = { a, b }: { result = a+b; }
nix-repl> res = makeOverridable f { a = 3; b = 5; }
nix-repl> res2 = res.override { a = 10; }
nix-repl> res2
{ override = «lambda»; result = 15; }
nix-repl> res2.override { b = 20; }
{ override = «lambda»; result = 30; }

Success! The result is 30, as expected because a is overridden to 10 in the first override, and b to 20.

Now it would be nice if callPackage made our derivations overridable. That was the goal of this pill after all. This is an exercise for the reader.

Conclusion

The "override" pattern simplifies the way we customize packages starting from an existing set of packages. This opens a world of possibilities about using a central repository like nixpkgs, and defining overrides on our local machine without even modifying the original package.

Dream of a custom isolated nix-shell environment for testing graphviz with a custom gd:

debugVersion (graphviz.override { gd = customgd; })

Once a new version of the overridden package comes out in the repository, the customized package will make use of it automatically.

The key in Nix is to find powerful yet simple abstractions in order to let the user customize his environment with highest consistency and lowest maintenance time, by using predefined composable components.

Next pill

...we will talk about Nix search paths. By search path I mean a place in the file system where Nix looks for expressions. You may have wondered, where does that holy <nixpkgs> come from?

Pill 15 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Nix pill 13: the callPackage design pattern

Welcome to the 13th Nix pill. In the previous 12th pill we have introduced the first basic design pattern for organizing a repository of software. In addition we packaged graphviz to have at least another package for our little repository.

The next design pattern worth noting is what I'd like to call the callPackage pattern. This technique is extensively used in nixpkgs, it's the current standard for importing packages in a repository.

The callPackage convenience

In the previous pill, we underlined the fact that the inputs pattern is great to decouple packages from the repository, in that we can pass manually the inputs to the derivation. The derivation declares its inputs, and the caller passes the arguments.
However as with usual programming languages, we declare parameter names, and then we have to pass arguments. We do the job twice.
With package management, we often see common patterns. In the case of nixpkgs it's the following.

Some package derivation:

{ input1, input2, ... }:
...

Repository derivation:

rec {
  lib1 = import package1.nix { inherit input1 input2 ...; };
  program2 = import package1.nix { inherit inputX inputY lib1 ...; };
}

Where inputs may even be packages in the repository itself (note the rec keyword). The pattern here is clear, often inputs have the same name of the attributes in the repository itself. Our desire is to pass those inputs from the repository automatically, and in case be able to specify a particular argument (that is, override the automatically passed default argument).

To achieve this, we will define a callPackage function with the following synopsis:

{
  lib1 = callPackage package1.nix { };
  program2 = callPackage package2.nix { someoverride = overriddenDerivation; };
}

What should it do?

• Import the given expression, which in turn returns a function.
• Determine the name of its arguments.
• Pass default arguments from the repository set, and let us override those arguments.

Implementing callPackage

First of all, we need a way to introspect (reflection or whatever) at runtime the argument names of a function. That's because we want to automatically pass such arguments.

Then callPackage requires access to the whole packages set, because it needs to find the packages to pass automatically.

We start off simple with nix-repl:

nix-repl> add = { a ? 3, b }: a+b
nix-repl> builtins.functionArgs add
{ a = true; b = false; }

Nix provides a builtin function to introspect the names of the arguments of a function. In addition, for each argument, it tells whether the argument has a default value or not. We don't really care about default values in our case. We are only interested in the argument names.

Now we need a set with all the values, let's call it values.  And a way to intersect the attributes of values with the function arguments:

nix-repl> values = { a = 3; b = 5; c = 10; }
nix-repl> builtins.intersectAttrs values (builtins.functionArgs add)
{ a = true; b = false; }
nix-repl> builtins.intersectAttrs (builtins.functionArgs add) values
{ a = 3; b = 5; }

Perfect, note from the example above that the intersectAttrs returns a set whose names are the intersection, and the attribute values are taken from the second set.

We're done, we have a way to get argument names from a function, and match with an existing set of attributes. This is our simple implementation of callPackage:

nix-repl> callPackage = set: f: f (builtins.intersectAttrs (builtins.functionArgs f) set)
nix-repl> callPackage values add
8
nix-repl> with values; add { inherit a b; }
8

Clearing up the syntax:

• We define a callPackage variable which is a function.
• First it accepts a set, and it returns another function accepting another parameter. In other words, let's simplify by saying it accepts two parameters.
• The second parameter is the function to "autocall".
• We take the argument names of the function and intersect with the set of all values.
• Finally we call the passed function f with the resulting intersection.

In the code above, I've also shown that the callPackage call is equivalent to directly calling add a b.

We achieved what we wanted. Automatically call functions given a set of possible arguments. If an argument is not found in the set, that's nothing special. It's a function call with a missing parameter, and that's an error (unless the function has varargs ... as explained in the 5th pill).

Or not. We missed something. Being able to override some of the parameters. We may not want to always call functions with values taken from the big set.
Then we add a further parameter, which takes a set of overrides:

nix-repl> callPackage = set: f: overrides: f ((builtins.intersectAttrs (builtins.functionArgs f) set) // overrides)
nix-repl> callPackage values add { }
8
nix-repl> callPackage values add { b = 12; }
15

Apart from the increasing number of parenthesis, it should be clear that we simply do a set union between the default arguments, and the overriding set.

Use callPackage to simplify the repository

Given our brand new tool, we can simplify the repository expression (default.nix).
Let me write it down first:

let
  nixpkgs = import <nixpkgs> {};
  allPkgs = nixpkgs // pkgs;
  callPackage = path: overrides:
    let f = import path;
    in f ((builtins.intersectAttrs (builtins.functionArgs f) allPkgs) // overrides);
  pkgs = with nixpkgs; {
    mkDerivation = import ./autotools.nix nixpkgs;
    hello = callPackage ./hello.nix { };
    graphviz = callPackage ./graphviz.nix { };
    graphvizCore = callPackage ./graphviz.nix { gdSupport = false; };
  };
in pkgs

Wow, there's a lot to say here:

• We renamed the old pkgs of the previous pill to nixpkgs. Our package set is now instead named pkgs. Sorry for the confusion.
• We needed a way to pass pkgs to callPackage somehow. Instead of returning the set of packages directly from default.nix, we first assign it to a let variable and reuse it in callPackage.
• For convenience, in callPackage we first import the file, instead of calling it directly. Otherwise for each package we would have to write the import.
• Since our expressions use packages from nixpkgs, in callPackage we use allPkgs, which is the union of nixpkgs and our packages.
• We moved mkDerivation in pkgs itself, so that it gets also passed automatically.

Note how easy is to override arguments in the case of graphviz without gd. But most importantly, how easy it was to merge two repositories: nixpkgs and our pkgs!

The reader should notice a magic thing happening. We're defining pkgs in terms of callPackage, and callPackage in terms of pkgs. That magic is possible thanks to lazy evaluation. 

Conclusion

The "callPackage" pattern has simplified a lot our repository. We're able to import packages that require some named arguments and call them automatically, given the set of all packages.

We've also introduced some useful builtin functions that allows us to introspect Nix functions and manipulate attributes. These builtin functions are not usually used when packaging software, rather to provide tools for packaging. That's why they are not documented in the nix manual.

Writing a repository in nix is an evolution of writing convenient functions for combining the packages. This demonstrates even more how nix is a generic tool to build and deploy something, and how suitable it is to create software repositories with your own conventions.

Next pill

...we will talk about the "override" design pattern. The graphvizCore seems straightforward. It starts from graphviz.nix and builds it without gd. Now I want to give you another point of view: what if we instead wanted to start from pkgs.graphviz and disable gd?

Pill 14 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.