Thursday, July 31, 2014

Nix pill 7: a working derivation

Welcome to the seventh Nix pill. In the previous sixth pill we introduced the notion of derivation in the Nix language. How to define a raw derivation and how to (try) to build it.

In this post, we will continue along the path, by creating a derivation that successfully builds something.
Then we try to package a real program: we compile a simple C file and create a derivation out of it, given a blessed toolchain.

I remind you how to enter the Nix environment: source ~/.nix-profile/etc/profile.d/nix.sh

Using a script as builder


What's the easiest way to run a sequence of commands for building something? A bash script. We write a custom bash script, and we want it to be our builder.
Given a builder.sh, we want the derivation to run bash builder.sh.

We don't use hash bangs in builder.sh, because at the time we are writing builder.sh we do not know the path to bash in the nix store. Yes, even bash is in the nix store, everything is in there.

We don't even use /usr/bin/env, because then we lose the cool stateless property of Nix. Not to say PATH gets cleared when building therefore it wouldn't work anyway.

In summary: we want the builder to be bash, and pass it an argument, builder.sh . Turns out the derivation function accepts an optional args attribute that is exactly used to pass arguments to the builder executable.

First of all, let's write our builder.sh in the current directory:
declare -xp
echo foo > $out
Ok, let's get the hang of it. I remind you from the previous pill, Nix creates the out path (not physically, you know the path) of the derivation. In the .drv there's a list of environment variables passed to the builder. One of them is $out.
What we have to do is to create something in $out, be it a file or a directory. In this case we are creating a file.

In addition, we also debug the environment variables during the build process. We cannot use env, because env is part of coreutils and we don't have a dependency to it. Not yet. It's plain bash, only bash.

Like for coreutils in the previous pill, we get a blessed bash for free from our magic nixpkgs stuff:
nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> "${bash}"
"/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45"

Great, with the usual trick we can then refer to bin/bash and create our derivation:
nix-repl> d = derivation { name = "foo"; builder = "${bash}/bin/bash"; args = [ ./builder.sh ]; system = builtins.currentSystem; }
nix-repl> :b d
these derivations will be built:
  /nix/store/ybnysdh5k6cjznhg4afjgbhr6czbwb4s-foo.drv
building path(s) `/nix/store/72v14vk4li47n8sx3z2ibd802ihpqyvx-foo'
these derivations will be built:
  /nix/store/ibwr68l3rjlx02kgz61dkkkrlpgljfxd-simple.drv
[...]
this derivation produced the following outputs:
  out -> /nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo

What? We did it! The contents of /nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo is really foo. We built our first derivation.

Note: we used ./builder.sh, not "./builder.sh". This way it gets parsed as path and Nix does wonders with it as we'll see later. Try using the string version, it will say it cannot find ./builder.sh , because that would be relative to the temporary build directory.

The builder environment


Let's inspect those debugged environment variables during the build process.
  • $HOME is not your home, and /homeless-shelter doesn't exist at all. We force packages to not depend upon $HOME during the build process.
  • $PATH plays the same game of $HOME
  • $NIX_BUILD_CORES and $NIX_STORE are nix configurations
  • $PWD and $TMP clearly shows nix created a temporary build directory.
  • Then builder, name, out and system are variables set due to the .drv contents.
And that's how we used the $out variable in our derivation, put stuff inside it. It's like Nix reserved a slot in the nix store for us, and we must fill it.

In terms of autotools, that will be the --prefix path. Yes, not the make DESTDIR, but the --prefix. That's a big difference between Nix and other package managers. That's the essence of stateless packaging. You don't install the package in a global common path under /, you install it in a local isolated path under your nix store slot.

The .drv contents


We added something else this time to the derivation. The args attribute. Let's see how this changed the .drv compared to the previous pill:
$ pp-aterm -i /nix/store/g6jj1mjzq68i66rbqyb3gpx3k0x606af-foo.drv
Derive(
  [("out", "/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo", "", "")]
, [("/nix/store/jdggv3q1sb15140qdx0apvyrps41m4lr-bash-4.2-p45.drv", ["out"])]
, ["/nix/store/5d1i99yd1fy4wkyx85iz5bvh78j2j96r-builder.sh"]
, "x86_64-linux"
, "/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45/bin/bash"
, ["/nix/store/5d1i99yd1fy4wkyx85iz5bvh78j2j96r-builder.sh"]
, [ ("builder", "/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45/bin/bash")
  , ("name", "foo")
  , ("out", "/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo")
  , ("system", "x86_64-linux")
  ]
)

Perfect, much like the usual .drv, except there's a list of arguments in there passed to the builder (bash), with the builder.sh... builder.sh... what? It's not pointing to my home's builder.sh .
Nix automatically copies files or directories needed for the build in the nix store, to ensure, for example, that they do not get changed during the build process. Also to ensure the deployment to be stateless and independent of the building machine.

Not only builder.sh is in the arguments passed to the builder, it's also in the input derivations.

Being builder.sh a plain file, it has no .drv associated with it. The store path will be computed based on the hash of its contents, and the name itself. We will talk about store paths at some point, in a dedicated pill.

Packaging a simple C executable


Start off writing a simple.c file:
void main () {
  puts ("Simple!");
}
And its simple_builder.sh:
export PATH="$coreutils/bin:$gcc/bin"
mkdir $out
gcc -o $out/simple $src
Don't spend time understanding where those variables come from. Let's write the derivation and build it:
nix-repl> :l <nixpkgs>
nix-repl> simple = derivation { name = "simple"; builder = "${bash}/bin/bash"; args = [ ./simple_builder.sh ]; gcc = gcc; coreutils = coreutils; src = ./simple.c; system = builtins.currentSystem; }
nix-repl> :b simple
this derivation produced the following outputs:
  out -> /nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-simple

Perfect, now you can run /nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-simple/simple in your shell.

Explanation

We added two new attributes to the derivation call, gcc and coreutils. Please, don't get an headache by reading "gcc = gcc". On the left, it's the attribute name of the set. On the right, there's an expression, it's the gcc derivation. Same goes for coreutils.

We also added the src attribute, nothing magic it's just a name with the ./simple.c path. Like for simple_builder.sh, simple.c will be added to the store.

The trick: every attribute in the set will be converted to a string and passed as environment variable to the builder. Now it's all clear. $coreutils and $gcc are then the out paths of the derivations, and of course appending "/bin" will point to their binaries.

Same goes for the src variable, $src is the path to simple.c in the nix store. As an exercise, pretty print the .drv file. You'll see in the input derivations simple_builder.sh and simple.c files, then bash, gcc and coreutils .drv files. Plus the new environment variables described above.

In simple_builder.sh we set the PATH for gcc and coreutils binaries, so that gcc can find the necessary binaries like "cat", "readlink", ecc. .
Then we create $out as a directory and inside it we put the binary.

Note: instead of running plain gcc (or mkdir), it would have been equivalent to run $gcc/bin/gcc (or $coreutils/bin/mkdir).

Enough with nix-repl


Drop out of nix-repl, write a simple.nix file:
with (import <nixpkgs> {});
derivation {
  name = "simple";
  builder = "${bash}/bin/bash";
  args = [ ./simple_builder.sh ];
  inherit gcc coreutils;
  src = ./simple.c;
  system = builtins.currentSystem;
}
Now you can build it with nix-build simple.nix. It will create a symlink "result" in the current directory, pointing to the out path of the derivation.

The nix-build tool does two main jobs:
  • nix-instantiate: parse simple.nix and return the .drv file relative to the parsed derivation set
  • nix-store -r: realise the .drv, which actually builds the derivation.
Finally creates the symlink.

Look the first line of the .nix file. We have an "import" function call nested in a "with" expression. I recall import accepts one argument, a nix file to parse. In this case it parsed a function out of the file.
Afterwards we call the parsed function with the empty set. We saw this already in nix pill 5.

Let me underline it: "import <nixpkgs> {}" are two function calls, not one. Read it like "(import <nixpkgs>) {}".

The final returned value of that import is a set. To simplify it: it's a set of derivations. Using the "with" expression we drop them into the scope. We basically simulated what :l does in nix-repl, so we can easily access derivations such as bash, gcc and coreutils.

Then we meet the inherit keyword. Doing inherit foo, is the same as doing foo = foo. Doing inherit foo bar, is the same as doing foo = foo; bar = bar. Literally.
This syntax only makes sense inside sets. Don't think it's black magic, it's just a convenience to avoid repeating the same name twice, once for the attribute name, once for the variable in the scope.

Next pill


...we will generalize the builder. If you have noticed, we have written two builder.sh files in this post. We would like to have a generic builder script instead, especially since every builder script goes in the nix store: that's a waste.

Again. Is it really that hard to package stuff in Nix? No, here we're studying the fundamentals of Nix.

Pill 8 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Tuesday, July 29, 2014

Nix pill 6: our first derivation

Welcome to the sixth Nix pill. In the previous fifth pill we introduced functions and imports. Functions and imports are very simple concepts that allows for building complex abstractions and composition of modules to build a flexible Nix system.

In this post we finally arrived to writing a derivation. Derivations are the building blocks of a Nix system, from a file system view point. The Nix language is used to describe such derivations.

I remind you how to enter the Nix environment: source ~/.nix-profile/etc/profile.d/nix.sh

The derivation function


The derivation built-in function is used to create derivations. I invite you to read the link in the Nix manual about the derivation built-in. A derivation from a Nix language view point is simply a set, with some attributes. Therefore you can pass the derivation around with variables like anything else.
That's where the real power comes in. 

The derivation function receives a set as first argument. This set requires at least the following three attributes:
  • name: the name of the derivation. In the nix store the format is hash-name, that's the name.
  • system: is the name of the system in which the derivation can be built. For example, x86_64-linux.
  • builder: it is the binary program that builds the derivation.
First of all, what's the name of our system as seen by nix?
nix-repl> builtins.currentSystem
"x86_64-linux"
Let's try to fake the name of the system:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> d
«derivation /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv»
Oh oh, what's that? Did it build the derivation? No it didn't, but it did create the .drv file. nix-repl does not build derivations unless you tell to do so.

Digression about .drv files


What's that .drv file? It is the specification of how to build the derivation, without all the Nix language fuzz.
Before continuing, some analogies with the C language:
  • .nix files are like .c files
  • .drv files are intermediate files like .o files. The .drv describes how to build a derivation, it's the bare minimum information.
  • out paths are then the product of the build
Both drv paths and out paths are stored in the nix store as you can see.

What's in that .drv file? You can read it, but it's better to pretty print it.
$ nix-env -i strategoxt
If you feel the above command being too slow (ignore the meaning of the command below, just do it):
$ nix-env -iA strategoPackages018.strategoxt -f '<nixpkgs>'
The installed pp-aterm program can be used to pretty print .drv files:
$ pp-aterm -i /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
Derive(
  [("out", "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname", "", "")]
, []
, []
, "mysystem"
, "mybuilder"
, []
, [ ("builder", "mybuilder")
  , ("name", "myname")
  , ("out", "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname")
  , ("system", "mysystem")
  ]
Ok we can see there's an out path, but it does not exist yet. We never told Nix to build it, but we know beforehand where the build output will be. Why?

Think, if Nix ever built the derivation just because we accessed it in Nix, we would have to wait a long time if it was, say, Firefox. That's why Nix let us know the path beforehand and keep evaluating the Nix expressions, but it's still empty because no build was ever made.

Important: the hash of the out path is based solely on the input derivations in the current version of Nix, not on the contents of the build product. It's possible however to have content-addressable derivations for e.g. tarballs as we'll see later on.

Many things are empty in that .drv, however I write a summary of the .drv format for you:
  1. The output paths (they can be multiple ones). By default nix creates one out path called "out".
  2. The list of input derivations. It's empty because we are not referring to any other derivation. Otherwise, there would a list of other .drv files.
  3. The system and the builder executable (yes, it's a fake one).
  4. Then a list of environment variables passed to the builder.
That's it, the minimum necessary information to build our derivation.

Important note: the environment variables passed to the builder are just those you see in the .drv plus some other Nix related configuration (number of cores, temp dir, ...). The builder will not inherit any variable from your running shell, otherwise builds would suffer from non-determinism.

Back to our fake derivation


Let's build our really fake derivation:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> :b d
[...]
these derivations will be built:
  /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
building path(s) `/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname'
error: a `mysystem' is required to build `/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv', but I am a `x86_64-linux'
The :b is a nix-repl specific command to build a derivation. You can see more commands with :? .
So in the output you can see that it takes the .drv as information on how to build the derivation. Then it says it's trying to produce our out path. Finally the error we were waiting for: that derivation can't be built on our system.

We're doing the build inside nix-repl, but what if we don't want to use nix-repl?
You can realise a .drv with:
$ nix-store -r /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
You will get the same output as before.

Let's fix the system attribute:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = builtins.currentSystem; }
nix-repl> :b d
[...]
build error: invalid file name `mybuilder'
A step forward: of course, that "mybuilder" executable does not really exist. Stop for a moment.

What's in a derivation set


I find useful to inspect the returned value from the derivation function for you.
First of all, the returned value is a plain set:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> builtins.isAttrs d
true
nix-repl> builtins.attrNames d
[ "all" "builder" "drvAttrs" "drvPath" "name" "out" "outPath" "outputName" "system" "type" ]
You can guess what builtins.isAttrs does, it returns true if the argument is a set. While builtins.attrNames returns a list of keys of the given set. Some kind of reflection, you might say.

Start from drvAttrs:
nix-repl> d.drvAttrs
{ builder = "mybuilder"; name = "myname"; system = "mysystem"; }
That's basically the input we gave to the derivation function. Also d.name, d.system and d.builder attributes are straight the ones we gave as input.
nix-repl> (d == d.out)
true
So out is just the derivation itself, it seems weird but the reason is that we only have one output from the derivation. That's also the reason why d.all is a singleton. We'll see multiple outputs later.

The d.drvPath is the path of the .drv file: /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv .

Something interesting is the type attribute. It's "derivation". Nix does add a little of magic to sets with type derivation, but not that much. To let you understand, you can create yourself a set with that type, it's a simple set:
nix-repl> { type = "derivation"; }
«derivation ???»
Of course it has no other information, so Nix doesn't know what to say :-) But you get it, the type = "derivation" is just a convention for Nix and for us to understand the set is a derivation.

When writing packages, we are interested in the outputs. The other metadata is needed for Nix to know how to create the drv path and the out path.
The outPath attribute is the build path in the nix store: /nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname .

Referring to other derivations


Just like dependencies in other package managers, how do we refer to other packages? How do we refer to other derivations in terms of files on the disk?
We use the outPath. The outPath tells where the files are of that derivation. To make it more convenient, Nix is able to do a conversion from a derivation set to a string.
nix-repl> d.outPath
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"
nix-repl> builtins.toString d
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"
Nix does the "set to string conversion" as long as there is the outPath attribute (much like a toString method in other languages):
nix-repl> builtins.toString { outPath = "foo"; }
"foo"
nix-repl> builtins.toString { a = "b"; }
error: cannot coerce a set to a string, at (string):1:1
Say we want to use binaries from coreutils (ignore the nixpkgs etc.):
nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> coreutils
«derivation /nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv»
nix-repl> builtins.toString coreutils
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21"
Apart the nixpkgs stuff, just think we added to the scope a series of variables. One of them is coreutils. It is the derivation of the coreutils package you all know of from other Linux distributions. 
It contains basic binaries for GNU/Linux systems (you may have multiple derivations of coreutils in the nix store, no worries):
$ ls /nix/store/*coreutils*/bin
[...]
I remind you, inside strings it's possible to interpolate Nix expressions with ${...}:
nix-repl> "${d}"
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"
nix-repl> "${coreutils}"
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21"
That's very convenient, because then we could refer to e.g. the bin/true binary like this:
nix-repl> "${coreutils}/bin/true"
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true"

An almost working derivation


In the previous attempt we used a fake builder, "mybuilder" which obviously does not exist. But we can use for example bin/true, which always exits with 0 (success).
nix-repl> :l <nixpkgs>
nix-repl> d = derivation { name = "myname"; builder = "${coreutils}/bin/true"; system = builtins.currentSystem; }
nix-repl> :b d
[...]
builder for `/nix/store/d4xczdij7xazjfm5kn4nmphx63mpv676-myname.drv' failed to produce output path `/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-myname'
Another step forward, it executed the builder (bin/true), but the builder did not create the out path of course, it just exited with 0.

Obvious note: everytime we change the derivation, a new hash is created.

Let's examine the new .drv now that we referred to another derivation:

$ pp-aterm -i /nix/store/d4xczdij7xazjfm5kn4nmphx63mpv676-myname.drv
Derive(
  [("out", "/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-myname", "", "")]
, [("/nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv", ["out"])]
, []
, "x86_64-linux"
, "/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true"
, []
, [ ("builder", "/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true")
  , ("name", "myname")
  , ("out", "/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-myname")
  , ("system", "x86_64-linux")
  ]

Aha! Nix added a dependency to our myname.drv, it's the coreutils.drv. Before doing our build, Nix should build the coreutils.drv. But since coreutils is already in our nix store, no build is needed, 
it's already there with out path /nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21 .

When is the derivation built


Nix does not build derivations during evaluation of Nix expressions. In fact, that's why we have to do ":b drv" in nix-repl, or use nix-store -r in the first place.

An important separation is made in Nix:
  • Instantiate/Evaluation time: the Nix expression is parsed, interpreted and finally returns a derivation set. During evaluation, you can refer to other derivations because Nix will create .drv files and we will know out paths beforehand. This is achieved with nix-instantiate.
  • Realise/Build time: the .drv from the derivation set is built, first building .drv inputs (build dependencies). This is achieved with nix-store -r.
Think of it as of compile time and link time like with C/C++ projects. You first compile all source files to object files. Then link object files in a single executable.

In Nix, first the Nix expression (usually in a .nix file) is compiled to .drv, then each .drv is built and the product is installed in the relative out paths.

Conclusion


Is that complicated to create a package for Nix? No it's not.

We're walking through the fundamentals of Nix derivations, to understand how they work, how they are represented.
Packaging in Nix is certainly easier than that, but we're not there yet in this post. More Nix pills are needed.

With the derivation function we provide a set of information on how to build a package, and we get back the information about where the package was built.
Nix converts a set to a string when there's an outPath, that's very convenient. With that, it's easy to refer to other derivations.

When Nix builds a derivation, it first creates a .drv file from a derivation expression, and uses it to build the output. It does so recursively for all the dependencies (inputs). It "executes" the .drv files like a machine. Not much magic after all.


Next pill


...we will finally write our first working derivation. Yes, this post is about "our first derivation", but I never said it was a working one

Pill 7 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Monday, July 28, 2014

Nix pill 5: functions and imports

Welcome to the fifth Nix pill. In the previous fourth pill we touched the Nix language for a moment. We introduced basic types and values of the Nix language, and basic expressions such as "if", "with" and "let". I invite you to re-read about these expressions and play with them in the repl.

Functions help to build reusable components in a big repository like nixpkgs. The Nix manual has a great explanation of functions. Let's go: pill on one hand, Nix manual on the other hand.

I remind you how to enter the Nix environment: source ~/.nix-profile/etc/profile.d/nix.sh

Nameless and single parameter


Functions are anonymous (lambdas), and only have a single parameter. The syntax is extremely simple. Type the parameter name, then ":", then the body of the function.
nix-repl> x: x*2
«lambda»
So here we defined a function that takes a parameter x, and returns x*2. The problem is that we cannot use it in any way, because it's unnamed... joke!
We can store functions in variables.
nix-repl> double = x: x*2
nix-repl> double
«lambda»
nix-repl> double 3
6
As usual, please ignore the special syntax for assignments inside nix-repl.
So, we defined a function x: x*2 that takes one parameter x, and returns x*2. This function is then assigned to the variable double.
Finally we did our first function call: double 3.
Big note: it's not like many other programming languages where you write double(3). It really is double 3.

In summary: to call a function, name the variable, then space, then the argument. Nothing else to say, it's as easy as that.

More than one parameter


How do we create a function that accepts more than one parameter? For people not used to functional programming, this may take a while to grasp. Let's do it step by step.
nix-repl> mul = a: (b: a*b)
nix-repl> mul
«lambda»
nix-repl> mul 3
«lambda»
nix-repl> (mul 3) 4
12
We defined a function that takes the parameter "a", the body returns another function. This other function takes a parameter "b" and returns a*b.
Therefore, calling "mul 3" returns this kind of function: b: 3*b. In turn, we call the returned function with 4, and get the expected result.

You don't have to use parenthesis at all, Nix has sane priorities when parsing the code:
nix-repl> mul = a: b: a*b
nix-repl> mul
«lambda»
nix-repl> mul 3
«lambda»
nix-repl> mul 3 4
12
nix-repl> mul (6+7) (8+9)
221
Much more readable, you don't even notice that functions only receive one argument.
Since the argument is separated by a space, to pass more complex expressions you need parenthesis. In other common languages you would write  mul(6+7, 8+9).

Given that functions have only one parameter, it is straightforward to use partial application:
nix-repl> foo = mul 3
nix-repl> foo 4
12
nix-repl> foo 5
15
We stored the function returned by mul 3 into a variable foo, then reused it.

Arguments set


Now this is a very cool feature of Nix. It is possible to pattern match over a set in the parameter. We write an alternative version of mul = a: b: a*b first by using a set as argument, then using pattern matching.
nix-repl> mul = s: s.a*s.b
nix-repl> mul { a = 3; b = 4; }
12
nix-repl> mul = { a, b }: a*b
nix-repl> mul { a = 3; b = 4; }
12
In the first case we defined a function that accepts a single parameter. We then access attributes "a" and "b" from the given set.
Note how the parenthesis-less syntax for function calls is very elegant in this case, instead of doing mul({ a=3; b=4; }) in other languages.

In the second case we defined an arguments set. It's like defining a set, except without values. We require that the passed set contains the keys "a" and "b". Then we can use those "a" and "b" in the function body directly.
nix-repl> mul { a = 3; b = 4; c = 6; }
error: anonymous function at (string):1:2 called with unexpected argument `c', at (string):1:1
nix-repl> mul { a = 3; }
error: anonymous function at (string):1:2 called without required argument `b', at (string):1:1
Only a set with exactly the attributes required by the function is accepted, nothing more, nothing less.

Default and variadic attributes


It is possible to specify default values of attributes in the arguments set:
nix-repl> mul = { a, b ? 2 }: a*b
nix-repl> mul { a = 3; }
6
nix-repl> mul { a = 3; b = 4; }
12
Also you can allow passing more attributes (variadic) than the expected ones:
nix-repl> mul = { a, b, ... }: a*b
nix-repl> mul { a = 3; b = 4; c = 2; }
However, in the function body you cannot access the "c" attribute. The solution is to give a name to the given set with the @-pattern:
nix-repl> mul = s@{ a, b, ... }: a*b*s.c
nix-repl> mul { a = 3; b = 4; c = 2; }
24
That's it, you give a name to the whole parameter with name@ before the set pattern.

Advantages of using argument sets:
  • Named unordered arguments: you don't have to remember the order of the arguments.
  • You can pass sets, that adds a whole new layer of flexibility and convenience. 
Disadvantages:
  • Partial application does not work with argument sets. You have to specify the whole attribute set, not part of it.
You may find similarities with Python **kwargs.

Imports


The "import" function is built-in and provides a way to parse a .nix file. The natural approach is to define each component in a .nix file, then compose by importing these files.

Let's start with the bare metal.

a.nix:
3
b.nix:
4
mul.nix:
a: b: a*b
nix-repl> a = import ./a.nix
nix-repl> b = import ./b.nix
nix-repl> mul = import ./mul.nix
nix-repl> mul a b
12
Yes it's really that straight. You import a file, and it gets parsed as expression. Note that the scope of the imported file does not inherit the scope of the importer.

test.nix:
x
nix-repl> let x = 5; in import ./test.nix
error: undefined variable `x' at /home/lethal/test.nix:1:1
So how do we pass information to the module? Use functions, like we did with mul.nix .
A more complex example:

test.nix:
{ a, b ? 3, trueMsg ? "yes", falseMsg ? "no" }:
if a > b
  then builtins.trace trueMsg true
  else builtins.trace falseMsg false
nix-repl> import ./test.nix { a = 5; trueMsg = "ok"; }
trace: ok
true
Explaining:
  • In test.nix we return a function. It accepts a set, with default attributes b, trueMsg and falseMsg.
  • builtins.trace is a built-in function that takes two arguments. The first is the message to display, the second is the value to return. It's usually used for debugging purposes.
  • Then we import test.nix, and call the function with that set.
So when is the message shown? Only when it's in need to be evaluated.

Next pill


...we will finally write our first derivation.

Nix pill 6 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Saturday, July 26, 2014

Nix pill 4: the basics of the language

Welcome to the fourth Nix pill. In the previous third pill we entered the Nix environment. We installed software as user, managed the profile, switched between generations, and queried the nix store. That's the very basics of nix administration somehow.

The Nix language is used to write derivations. The nix-build tool is used to build derivations. Even as a system administrator that wants to customize the installation, it's necessary to master Nix. Using Nix for your jobs means you get the features we saw in the previous pills for free.

The syntax is very uncommon thus looking at existing examples may lead to thinking that there's a lot of magic behind. In reality, it's only about writing utility functions for making things convenient.
On the other hand, this same syntax is great for describing packages.

Important: in Nix, everything is an expression, there are no statements. This is common to many functional languages.
Important: values in Nix are immutable.

Value types


We've installed nix-repl in the previous pill. If you didn't, nix-env -i nix-repl. The nix-repl syntax is slightly different than nix syntax when it comes to assigning variables, but no worries.  I prefer playing with nix-repl with you before cluttering your mind with more complex expressions.

Launch nix-repl. First of all, nix supports basic arithmetic operations: +, -, and *. The integer division can be done with builtins.div.
nix-repl> 1+3
4
nix-repl> builtins.div 6 3
2
Really, why doesn't nix have basic operations such as division? Because it's not needed for creating packages. Nix is not a general purpose language, it's a domain-specific language for writing packages.
Just think that builtins.div is not being used in the whole of our nixpkgs repository: it's useless.

Other operators are ||, && and ! for booleans, and relational operators such as !=, ==, <, >, <=, >=. In Nix, <, >, <= and >= are not much used. There are also other operators we will see in the course of this series.

Nix has integer (not floating point), string, path, boolean and null simple types. Then there are lists, sets and functions. These types are enough to build an operating system.

Nix is strongly typed, but it's not statically typed. That is, you cannot mix strings and integers, you must first do the conversion. 

Try to use / between two numbers:
nix-repl> 2/3
/home/nix/2/3
Nix parsed 2/3 as a relative path to the current directory. Paths are parsed as long as there's a slash. Therefore to specify the current directory, use ./.
In addition, Nix also parses urls.

Not all urls or paths can be parsed this way. If a syntax error occurs, it's still possible to fallback to plain strings. Parsing urls and paths are convenient for additional safety.


Identifiers


Not much to say, except that dash (-) is allowed in identifiers. That's convenient since many packages use dash in its name. In fact:
nix-repl> a-b
error: undefined variable `a-b' at (string):1:1
nix-repl> a - b
error: undefined variable `a' at (string):1:1
As you can see, a-b is parsed as identifier, not as operation between a and b.

Strings


It's important to understand the syntax for strings. When reading Nix expressions at the beginning, you may find dollars ($) ambiguous in their usage.
Strings are enclosed by double quotes ("), or two single quotes ('').
nix-repl> "foo"
"foo"
nix-repl> ''foo''
"foo"
In python you can use also single quotes for strings like 'foo', but not in Nix.

It's possible to interpolate whole Nix expressions inside strings with ${...} and only with ${...}, not $foo or {$foo} or anything else.
nix-repl> foo = "strval"
nix-repl> "$foo"
"$foo"
nix-repl> "${foo}"
"strval"
nix-repl> "${2+3}"
error: cannot coerce an integer to a string, at (string):1:2
Note: ignore the foo = "strval" assignment, it's nix-repl special syntax.

As said previously, you cannot mix integers and strings. You explicitly need conversion. We'll see this later: function calls are another story.

Using the syntax with two single quotes, it's useful for writing double quotes inside strings instead of escaping:
nix-repl> ''test " test''
"test \" test"
nix-repl> ''${foo}''
"strval"
Escaping ${...} within double quoted strings is done with the backslash. Within two single quotes, it's done with '':
nix-repl> "\${foo}"
"${foo}"
nix-repl> ''test ''${foo} test''
"test ${foo} test"
No other magic about strings for now.

Lists


Lists are a sequence of expressions delimited by space (not comma):
nix-repl> [ 2 "foo" true (2+3) ]
[ 2 "foo" true 5 ]
Lists, like anything else in Nix, are immutable. Adding or removing elements from a list is possible, but will return a new list.

Sets


Sets are an association between a string key and a Nix expression. Keys can only be strings. When writing sets you can also use identifiers as keys.
nix-repl> s = { foo = "bar"; a-b = "baz"; "123" = "num"; }
nix-repl> s
{ 123 = "num"; a-b = "baz"; foo = "bar"; }
Note: here the string representation from nix is wrong, you can't write { 123 = "num"; } because 123 is not an identifier.
You need semicomma (;) after every key-value assignment.

For those reading Nix expressions from nixpkgs: do not confuse sets with argument sets used in functions.

To access elements in the set:
nix-repl> s.a-b
"baz"
nix-repl> s."123"
"num"
Yes, you can use strings for non-identifiers to address keys in the set.

You cannot refer inside a set to elements of the same set:
nix-repl> { a = 3; b = a+4; }
error: undefined variable `a' at (string):1:10
To do so, use recursive sets:
nix-repl> rec { a= 3; b = a+4; }
{ a = 3; b = 7; }
This will be very convenient when defining packages.

If expression


Expressions, not statements.
nix-repl> a = 3
nix-repl> b = 4
nix-repl> if a > b then "yes" else "no"
"no"
You can't have only the "then" branch, you must specify also the "else" branch, because an expression must have a value in all cases.


Let expression


This kind of expression is used to define local variables to inner expressions.
nix-repl> let a = "foo"; in a
"foo"
The syntax is: first assign variables, then "in" expression. The overall result will be the final expression after "in".
nix-repl> let a = 3; b = 4; in a + b
7
Let's write two let expressions, one inside the other:
nix-repl> let a = 3; in let b = 4; in a + b
7
With let you cannot assign twice to the same variable. You can however shadow outer variables:
nix-repl> let a = 3; a = 8; in a
error: attribute `a' at (string):1:12 already defined at (string):1:5
nix-repl> let a = 3; in let a = 8; in a
8
You cannot refer to variables in a let expression outside of it:
nix-repl> let a = (let b = 3; in b); in b
error: undefined variable `b' at (string):1:31
You can refer to variables in the let expression when assigning variables like with recursive sets:
nix-repl> let a = 4; b = a + 5; in b
9
So beware when you want to refer to a variable from the outer scope, but it's being defined in the current let expression. Same applies to recursive sets.

With expression


This kind of expression is something you hardly see in other languages. Think of it like a more granular "using" of C++, or "from module import *" from Python. You decide per-expression when to include symbols into the scope.
nix-repl> longName = { a = 3; b = 4; }
nix-repl> longName.a + longName.b
7
nix-repl> with longName; a + b
7
That's it, it takes a set and includes symbols in the scope of the inner expression. Of course, only valid identifiers from the set keys will be included.
If a symbol exists in the outer scope and also in the "with" scope, it will not be shadowed. You can however still refer to the set:
nix-repl> let a = 10; in with longName; a + b
14
nix-repl> let a = 10; in with longName; longName.a + b
7

Laziness


Nix evaluates expression only when needed. This is a great feature when working with packages.
nix-repl> let a = builtins.div 4 0; b = 6; in b
6
Since "a" is not needed, there's no error about division by zero, because the expression is not in need to be evaluated.
That's why we can have all the packages defined here, yet access to specific packages very fast.

Next pill


...we will talk about functions and imports. In this pill I've tried to avoid function calls as much as possible, otherwise the post would have been too long.

Nix pill 5 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Friday, July 25, 2014

Nix pill 3: enter the environment

Welcome to the third Nix pill. In the previous second pill we have installed Nix on our running system. Now we can finally play with it a little, things also apply to NixOS users.

Enter the environment


In the previous pill we created a nix user, so let's start by switching user with su - nix. If your ~/.profile got evaluated, then you should now be able to run commands like nix-env and nix-store.
If that's not the case:
$ source ~/.nix-profile/etc/profile.d/nix.sh


I remind you, ~/.nix-profile/etc points to the nix-1.7 derivation. At this point, we are in our nix user profile.


Install something


Finally something practical! Installation in the nix environment is an interesting process. Let's install nix-repl, a simple command line tool for playing with the Nix language. Yes, Nix is a pure, lazy, functional language, not only a set of tools to manage derivations.

Back to the installation:
$ nix-env -i nix-repl

installing `nix-repl-1.7-1734e8a'
these paths will be fetched (18.61 MiB download, 69.53 MiB unpacked):
[...]
building path(s) `/nix/store/f01lfzbw7n0yzhsjd33xfj77li9raljv-user-environment'
created 24 symlinks in user environment
Now you can run nix-repl. Things to notice:
  • We did install software as user, only for the nix user.
  • It created a new user environment. That's a new generation of our nix user profile.
  • The nix-env tool manages environments, profiles and their generations.
  • We installed nix-repl by derivation name minus the version. I repeat: we did specify the derivation name (minus the version) to install.
We can list generations without walking through the /nix hierarchy:
$ nix-env --list-generations
   1   2014-07-24 09:23:30   
   2   2014-07-25 08:45:01   (current)
List installed derivations:
$ nix-env -q
nix-1.7
nix-repl-1.7-1734e8a
So, where did nix-repl really got installed? which nix-repl is ~/.nix-profile/bin/nix-repl which points to the store.
We can also list the derivation paths with nix-env -q --out-path . So that's how those derivation paths are called: the output of a build.

Path merging


At this point you sure have the necessity to run "man". Even if you already have man system-wide outside of the nix environment, you can install and use it within nix with nix-env -i man. As usual, a new generation will be created, and ~/.nix-profile will point to it.
Let's inspect the profile a bit:
$ ls -l ~/.nix-profile/
dr-xr-xr-x 2 nix nix 4096 Jan  1  1970 bin
lrwxrwxrwx 1 nix nix   55 Jan  1  1970 etc -> /nix/store/clnpynyac3hx3a6z5lsy893p7b4rwnyf-nix-1.7/etc
[...]
Now that's interesting. When only nix-1.7 was installed, bin/ was a symlink to nix-1.7. Now it's a real directory, no symlink.
$ ls -l ~/.nix-profile/bin/
[...]
man -> /nix/store/83cn9ing5sc6644h50dqzzfxcs07r2jn-man-1.6g/bin/man
[...]
nix-env -> /nix/store/clnpynyac3hx3a6z5lsy893p7b4rwnyf-nix-1.7/bin/nix-env
[...]
nix-repl -> /nix/store/0fcl92chxbbs8axb994rg12vxddg1ivs-nix-repl-1.7-1734e8a/bin/nix-repl
[...]
All clear. nix-env merged the paths from the installed derivations. which man points to the nix profile, rather than the system man, because ~/.nix-profile/bin is at the head of $PATH.

Rollback / switch generation


The last command installed "man". We should be at generation #3, unless you changed something in the middle. Let's say we want to rollback to the old generation:
$ nix-env --rollback
switching from generation 3 to 2
Now nix-env -q does not list "man" anymore. ls -l `which man` should now be your system installed one.
Enough with the joke, let's go back to the last generation:
$ nix-env -G 3
switching from generation 2 to 3
I invite you to read the manpage of nix-env. nix-env requires an operation to perform, then there are common options for all operations, and there are options specific to an operation.

You can of course also delete and upgrade packages.

Querying the store


So far we learned how to query and manipulate the environment. But all of the environment components point to the store.
To query and manipulate the store, there's the nix-store command. We can do neat things, but we'll only see some queries for now.

Show direct runtime dependencies of nix-repl:
$ nix-store -q --references `which nix-repl`
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19
/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3
/nix/store/25lg5iqy68k25hdv17yac72kcnwlh4lm-boehm-gc-7.2d
/nix/store/g21di262aql6xskx15z3qiw3zh3wmjlb-nix-1.7
/nix/store/jxs2k83npdin18ysnd104xm4sy29m8xp-readline-6.2
The argument to nix-store can be anything as long as it points to the nix store. It will follow symlinks.
It may not make sense for you right now, but let's print reverse dependencies of nix-repl:
$ nix-store -q --referrers `which nix-repl`
/nix/store/8rj57vahlndqwg4q095x5qvfa1g4rmvr-env-manifest.nix
/nix/store/9c8ak2h7c6vbr9kqk8i2n96bwg55br7y-env-manifest.nix
/nix/store/dr1y2saask2k4y2wrmxw443302pp02z2-user-environment
/nix/store/f01lfzbw7n0yzhsjd33xfj77li9raljv-user-environment
Did you expect it? Our environments depend upon nix-repl. Yes, the environments are in the store, and since there are symlinks to nix-repl, therefore the environment depends upon nix-repl.
It lists two environments, generation 2 and generation 3.

The manifest.nix file contains metadata about the environment, such as which derivations are installed. So that nix-env can list them, upgrade or remove them. Guess what, the current manifest.nix can be found in ~/.nix-profile/manifest.nix.

Closures


The closure of a derivation is the list of all dependencies, recursively, down to the bare minimum necessary to use that derivation.
$ nix-store -qR `which man`
[...]

Copying all those derivations to the nix store of another machine makes you able to run "man" out of the box on that other machine. That's the base of nix deployment, you can already foresee the potential when deploying software in the cloud (hint: nix-copy-closure and nix-store --export).

A nicer view of the closure:
$ nix-store -qR --tree `which man`
[...]

With the above command, you can know exactly why a runtime dependency, being it direct or indirect, has been picked for a given derivation.

Same applies to environments of course. As an exercise run nix-store -qR --tree ~/.nix-profile , see that the first children are direct dependencies of the user environment: the installed derivations, and the manifest.nix.

Dependency resolution


There isn't anything like apt which solves a SAT problem in order to satisfy dependencies with lower and upper bounds on versions. Because there's no need. A derivation X depends on derivation Y, always.

Fancy disrupt


$ nix-env -e '*'
uninstalling `man-1.6g'
uninstalling `nix-repl-1.7-1734e8a'
uninstalling `nix-1.7'
[...]

Ops, that uninstalled all derivations from the environment, including nix. We are not able to run nix-env, what now?
Environments are a convenience for the user, but Nix is still there, in the store!

First pick one nix-1.7 derivation: ls /nix/store/*nix-1.7, say /nix/store/g21di262aql6xskx15z3qiw3zh3wmjlb-nix-1.7.

The first possibility is to rollback:
$ /nix/store/g21di262aql6xskx15z3qiw3zh3wmjlb-nix-1.7/bin/nix-env --rollback

The second possibility is to install nix, thus creating a new generation:
$ /nix/store/g21di262aql6xskx15z3qiw3zh3wmjlb-nix-1.7/bin/nix-env -i /nix/store/g21di262aql6xskx15z3qiw3zh3wmjlb-nix-1.7

Channels


So where are we getting packages from? We said something already in pill 2. There's a list of channels from which we get packages, usually we use a single channel. The tool to manage channels is nix-channel.
$ nix-channel --list
nixpkgs http://nixos.org/channels/nixpkgs-unstable

That's basically the contents of ~/.nix-channels. Note: ~/.nix-channels is not a symlink to the nix store!

To update the channel run nix-channel --update . It will download the new nix expressions (descriptions of the packages), create a new generation of the channels profile and unpack under ~/.nix-defexpr/channels .
That's much similar to apt-get update.

Conclusion


We learned how to query the user environment and to manipulate it by installing and uninstalling software. Upgrading software is as straight as it gets by reading the manual (nix-env -u '*' will upgrade all packages in the environment).
Everytime we change the environment, a new generation gets created. Switching between generations is easy and immediate.

Then we queried the store. We inspected the dependencies and reverse dependencies of store paths.
We still see symlinks to compose paths from the nix store, our lovely trick.

Quick analogy with programming languages. You have the heap with all the objects, that's the nix store. You have objects that point to other objects, those are the derivations. Will be this the right path?

Next pill


...we will learn the basics of the Nix language. The Nix language is used to describe how to build derivations, and it's the base for everything else including NixOS. Therefore it's very important to understand the syntax and the semantics.

Nix pill 4 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Thursday, July 24, 2014

Nix pill 2: install on your running system

Welcome to the second Nix pill. In the first pill we briefly described Nix.
Now we'll install Nix on our running system and understand what changed in our system after the installation.

Nix installation is as easy as installing any other package. It will not revolutionize our system, it will stay in its own place out of our way.

Download


You can grab the last stable tarball (nix 1.7 during this writing) or the package for your distro here: http://hydra.nixos.org/release/nix/nix-1.7 .
At the time you are reading, there may be nix 1.8. You can see a list of releases here: http://hydra.nixos.org/project/nix#tabs-releases .

I prefer using the precompiled binaries, because it's a nice self-contained environment, just works and it's a little less invasive (in that it does not install anything under /etc or /usr). So in this series I will use the nix bootstrap distribution.
In the download page above you can find binaries for linux i686, x86_64 and darwin.

Note: if you are using a rolling distribution however, I suggest not to install the package (for example on Debian sid) because you will experience broken perl dependencies when running nix tools.

Installation


To ensure we don't mess with the system, you could create a custom user to let him own the Nix store, let's call it "nix".

As root:
# adduser nix
# mkdir -m 0755 /nix && chown nix /nix
From now on, all the operations we do on the shell are done from this nix user:
# su - nix
$ tar -xf nix-1.7-x86_64-linux.tar.bz2
$ cd nix-1.7-x86_64-linux
$ ./install

My pills are not a simple tutorial, there's are several articles out there to learn the basics of nix and unix. We'll instead walk through the nix system to understand the fundamentals.

First thing to note: those stuff in nix store refer to software in nix store itself. It doesn't use libc from our system or whatelse. It's a self-contained nix bootstrap.

Quick note: in a normal installation, the store is owned by root and multiple users can install and build software through a nix daemon.

So small nix store


Start inspecting the output of the install command:
copying Nix to /nix/store..........................

Effectively, that's right the /nix/store we were talking in the first pill. The contents is the strictly necessary software to bootstrap a Nix system. You can see bash, core utils, the toolchain, perl libraries, sqlite and nix itself with its own tools and libnix.
You surely noticed that in /nix/store not only directories are allowed, but also files, always in the form hash-name.

The holy database


Right after copying the store, the installation process initializes the database with the current information:
initialising Nix database...

Oh Nix also has a database. It's under /nix/var/nix/db. It is an sqlite database that keeps track of the dependencies between derivations.
The schema is very simple: there's a table of valid paths, mapping from auto increment integer to store path.
Then there's a dependency relation from one path to other paths.
Inspect it running /nix/store/*sqlite*/bin/sqlite3 /nix/var/nix/db/db.sqlite

Important rule: never change /nix/store manually because that wouldn't be in sync with the sqlite db, unless you know what you are doing.

The first profile


Then we discover the profile concept during the installation:
creating /home/nix/.nix-profile
installing `nix-1.7'
building path(s) `/nix/store/xxhdgml3rshn8mkaqxb86gp4r276sp9d-user-environment'
created 6 symlinks in user environment
A profile in nix is a general and very convenient concept for realizing rollbacks. Profiles are used to compose more components that are spread among multiple paths, under a new unified path. Not only, profiles are made up of multiple generations: they are versioned. Whenever you change a profile, a new generation is created.
Generations thus can be switched and rollback-ed atomatically.

Let's take a closer look at our profile:

$ ls -l ~/.nix-profile/
bin -> /nix/store/clnpynyac3hx3a6z5lsy893p7b4rwnyf-nix-1.7/bin
[...]
manifest.nix -> /nix/store/82dg18wz250vvcxjbclgyy5yg2kfz1gw-env-manifest.nix
[...]
share -> /nix/store/clnpynyac3hx3a6z5lsy893p7b4rwnyf-nix-1.7/share
That nix-1.7 derivation in the nix store is nix itself, with binaries and libraries. The installation basically reproduced the hierarchy of the nix-1.7 derivation in the profile by means of symbolic links.
The contents of this profile are special, because only one program has been installed in our profile, therefore e.g. the bin directory fully points to the only program being installed.

But that's only the contents of the latest generation of our profile. In fact, ~/.nix-profile itself is a symbolic link to /nix/var/nix/profiles/default .
In turn, that's a symlink to default-1-link in the same directory. Yes, that means it's the generation #1 of the default profile.
Finally that's a symlink to the nix store "user-environment" derivation that you saw printed during the installation process.

The manifest.nix will be meat for the next pill.

Meet nixpkgs expressions


More wild output from the installer:
downloading Nix expressions from `http://releases.nixos.org/nixpkgs/nixpkgs-14.10pre46060.a1a2851/nixexprs.tar.xz'...
unpacking channels...
created 2 symlinks in user environment
modifying /home/nix/.profile...

Nix expressions are used to describe packages and how to build them. Nixpkgs is the repository containing all these expressions: https://github.com/NixOS/nixpkgs .
The installer downloaded the package descriptions from commit a1a2851.

The second profile we meet is the channels profile. ~/.nix-defexpr/channels points to /nix/var/nix/profiles/per-user/nix/channels which points to channels-1-link which points to a nix store directory containing the downloaded nix expressions.

Channels are a set of packages and expressions available for download. Similar to debian stable and unstable, there's a stable and unstable channel. In this installation, we're tracking nixpkgs unstable.

Don't bother yet with nix expressions.

Finally, for your own convenience, it modified ~/.profile to automatically enter the nix environment. What ~/.nix-profile/etc/profile.d/nix.sh really does is simply adding ~/.nix-profile/bin to PATH and ~/.nix-defexpr/channels/nixpkgs to NIX_PATH. We'll discuss about NIX_PATH in another pill.
Read nix.sh, it's short.

FAQ: Can I change /nix to something else?


You can, but there's a strong reason to keep using /nix instead of a different directory. All the derivations depend on other derivations by absolute path. I remind you in pill 1 that bash pointed to a glibc under /nix/store.
You can see now by yourself, don't worry if you see multiple bash derivations:
$ ldd /nix/store/*bash*/bin/bash
[...]
Keeping the store in /nix means we can grab the binary cache from nixos.org (just like you grab packages from debian mirrors) otherwise:

  1. glibc would be installed under /foo/store
  2. thus bash needs to point to glibc under /foo/store
  3. the binary cache won't help, and we'd have to recompile all the stuff by ourselves
After all /nix is a cool place.

Conclusion


We've installed nix on our system, fully isolated and owned by the "nix" user as we're still diffident with this new system.
We learned some new concepts like profiles and channels. In particular, with profiles we're able to manage multiple generations of a composition of packages, while with channels we're able to download binaries from nixos.org .
The installation put everything under /nix, and some symlinks in the nix user home. That's because every user is able to install and use software in her own environment.

I hope I left nothing uncovered in a way that you think there's some kind of magic behind. It's all about putting components in the store and symlinking these components together.

Also I hope the commands in this pill were consistent with your fresh nix installation.

Next pill... 


...we will enter the nix environment and learn how to interact with the store.
Pill 3 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Wednesday, July 23, 2014

Nix pill 1: why you should give it a try

Introduction

Welcome to the first post of the Nix in pills series. Nix is a purely functional package manager and deployment system for POSIX. There's a lot of documentation that describes what Nix, NixOS and related projects are.
The purpose of this post is to convince you to give Nix a try. Installing NixOS is not required, but sometimes I may refer to NixOS as a real world example of Nix usage for building a whole operating system.

Rationale of pills

The Nix manualrelated papersarticles and wiki are excellent in explaining how Nix/NixOS works, how you can use it and the amount of cools thing being done with it, however sometimes you may feel some magic happening behind the scenes hard to grasp in the beginning.
This series aims to complement the missing explanations from the more formal documents.

Follows a description of Nix. Just as pills, I'll try to be as short as possible.

Not being pure functional

Most, if not all, widely used package managers (dpkg, rpm, ...) mutate the global state of the system. If a package foo-1.0 installs a program to /usr/bin/foo, you cannot install foo-1.1 as well, unless you change the installation paths or the binary name.
Changing the installation paths means breaking the assumptions about the file system hierarchy.
Changing the binary names means breaking possible users of that binary.

Debian for example partially solves the problem with the alternatives.

So in theory it's possible with the current systems to install multiple versions of the same package, in practice it's very painful.
Let's say you need an nginx service and also an nginx-openresty service. You have to create a new package that changes all the paths with e.g. an -openresty suffix.
Or you want to run two different instances of mysql, 5.2 and 5.5. Same thing applies plus you have to also make sure the two mysqlclient libraries do not collide.

This is not impossible but very inconvenient. If you want to install two whole stack of software like GNOME 3.10 and GNOME 3.12, you can imagine the amount of work.

From an administrator view point: containers to the rescue. The solution nowadays is to create a container per service, especially when different versions are needed. That somehow solves the problem, but at a different level and with other drawbacks. Needing orchestration tools, setting up a shared cache of packages, and new wild machines to monitor rather than simple services.

From a developer view point: virtualenv for python or jhbuild for gnome or whatelse. But then, how do you mix the two stacks? How do you avoid recompiling the same thing when it could be instead be shared? Also you need to setup your development tools to point to the different directories where libraries are installed. Not only, there's the risk that some of the software incorrectly uses system libraries.

And so on. Nix solves all this at the packaging level and solves it well. A single tool to rule them all.

Being pure functional

Nix does no assumption about the global state of the system. This has many advantages, but also some drawbacks of course. The core of a Nix based system is the Nix store, usually installed under /nix/store, and some tools to manipulate the store. In Nix there's the notion of derivation rather than package. The difference might be subtle at the beginning, so I will often mix both of the words, ignore that.

Derivations/packages are stored in the nix store as follows: /nix/store/hash-name, where the hash uniquely identifies the derivation (not true, it's a little more complex than this), and name is the name of the derivation.

Let's take the bash derivation as example: /nix/store/s4zia7hhqkin1di0f187b79sa2srhv6k-bash-4.2-p45/ . It is a directory that contains bin/bash.
You get it, there's no /bin/bash, there's only that self-contained build output in the store. Same goes for coreutils and everything else. To make their usage convenient from the shell, nix will put those paths in PATH.
What we have is basically a store of all packages and different versions of them, and things in the nix store are immutable.

There's no ldconfig cache either. So where does bash find libc?
$ ldd  `which bash`
libc.so.6 => /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/libc.so.6 (0x00007f0248cce000)

Turns out that when bash was built, it used that specific version of glibc and at runtime it will require exactly that glibc version.
Don't be doubtful about the version in the derivation name: it's only a name for us humans. You may end up having a different hash given the same derivation name.

What does it all mean? You could run mysql 5.2 with glibc-2.18, and mysql 5.5 with glibc-2.19. You could use your python module with python 2.7 compiled with gcc 4.6 and the same python module with python 3 compiled with gcc 4.8, all in the same system.
In other words, no dependency hell, not even a dependency resolution algorithm. Straight dependencies from derivations to other derivations.

From an administrator view point: need an old PHP version from lenny and want to upgrade to wheezy, not painful.

From a developer view point: want to develop webkit with llvm 3.4 and 3.3, not painful.

Mutable vs immutable

When upgrading a library, package managers replace it in-place. All new applications run afterwards with the new library without being recompiled. After all, they all refer dynamically to libc6.so.
While being immutable with Nix, upgrading a library like glibc means recompiling all applications, because the glibc path to the nix store is being hardcoded.

So how do we deal with security updates? In Nix we have some tricks (yet pure) to solve this problem, but that's another story.

Another problem is that unless software has in mind a pure functional model, or it can be adapted to, it's hard to compose applications at runtime.
Let's take for example firefox. You install flash, and you get it working because firefox looks in a global path for plugins.
In Nix, there's no such global path for plugins. Firefox therefore must know explicitly about the path to flash. We wrap the firefox binary to setup the necessary environment to make it find flash in the nix store. That will produce a new firefox derivation: be aware it takes a few seconds, but it made composition harder at runtime.

No upgrade/downgrade scripts for your data. It doesn't make sense with this approach, because there's no real derivation to be upgraded. With nix you switch to using another software with its own stack of dependencies, but there's no formal notion of upgrade or downgrade when doing so.
Migrating to the new data format is your own responsibility.

Conclusion

Nix lets you compose software at build time with maximum flexibility and with builds being as reproducible as possible. Not only, due to its nature deploying systems in the cloud is so easy, consistent and reliable that in the Nix world all existing self-containment and orchestration tools are deprecated by NixOps. Talking about this, no comparison with other tools is fair, nix rocks.
It however currently falls off when speaking about dynamic composition at runtime or replacing low level libraries due to massive rebuilds.

That may sound scary, however after running NixOS on both a server and a laptop desktop, I'm very satisfied so far. Some of the architectural problems just need some man power, other design problems are to be solved as a community yet.

Considering Nixpkgs (github link) is a completely new repository of all the existing software, with a completely fresh concept, and with few core developers but overall year-over-year increasing contributions, the current state is way more than acceptable and beyond the experimental stage. In other words, it's worth your investment.

Next pill...

...we will install Nix on top of your current system (I assume GNU/Linux, but we also have OSX users) and start inspecting the installed stuff.
Pill 2 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Wednesday, July 09, 2014

Debugging and fixing a wmctrl bug in NixOS

Without wmctrl I'm lost. I use it together with xbindkeys so that pressing a combination of keys will raise a window of the desktop (editor, shell, browser, chat, ...).

It turns out however that in NixOS wmctrl didn't work well. I had 3 windows opened, and wmctrl -l only showed one. Since gnome3 is still young in nixos, the first thought was that mutter did something wrong when returning the list of clients, due to some possible packaging mistake.

However, xprop -root correctly listed 3 window ids for the _NET_CLIENT_LIST atom, so mutter did its job.

Therefore it might be a problem of wmctrl. I'm always sure in these cases Debian has a patch, however I wanted to find out by myself.

Let's prepare an environment for hacking wmctrl:

$ nix-shell ~/nixpkgs -A wmctrl
$ unpackPhase
$ cd $sourceRoot
$ configurePhase
$ buildPhase
$ ./wmctrl -l

Ok it runs. Now let's dive into the code.
  1. Open main.c
  2. Search for _NET_CLIENT_LIST
  3. Note the for loop in list_windows function right below, it uses client_list_size / sizeof(Window) in the condition. That may be the culprit. 
  4. We make use of the most advanced debugging technique: before the for loop add a printf("%lu\n%lu\n", client_list_size, sizeof(Window));
Now let's rebuild and run wmctrl:

$ buildPhase
$ ./wmctrl -l

Bingo! client_list_size is 12 (3 windows x 4 byte) while sizeof(Window) is 8 instead of 4. Definitely a bug for 64-bit systems.
Of course Debian has the patch: http://ftp.de.debian.org/debian/pool/main/w/wmctrl/wmctrl_1.07-7.debian.tar.gz

  1. Exit the nix-shell and delete the wmctrl source directory.
  2. Copy debian/patches/01_64-bit-data.patch under ~/nixpkgs/pkgs/tools/X11/wmctrl/64-bit-data.patch.
  3. Edit ~/nixpkgs/pkgs/tools/X11/wmctrl/default and add patches = [ ./64-bit-data.patch ]; in the derivation.
  4. Finally install the patched wmctrl with nix-env ~/nixpkgs -iA wmctrl and see that wmctrl -l now works.
  5. Commit, push to nixpkgs master, profit.