Thursday, February 19, 2015

NixOS, Consul, Nginx and containers

This is a follow up post on https://medium.com/@dan.ellis/you-dont-need-1mm-for-a-distributed-system-70901d4741e1 . I think the post was well written, so I decided to write a variant using NixOS.

We'll be using declarative nixos containers, which do not use docker but systemd-nspawn. Also systemd is started as init system inside containers.

Please note that this configuration can be applied to any nixos machine, and also the containers configuration could be applied to real servers or other kinds of virtualization, e.g. via nixops. That is, the same syntax and configuration can be reused anywhere else within the nix world.

For example, you could create docker containers with nixos, and keep running the host with another distribution.

However for simplicity we'll use a NixOS system.

Architecture: the host runs nginx and a consul server, then spawns several containers with a python service and a consul client. On the host, consul-template will rewrite the nginx configuration when the health check status of container services change.

Please use a recent unstable release of nixos at the time of this writing (19 Feb 2015, at least commit aec96d4), as it contains the recently packaged consul-template.

Step 1: write the service


Let's write our python service in /root/work.py:
#!/usr/bin/env python

import random

from flask import Flask
app = Flask(__name__)

def find_prime(start):
    """
    Find a prime greater than `start`.
    """
    current = start
    while True:
        for p in xrange(2, current):
            if current % p == 0:
                break
        else:
            return current
        current += 1

@app.route("/")
def home():
    return str(find_prime(random.randint(2 ** 25, 2 ** 26)))

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=8080, debug=True)
The only difference with the original post is that we explicitly set the port to 8080 (who knows if a day flask changes the default port).

Step 2: write the nixos config


Write the following in /etc/nixos/prime.nix:
{ lib, pkgs, config, ... }:

let

  pypkgs = pkgs.python27Packages;

  # Create a self-contained package for our service
  work = pkgs.stdenv.mkDerivation {
    name = "work";
    unpackPhase = "true";
    buildInputs = [ pkgs.makeWrapper pypkgs.python pypkgs.flask ];
    installPhase = ''
      mkdir -p $out/bin
      cp ${/root/work.py} $out/bin/work.py
      chmod a+rx $out/bin/work.py
      wrapProgram $out/bin/work.py --prefix PYTHONPATH : $PYTHONPATH
    '';
  };

  # Function which takes a network and the final octet, and returns a container
  mkContainer = net: octet: {
    privateNetwork = true;
    hostAddress = "${net}.1";
    localAddress = "${net}.${octet}";
    autoStart = true;

    config = { config, pkgs, ... }:
      {
        users.mutableUsers = false;
        # Use this only for debugging, login the machine with machinectl
        users.extraUsers.root.password = "root";
        # Let consul run check scripts
        users.extraUsers.consul.shell = "/run/current-system/sw/bin/bash";

        environment.etc."consul/prime.json".text = builtins.toJSON {
          service = {
            name = "prime";
            tags = [ "nginx" ];
            port = 8080;
            check = {
              script = "${pkgs.curl}/bin/curl localhost:8080 >/dev/null 2>&1";
              interval = "30s";
            };
          };
        };

        systemd.services.prime = {
          wantedBy = [ "multi-user.target" ];

          serviceConfig = {
            ExecStart = "${work}/bin/work.py";
          };
        };

        services.consul = {
          enable = true;
          extraConfig = { 
            server = false;
            start_join = [ "${net}.1" ];
          };
          extraConfigFiles = [ "/etc/consul/prime.json" ];
        };

        networking.firewall = {
          allowedTCPPorts = [ 8080 8400 ];
          allowPing = true;
        };
      };
  };

  nginxTmpl = pkgs.writeText "prime.conf" ''
    upstream primes {
        {{range service "prime"}}
        server {{.Address}}:8080;{{end}}
    }
  '';

in
{
  containers.prime1 = mkContainer "10.50.0" "2";
  containers.prime2 = mkContainer "10.50.0" "3";
  containers.prime3 = mkContainer "10.50.0" "4";
  containers.prime4 = mkContainer "10.50.0" "5";

  services.consul = {
    enable = true;
    extraConfig = {
      bootstrap = true;
      server = true;
    };
  };

  services.nginx = {
    enable = true;
    httpConfig = ''
      include /etc/nginx/prime.conf;

      server {
        listen 80;

        location / {
          proxy_pass http://primes;
        }
      }
    '';
  };

  systemd.services.nginx = {
    preStart = ''
      mkdir -p /etc/nginx
      touch -a /etc/nginx/prime.conf
    '';

    serviceConfig = {
      Restart = "on-failure";
      RestartSec = "1s";
    };
  };

  # Start order: consul -> consul-template -> nginx
  systemd.services.consul-template = {
    wantedBy = [ "nginx.service" ];
    before = [ "nginx.service" ];
    wants = [ "consul.service" ];
    after = [ "consul.service" ];

    serviceConfig = {
      Restart = "on-failure";
      RestartSec = "1s";
      ExecStart = "${pkgs.consul-template}/bin/consul-template -template '${nginxTmpl}:/etc/nginx/prime.conf:systemctl kill -s SIGHUP nginx'";
    };
  };

  boot.kernel.sysctl."net.ipv4.ip_forward" = true;
}
Differences with the original post:
  • We only create 4 containers instead of 10. I was lazy here. If you are lazy too, you can still automatize the process with nix functions (for example map).
  • We define some ordering in how services start and how they restart with systemd.
  • For simplicity we include the prime.conf nginx config instead of rewriting the whole nginx config with consul-template.
  • We create a self-contained package for our python service, so that anywhere it runs the dependencies will be satisfied.
Finally import this config in your /etc/nixos/configuration.nix with imports = [ ./prime.nix ];.

Step 3: apply the configuration


Type nixos-rebuild switch and then curl http://localhost. You may have to wait some seconds before consul writes the nginx config. In the while, nginx may have failed to start. If it exceeded the StartTime conditions, you can systemctl start nginx manually.
Fixing this is about tweaking the systemd service values about the StartTime.

Each container consumes practically no disk space at base. Everything else is shared through the host nix store, except logs, consul state, ecc. of course.

Have fun!

Sunday, February 08, 2015

Developing in golang with Nix package manager

I've been using Go since several months. It's a pleasant language, even though it has its own drawbacks.

In our Nixpkgs repository we have support for several programming languages: perl, python, ruby, haskell, lua, ... We've merged a better support for Go.

What kind of support are we talking about? In Nix, you never install libraries. Instead, you define an environment in which to use a certain library compiled for a certain version of the language. The library will be available only within this environment.

Think of it like virtualenv for python, except for any language, and also being able to mix them.
On the other hand Nix requires the src url and the checksum of every dependency of your project. So before starting, make sure you are willing to write nix packages that are not currently present in nixpkgs.

Also you probably have to wait a couple of days before this PR will be available in the unstable channel, at the time of this writing (otherwise git clone https://github.com/NixOS/nixpkgs.git).

Using nix-shell -p

First a quick example using nix-shell -p for your own project:
$ nix-shell '<nixpkgs>' -p goPackages.go goPackages.net goPackages.osext
[nix-shell]$ echo $GOPATH
/nix/store/kw9dryid364ac038zmbzq72bnh3zsinz-go-1.4.1-go.net-3338d5f109e9/share/go:...
That's how nix mostly works, it's as simple as that. The GOPATH is set for every package that provides a directory share/go.

What is goPackages? Currently it's go14Packages, which is all the go packages we have, compiled with go 1.4. There's also go13Packages, you know some particular packages don't work with go 1.4 yet.

Writing a nix file

A more structured example by writing a default.nix file in your project:
with import <nixpkgs> {}; with goPackages;

buildGoPackage rec {
  name = "yourproject";
  buildInputs = [ net osext ];
  goPackagePath = "github.com/you/yourproject";
}
Then you can just run nix-shell in your project directory and have your dev environment ready to compile your code.
The goPackagePath is something needed by buildGoPackage, in case you are going to run nix-build. Ignore it for now.

Writing a dependency

But nixpkgs doesn't have listed all the possible go projects. What if you need to use a particular library?
Let's take for example github.com/kr/pty. Write something like this in a pty.nix file:
{ goPackages, fetchFromGitHub }:

goPackages.buildGoPackage rec {
  rev = "67e2db24c831afa6c64fc17b4a143390674365ef";
  name = "pty-${rev}";
  goPackagePath = "github.com/kr/pty";
  src = fetchFromGitHub {
    inherit rev;
    owner = "kr";
    repo = "pty";
    sha256 = "1l3z3wbb112ar9br44m8g838z0pq2gfxcp5s3ka0xvm1hjvanw2d";
  };
}
Then in your default.nix:
with import <nixpkgs> {}; with goPackages;

let
  pty = callPackage ./pty.nix {};
in
buildGoPackage rec {
  name = "yourproject";
  buildInputs = [ net osext pty ];
  goPackagePath = "github.com/you/yourproject";
}
Type nix-shell and now you will also have pty in your dev environment.
So as you can see, for each go package nix requires a name, the go path, where to fetch the sources, and the hash.

You may be wondering how do you get the sha256: a dirty trick is to write a wrong sha, then nix will tell you the correct sha.

Conclusion and references

Nix looks a little complex and boring due to writing a package for each dependency. On the other hand you get for free:
  • Exact build and runtime dependencies
  • Sharing build and runtime dependencies between multiple projects
  • Easily test newer or older versions of libraries, without messing with system-wide installations
  • Mix with other programming languages, using a similar approach
  • Packages using C libraries don't need to be compiled manually by you: define the nix package once, reuse everywhere
For installing nix, follow the manual. Make sure you read the entire document to learn the nix syntax.

For more examples on how to write dependencies, you can look at nixpkgs goPackages itself.

Drop by #nixos on irc.freenode.net for any doubts.


Monday, January 12, 2015

Nix pill 18: nix store paths

Welcome to the 18th Nix pill. In the previous 17th pill we have scratched the surface of the nixpkgs repository structure. It is a set of packages, and it's possible to override such packages so that all other packages will use the overrides.

Before reading existing derivations, I'd like to talk about store paths and how they are computed. In particular we are interested in fixed store paths that depend on an integrity hash (e.g. a sha256), which is usually applied to source tarballs.

The way store paths are computed is a little contrived, mostly due to historical reasons. Our reference will be the Nix source code.

Source paths


Let's start simple. You know nix allows relative paths to be used, such that the file or directory is stored in the nix store, that is ./myfile gets stored into /nix/store/....... We want to understand how is the store path generated for such a file:
$ echo mycontent > myfile
I remind you, the simplest derivation you can write has a name, a builder and the system:
$ nix-repl
nix-repl> derivation { system = "x86_64-linux"; builder = ./myfile; name = "foo"; }
«derivation /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv»
Now inspect the .drv to see where is ./myfile being stored:
$ pp-aterm -i /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv
Derive(
  [("out", "/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo", "", "")]
, []
, ["/nix/store/xv2iccirbrvklck36f1g7vldn5v58vck-myfile"]
, "x86_64-linux"
...
Great, how did nix decide to use xv2iccirbrvklck36f1g7vldn5v58vck ? Keep looking at the nix comments.

Note: doing nix-store --add myfile will store the file in the same store path.

Step 1, compute the hash of the file


The comments tell us to first compute the sha256 of the NAR serialization of the file. Can be done in two ways:
$ nix-hash --type sha256 myfile
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3
Or:
$ nix-store --dump myfile|sha256sum
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3  -
In general, Nix understands two contents: flat for regular files, or recursive for NAR serializations which can be anything.

Step 2, build the string description


Then nix uses a special string which includes the hash, the path type and the file name. We store this in another file:
$ echo -n "source:sha256:2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3:/nix/store:myfile" > myfile.str

Step 3, compute the final hash


Finally the comments tell us to compute the base-32 representation of the first 160 bits (truncation) of a sha256 of the above string:
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
xv2iccirbrvklck36f1g7vldn5v58vck

Output paths


Output paths are usually generated for derivations. We use the above example because it's simple. Even if we didn't build the derivation, nix knows the out path hs0yi5n5nw6micqhy8l1igkbhqdkzqa1. This is because the out path only depends on inputs.

It's computed in a similar way to source paths, except that the .drv is hashed and the type of derivation is output:out. In case of multiple outputs, we may have different output:<id>.

At the time nix computes the out path, the .drv contains an empty string for each out path. So what we do is getting our .drv and replacing the out path with an empty string:
$ cp -f /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv myout.drv
$ sed -i 's,/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo,,g' myout.drv
The myout.drv is the .drv state in which nix is when computing the out path for our derivation:
$ sha256sum myout.drv
1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5  myout.drv
$ echo -n "output:out:sha256:1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5:/nix/store:foo" > myout.str
$ nix-hash --type sha256 --truncate --base32 --flat myout.str
hs0yi5n5nw6micqhy8l1igkbhqdkzqa1
Then nix puts that out path in the .drv, and that's it.

In case the .drv has input derivations, that is it references other .drv, then such .drv paths are replaced by this same algorithm which returns an hash.

In other words, you get a final .drv where every other .drv path is replaced by its hash.

Fixed-output paths


Finally, the other most used kind of path is when we know beforehand an integrity hash of a file. This is usual for tarballs.

A derivation can take three special attributes: outputHashMode, outputHash and outputHashAlgo which are well documented in the nix manual.

The builder must create the out path and make sure its hash is the same as the one declared with outputHash.

Let's say our builder should create a file whose contents is mycontent:
$ echo mycontent > myfile
$ sha256sum myfile
f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb  myfile
nix-repl> derivation { name = "bar"; system = "x86_64-linux"; builder = "none"; outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb"; }
«derivation /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv»
Inspect the .drv and see that it also stored the fact that it's a fixed-output derivation with sha256 algorithm, compared to the previous examples:
$ pp-aterm -i /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv
Derive(
  [("out", "/nix/store/a00d5f71k0vp5a6klkls0mvr1f7sx6ch-bar", "sha256", "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb")]
...
It doesn't matter which input derivations are being used, the final out path must only depend on the declared hash.
What nix does is to create an intermediate string representation of the fixed-output content:
$ echo -n "fixed:out:sha256:f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb:" > mycontent.str
$ sha256sum mycontent.str 
423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639  myfile.str
Then proceed as it was a normal derivation output path:
$ echo -n "output:out:sha256:423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639:/nix/store:bar" > myfile.str
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
a00d5f71k0vp5a6klkls0mvr1f7sx6ch
Hence, the store path only depends on the declared fixed-output hash.

Conclusion


There are other types of store paths, but you get the idea. Nix first hashes the contents, then creates a string description, and the final store path is the hash of this string.

Also we've introduced some fundamentals, in particular the fact that Nix knows beforehand the out path of a derivation since it only depends on the inputs. We've also introduced fixed-output derivations which are especially used by the nixpkgs repository for downloading and verifying source tarballs.

Next pill


...we will introduce stdenv. In the previous pills we rolled our own mkDerivation convenience function for wrapping the builtin derivation, but the nixpkgs repository also has its own convenience functions for dealing with autotools projects and other build systems.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Monday, November 10, 2014

Nix pill 17: nixpkgs, overriding packages

Welcome to the 17th Nix pill. In the previous 16th pill we have started to dive into the nixpkgs repository. Nixpkgs is a function, and we've looked at some parameters like system and config.

Today we'll talk about a special attribute: config.packageOverrides. Overriding packages in a set with fixed point can be considered another design pattern in nixpkgs.

Overriding a package


I recall the override design pattern from the nix pill 14. Instad of calling a function with parameters directly, we make the call (function + parameters) overridable.
We put the override function in the returned attribute set of the original function call.

Take for example graphviz. It has an input parameter xlibs. If it's null, then graphviz will build without X support.
$ nix-repl
nix-repl> :l <nixpkgs>
Added 4360 variables.
nix-repl> :b graphviz.override { xlibs = null; }
This will build graphviz without X support, it's as simple as that.

However let's say a package P depends on graphviz, how do we make P depend on the new graphviz without X support?

In an imperative world...


...you could do something like this:
pkgs = import <nixpkgs> {};
pkgs.graphviz = pkgs.graphviz.override { xlibs = null; };
build(pkgs.P)
Given pkgs.P depends on pkgs.graphviz, it's easy to build P with the replaced graphviz. On a pure functional language it's not that easy because you can assign to variables only once.

Fixed point


The fixed point with lazy evaluation is crippling but about necessary in a language like Nix. It lets us achieve something similar to what we'd do imperatively.
Follows the definition of fixed point in nixpkgs:
# Take a function and evaluate it with its own returned value.
fix = f: let result = f result; in result;
It's a function that accepts a function f, calls f result on the result just returned by f result and returns it. In other words it's f(f(f(....
At first sight, it's an infinite loop. With lazy evaluation it isn't, because the call is done only when needed.
nix-repl> fix = f: let result = f result; in result
nix-repl> pkgs = self: { a = 3; b = 4; c = self.a+self.b; }
nix-repl> fix pkgs
{ a = 3; b = 4; c = 7; }
Without the rec keyword, we were able to refer to a and b of the same set.
  1. First pkgs gets called with an unevaluated thunk (pkgs(pkgs(...)
  2. To set the value of c then self.a and self.b are evaluated.
  3. The pkgs function gets called again to get the value of a and b.
The trick is that c is not needed to be evaluated in the inner call, thus it doesn't go in an infinite loop.

Won't go further with the explanation here. A good post about fixed point and Nix can be found here.

Overriding a set with fixed point


Given that self.a and self.b refer to the passed set and not to the literal set in the function, we're able to override both a and b and get a new value for c:
nix-repl> overrides = { a = 1; b = 2; }
nix-repl> let newpkgs = pkgs (newpkgs // overrides); in newpkgs
{ a = 3; b = 4; c = 3; }
nix-repl> let newpkgs = pkgs (newpkgs // overrides); in newpkgs // overrides
{ a = 1; b = 2; c = 3; }
In the first case we computed pkgs with the overrides, in the second case we also included the overriden attributes in the result.

Overriding nixpkgs packages


We've seen how to override attributes in a set such that they get recursively picked by dependant attributes. This approach can be used for derivations too, after all nixpkgs is a giant set of attributes that depend on each other.

To do this, nixpkgs offers config.packageOverrides. So nixpkgs returns a fixed point of the package set, and packageOverrides is used to inject the overrides.

Create a config.nix file like this somewhere:
{
  packageOverrides = pkgs: {
    graphviz = pkgs.graphviz.override { xlibs = null; };
  };
}
Now we can build e.g. asciidocFull and it will automatically use the overridden graphviz:
nix-repl> pkgs = import <nixpkgs> { config = import ./config.nix; }
nix-repl> :b pkgs.asciidocFull
Note how we pass the config with packageOverrides when importing nixpkgs. Then pkgs.asciidocFull is a derivation that has graphviz input (pkgs.asciidoc is the lighter version and doesn't use graphviz at all).

Since there's no version of asciidoc with graphviz without X support in the binary cache, Nix will recompile the needed stuff for you.

The ~/.nixpkgs/config.nix file


In the previous pill we already talked about this file. The above config.nix that we just wrote could be the content of ~/.nixpkgs/config.nix.

Instead of passing it explicitly whenever we import nixpkgs, it will be automatically imported by nixpkgs.

Conclusion


We've learned about a new design pattern: using fixed point for overriding packages in a package set.

Whereas in an imperative setting, like with other package managers, a library is installed replacing the old version and applications will use it, in Nix it's not that straight and simple. But it's more precise.

Nix applications will depend on specific versions of libraries, hence the reason why we have to recompile asciidoc to use the new graphviz library.

The newly built asciidoc will depend on the new graphviz, and old asciidoc will keep using the old graphviz undisturbed.

Next pill


...we will stop diving nixpkgs for a moment and talk about store paths. How does Nix compute the path in the store where to place the result of builds? How to add files to the store for which we have an integrity hash?

Pill 18 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Tuesday, November 04, 2014

Nix pill 16: nixpkgs, the parameters

Welcome to the 16th Nix pill. In the previous 15th pill we've realized how nix finds expressions with the angular brackets syntax, so that we finally know where is <nixpkgs> located on our system.

We can start diving into the nixpkgs repository, through all the various tools and design patterns. Please note that also nixpkgs has its own manual, underlying the difference between the general "nix" language and the "nixpkgs" repository.

The default.nix expression


We will not start inspecting packages at the beginning, rather the general structure of nixpkgs.
In our custom repository we created a default.nix which composed the expressions of the various packages.
Also nixpkgs has its own default.nix, which is the one being loaded when referring to <nixpkgs>. It does a simple thing: check whether the nix version is at least 1.7 (at the time of writing this blog post). Then import pkgs/top-level/all-packages.nix. From now on, we will refer to this set of packages as pkgs.

The all-packages.nix is then the file that composes all the packages. Note the pkgs/ subdirectory, while nixos is in the nixos/ subdirectory.

The all-packages.nix is a bit contrived. First of all, it's a function. It accepts a couple of interesting parameters:
  • system: defaults to the current system
  • config: defaults to null
  • others...
The system parameter, as per comment in the expression, it's the system for which the packages will be built. It allows for example to install i686 packages on amd64 machines.

The config parameter is a simple attribute set. Packages can read some of its values and change the behavior of some derivations.

The system parameter


You will find this parameter in many other .nix expressions (e.g. release expressions). The reason is that, given pkgs accepts a system parameter, then whenever you want to import pkgs you also want to pass through the value of system. E.g.:

myrelease.nix:
{ system ? builtins.currentSystem }:

let pkgs = import <nixpkgs> { inherit system; };
...
Why is it useful? With this parameter it's very easy to select a set of packages for a particular system. For example:
nix-build -A psmisc --argstr system i686-linux
This will build the psmisc derivation for i686-linux instead of x86_64-linux. This concept is very similar to multi-arch of Debian.

The setup for cross compiling is also in nixpkgs, however it's a little contrived to talk about it and I don't know much of it either.

The config parameter


I'm sure on the wiki or other manuals you've read about ~/.nixpkgs/config.nix and I'm sure you've wondered whether that's hardcoded in nix. It's not, it's in nixpkgs.

The all-packages.nix expression accepts the config parameter. If it's null, then it reads the NIXPKGS_CONFIG environment variable. If not specified, nixpkgs will peek $HOME/.nixpkgs/config.nix .  

After determining config.nix, it will be imported as nix expression, and that will be the value of config (in case it hasn't been passed as parameter to import <nixpkgs>).

The config is available in the resulting repository:
$ nix-repl
nix-repl> pkgs = import <nixpkgs> {}
nix-repl> pkgs.config
{ }
nix-repl> pkgs = import <nixpkgs> { config = { foo = "bar"; }; }
nix-repl> pkgs.config
{ foo = "bar"; }
What attributes go in config is a matter of convenience and conventions.

For example, config.allowUnfree is an attribute that forbids building packages that have an unfree license by default. The config.pulseaudio setting tells whether to build packages with pulseaudio support or not where applicable and when the derivation obeys to the setting.

About .nix functions


A .nix file contains a nix expression. Thus it can also be a function.
I remind you that nix-build expects the expression to return a derivation. Therefore it's natural to return straight a derivation from a .nix file.
However, it's also very natural for the .nix file to accept some parameters, in order to tweak the derivation being returned.

In this case, nix does a trick:
  • If the expression is a derivation, well build it.
  • If the expression is a function, call it and build the resulting derivation.
For example you can nix-build the .nix file below:
{ pkgs ? import <nixpkgs> {} }:

pkgs.psmisc
Nix is able to call the function because the pkgs parameter has a default value. This allows you to pass a different value for pkgs using the --arg option.

Does it work if you have a function returning a function that returns a derivation? No, Nix only calls the function it encounters once.

Conclusion


We've unleashed the <nixpkgs> repository. It's a function that accepts some parameters, and returns the set of all packages. Due to laziness, only the accessed derivations will be built.
You can use this repository to build your own packages as we've seen in the previous pill when creating our own repository.

Lately I'm a little busy with the NixOS 14.11 release and other stuff, and I'm also looking toward migrating from blogger to a more coder-oriented blogging platform. So sorry for the delayed  and shorter pills :)

Next pill


...we will talk about overriding packages in the nixpkgs repository. What if you want to change some options of a library and let all other packages pick the new library? One possibility is to use, like described above, the config parameter when applicable. The other possibility is to override derivations.

Pill 17 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Tuesday, September 16, 2014

Nix pill 15: nix search paths

Welcome to the 15th Nix pill. In the previous 14th pill we have introduced the "override" pattern, useful for writing variants of derivations by passing different inputs.

Assuming you followed the previous posts, I hope you are now ready to understand nixpkgs. But we have to find nixpkgs in our system first! So this is the step: introducing some options and environment variables used by nix tools.

The NIX_PATH


The NIX_PATH environment variable is very important. It's very similar to the PATH environment variable. The syntax is similar, several paths are separated by a colon ":". Nix will then search for something in those paths from left to right.

Who uses NIX_PATH? The nix expressions! Yes, NIX_PATH is not of much use by the nix tools themselves, rather it's used when writing nix expressions.

In the shell for example, when you execute the command "ping", it's being searched in the PATH directories. The first one found is the one being used.

In nix it's exactly the same, however the syntax is different. Instead of just typing "ping" you have to type <ping>. Yes, I know... you are already thinking of <nixpkgs>.
However don't stop reading here, let's keep going.

What's NIX_PATH good for? Nix expressions may refer to an "abstract" path such as <nixpkgs>, and it's possible to override it from the command line.

For ease we will use nix-instantiate --eval to do our tests. I remind you, nix-instantiate is used to evaluate nix expressions and generate the .drv files. Here we are not interested in building derivations, so evaluation is enough. It can be used for one-shot expressions.

Fake it a little


It's useless from a nix view point, but I think it's useful for your own understanding. Let's use PATH itself as NIX_PATH, and try to locate ping (or another binary if you don't have it).
$ nix-instantiate --eval -E '<ping>'
error: file `ping' was not found in the Nix search path (add it using $NIX_PATH or -I)
$ NIX_PATH=$PATH nix-instantiate --eval -E '<ping>'
/bin/ping
$ nix-instantiate -I /bin --eval -E '<ping>'
/bin/ping
Great. At first attempt nix obviously said  could not be found anywhere in the search path. Note that the -I option accepts a single directory. Paths added with -I take precedence over NIX_PATH.

The NIX_PATH also accepts a different yet very handy syntax: "somename=somepath". That is, instead of searching inside a directory for a name, we specify exactly the value of that name.
$ NIX_PATH="ping=/bin/ping" nix-instantiate --eval -E '<ping>'
/bin/ping
$ NIX_PATH="ping=/bin/foo" nix-instantiate --eval -E '<ping>'
error: file `ping' was not found in the Nix search path (add it using $NIX_PATH or -I)
Note in the second case how Nix checks whether the path exists or not.

The path to repository


You are out of curiosity, right?
$ nix-instantiate --eval -E '<nixpkgs>'
/home/nix/.nix-defexpr/channels/nixpkgs
$ echo $NIX_PATH
nixpkgs=/home/nix/.nix-defexpr/channels/nixpkgs
You may have a different path, depending on how you added channels etc.. Anyway that's the whole point. The <nixpkgs> stranger that we used in our nix expressions, is referring to a path in the filesystem specified by NIX_PATH.
You can list that directory and realize it's simply a checkout of the nixpkgs repository at a specific commit (hint: .version-suffix).
The NIX_PATH variable is exported by nix.sh, and that's the reason why I always asked you to source nix.sh at the beginning of my posts.

You may wonder: then I can also specify a different nixpkgs path to, e.g., a git checkout of nixpkgs? Yes, you can and I encourage doing that. We'll talk about this in the next pill.

Let's define a path for our repository, then! Let's say all the default.nix, graphviz.nix etc. are under /home/nix/mypkgs:
$ export NIX_PATH=mypkgs=/home/nix/mypkgs:$NIX_PATH
$ nix-instantiate --eval '<mypkgs>'
{ graphviz = <code>; graphvizCore = <code>; hello = <code>; mkDerivation = <code>; }
As expected we got the set of our packages (well except the mkDerivation utility), that's our repository.

Until now we used nix-build directly in the directory of default.nix. However nix-build generally needs a .nix to be specified to the command line:
$ nix-build /home/nix/mypkgs -A graphviz
/nix/store/dhps2c1fnsfxssdri3yxbdiqmnhzg6yr-graphviz
$ nix-build '<mypkgs>' -A graphviz
/nix/store/dhps2c1fnsfxssdri3yxbdiqmnhzg6yr-graphviz
Yes, nix-build also accepts paths with angular brackets. We first evaluate the whole repository (default.nix) and then peek the graphviz attribute.

A big word about nix-env


The nix-env command is a little different than nix-instantiate and nix-build. Whereas nix-instantiate and nix-build require a starting nix expression, nix-env does not.

You may be crippled by this concept at the beginning, you may think nix-env uses NIX_PATH to find the nixpkgs repository. But that's not it.

The nix-env command uses ~/.nix-defexpr, which is also part of NIX_PATH by default, but that's only a coincidence. If you empty NIX_PATH, nix-env will still be able to find derivations because of ~/.nix-defexpr.

So if you run nix-env -i graphviz inside your repository, it will install the nixpkgs one. Same if you set NIX_PATH to point to your repository.

In order to specify an alternative to ~/.nix-defexpr it's possible to use the -f option:
$ nix-env -f '<mypkgs>' -i graphviz
warning: there are multiple derivations named `graphviz'; using the first one
replacing old `graphviz'
installing `graphviz'
Oh why did it say there's another derivation named graphviz? Because both graphviz and graphvizCore attributes in our repository have the name "graphviz" for the derivation:
$ nix-env -f '<mypkgs>' -qaP
graphviz      graphviz
graphvizCore  graphviz
hello         hello
By default nix-env parses all derivations and use the derivation names to interpret the command line. So in this case "graphviz" matched two derivations. Alternatively, like for nix-build, one can use -A to specify an attribute name instead of a derivation name:
$ nix-env -f '<mypkgs>' -i -A graphviz
replacing old `graphviz'
installing `graphviz'
This form, other than being more precise, it's also faster because nix-env does not have to parse all the derivations.

For completeness: you may install graphvizCore with -A, since without the -A switch it's ambiguous.

In summary, it may happen when playing with nix that nix-env peeks a different derivation than nix-build. In such case you probably specified NIX_PATH, but nix-env is instead looking into ~/.nix-defexpr.

Why is nix-env having this different behavior? I don't know specifically by myself either, but the answers could be:
  • nix-env tries to be generic, thus it does not look for "nixpkgs" in NIX_PATH, rather it looks in ~/.nix-defexpr.
  • nix-env is able to merge multiple trees in ~/.nix-defexpr by looking at all the possible derivations
It may also happen to you that you cannot match a derivation name when installing, because of the derivation name vs -A switch described above. Maybe nix-env wanted to be more friendly in this case for default user setups.

It may or may not make sense for you, or it's like that for historical reasons, but that's how it works currently, unless somebody comes up with a better idea.

Conclusion


The NIX_PATH variable is the search path used by nix when using the angular brackets syntax. It's possible to refer to "abstract" paths inside nix expressions and define the "concrete" path by means of NIX_PATH, or the usual -I flag in nix tools.

We've also explained some of the uncommon nix-env behaviors for newcomers. The nix-env tool does not use NIX_PATH to search for packages, but rather for ~/.nix-defexpr. Beware of that!

In general do not abuse NIX_PATH, when possible use relative paths when writing your own nix expressions. Of course, in the case of <nixpkgs> in our repository, that's a perfectly fine usage of NIX_PATH. Instead, inside our repository itself, refer to expressions with relative paths like ./hello.nix.


Next pill


...we will finally dive into nixpkgs. Most of the techniques we have developed in this series are already in nixpkgs, like mkDerivation, callPackage, override, etc., but of course better. With time, those base utilities get enhanced by the community with more features in order to handle more and more use cases and in a more general way.

Pill 16 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Wednesday, September 10, 2014

Nix pill 14: the override design pattern

Welcome to the 14th Nix pill. In the previous 13th pill we have introduced the callPackage pattern, used to simplify the composition of software in a repository.

The next design pattern is less necessary but useful in many cases and it's a good exercise to learn more about Nix.

About composability


Functional languages are known for being able to compose functions. In particular, you gain a lot from functions that are able to manipulate the original value into a new value having the same structure. So that in the end we're able to call multiple functions to have the desired modifications.

In Nix we mostly talk about functions that accept inputs in order to return derivations. In our world we want nice utility functions that are able to manipulate those structures. These utilities add some useful properties to the original value, and we must be able to apply more utilities on top of it.

For example let's say we have an initial derivation drv and we want it to be a drv with debugging information and also to apply some custom patches:
debugVersion (applyPatches [ ./patch1.patch ./patch2.patch ] drv)
The final result will be still the original derivation plus some changes. That's both interesting and very different from other packaging approaches, which is a consequence of using a functional language to describe packages.

Designing such utilities is not trivial in a functional language that is not statically typed, because understanding what can or cannot be composed is difficult. But we try to do the best.

The override pattern


In the pill 12 we introduced the inputs design pattern. We do not return a derivation picking dependencies directly from the repository, rather we declare the inputs and let the callers pass the necessary arguments.

In our repository we have a set of attributes that import the expressions of the packages and pass these arguments, getting back a derivation. Let's take for example the graphviz attribute:
graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpeg bzip2; };
If we wanted to produce a derivation of graphviz with a customized gd version, we would have to repeat most of the above plus specifying an alternative gd:
mygraphviz = import ./graphviz.nix {
  inherit mkDerivation fontconfig libjpeg bzip2;
  gd = customgd;
};
That's hard to maintain. Using callPackage it would be easier:
mygraphviz = callPackage ./graphviz.nix { gd = customgd; };
But we may still be diverging from the original graphviz in the repository.

We would like to avoid specifying the nix expression again, instead reuse the original graphviz attribute in the repository and add our overrides like this:
mygraphviz = graphviz.override { gd = customgd; };
The difference is obvious, as well as the advantages of this approach.

Note: that .override is not a "method" in the OO sense as you may think. Nix is a functional language. That .override is simply an attribute of a set.

The override implementation


I remind you, the graphviz attribute in the repository is the derivation returned by the function imported from graphviz.nix. We would like to add a further attribute named "override" to the returned set.

Let's start simple by first creating a function "makeOverridable" that takes a function and a set of original arguments to be passed to the function.

Contract: the wrapped function must return a set.

Let's write a lib.nix:
{
  makeOverridable = f: origArgs:
    let
      origRes = f origArgs;
    in
      origRes // { override = newArgs: f (origArgs // newArgs); };
}
So makeOverridable takes a function and a set of original arguments. It returns the original returned set, plus a new override attribute.
This override attribute is a function taking a set of new arguments, and returns the result of the original function called with the original arguments unified with the new arguments. What a mess.

Let's try it with nix-repl:
$ nix-repl
nix-repl> :l lib.nix
Added 1 variables.
nix-repl> f = { a, b }: { result = a+b; }
nix-repl> f { a = 3; b = 5; }
{ result = 8; }
nix-repl> res = makeOverridable f { a = 3; b = 5; }
nix-repl> res
{ override = «lambda»; result = 8; }
nix-repl> res.override { a = 10; }
{ result = 15; }
Note that the function f does not return the plain sum but a set, because of the contract. You didn't forget already, did you? :-)
The variable res is the result of the function call without any override. It's easy to see in the definition of makeOverridable. In addition you can see the new override attribute being a function.

Calling that .override with a set will invoke the original function with the overrides, as expected.
But: we can't override again! Because the returned set with result 15 does not have an override attribute!
That's bad, it breaks further compositions.

The solution is simple, the .override function should make the result overridable again:
rec {
  makeOverridable = f: origArgs:
    let
      origRes = f origArgs;
    in
      origRes // { override = newArgs: makeOverridable f (origArgs // newArgs); };
}
Please note the rec keyword. It's necessary so that we can refer to makeOverridable from makeOverridable itself.

Now let's try overriding twice:
nix-repl> :l lib.nix
Added 1 variables.
nix-repl> f = { a, b }: { result = a+b; }
nix-repl> res = makeOverridable f { a = 3; b = 5; }
nix-repl> res2 = res.override { a = 10; }
nix-repl> res2
{ override = «lambda»; result = 15; }
nix-repl> res2.override { b = 20; }
{ override = «lambda»; result = 30; }

Success! The result is 30, as expected because a is overridden to 10 in the first override, and b to 20.

Now it would be nice if callPackage made our derivations overridable. That was the goal of this pill after all. This is an exercise for the reader.

Conclusion


The "override" pattern simplifies the way we customize packages starting from an existing set of packages. This opens a world of possibilities about using a central repository like nixpkgs, and defining overrides on our local machine without even modifying the original package.

Dream of a custom isolated nix-shell environment for testing graphviz with a custom gd:
debugVersion (graphviz.override { gd = customgd; })
Once a new version of the overridden package comes out in the repository, the customized package will make use of it automatically.

The key in Nix is to find powerful yet simple abstractions in order to let the user customize his environment with highest consistency and lowest maintenance time, by using predefined composable components.

Next pill


...we will talk about Nix search paths. By search path I mean a place in the file system where Nix looks for expressions. You may have wondered, where does that holy <nixpkgs> come from?

Pill 15 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.