Thursday, February 19, 2015

NixOS, Consul, Nginx and containers

This is a follow up post on https://medium.com/@dan.ellis/you-dont-need-1mm-for-a-distributed-system-70901d4741e1 . I think the post was well written, so I decided to write a variant using NixOS.

We'll be using declarative nixos containers, which do not use docker but systemd-nspawn. Also systemd is started as init system inside containers.

Please note that this configuration can be applied to any nixos machine, and also the containers configuration could be applied to real servers or other kinds of virtualization, e.g. via nixops. That is, the same syntax and configuration can be reused anywhere else within the nix world.

For example, you could create docker containers with nixos, and keep running the host with another distribution.

However for simplicity we'll use a NixOS system.

Architecture: the host runs nginx and a consul server, then spawns several containers with a python service and a consul client. On the host, consul-template will rewrite the nginx configuration when the health check status of container services change.

Please use a recent unstable release of nixos at the time of this writing (19 Feb 2015, at least commit aec96d4), as it contains the recently packaged consul-template.

Step 1: write the service


Let's write our python service in /root/work.py:
#!/usr/bin/env python

import random

from flask import Flask
app = Flask(__name__)

def find_prime(start):
    """
    Find a prime greater than `start`.
    """
    current = start
    while True:
        for p in xrange(2, current):
            if current % p == 0:
                break
        else:
            return current
        current += 1

@app.route("/")
def home():
    return str(find_prime(random.randint(2 ** 25, 2 ** 26)))

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=8080, debug=True)
The only difference with the original post is that we explicitly set the port to 8080 (who knows if a day flask changes the default port).

Step 2: write the nixos config


Write the following in /etc/nixos/prime.nix:
{ lib, pkgs, config, ... }:

let

  pypkgs = pkgs.python27Packages;

  # Create a self-contained package for our service
  work = pkgs.stdenv.mkDerivation {
    name = "work";
    unpackPhase = "true";
    buildInputs = [ pkgs.makeWrapper pypkgs.python pypkgs.flask ];
    installPhase = ''
      mkdir -p $out/bin
      cp ${/root/work.py} $out/bin/work.py
      chmod a+rx $out/bin/work.py
      wrapProgram $out/bin/work.py --prefix PYTHONPATH : $PYTHONPATH
    '';
  };

  # Function which takes a network and the final octet, and returns a container
  mkContainer = net: octet: {
    privateNetwork = true;
    hostAddress = "${net}.1";
    localAddress = "${net}.${octet}";
    autoStart = true;

    config = { config, pkgs, ... }:
      {
        users.mutableUsers = false;
        # Use this only for debugging, login the machine with machinectl
        users.extraUsers.root.password = "root";
        # Let consul run check scripts
        users.extraUsers.consul.shell = "/run/current-system/sw/bin/bash";

        environment.etc."consul/prime.json".text = builtins.toJSON {
          service = {
            name = "prime";
            tags = [ "nginx" ];
            port = 8080;
            check = {
              script = "${pkgs.curl}/bin/curl localhost:8080 >/dev/null 2>&1";
              interval = "30s";
            };
          };
        };

        systemd.services.prime = {
          wantedBy = [ "multi-user.target" ];

          serviceConfig = {
            ExecStart = "${work}/bin/work.py";
          };
        };

        services.consul = {
          enable = true;
          extraConfig = { 
            server = false;
            start_join = [ "${net}.1" ];
          };
          extraConfigFiles = [ "/etc/consul/prime.json" ];
        };

        networking.firewall = {
          allowedTCPPorts = [ 8080 8400 ];
          allowPing = true;
        };
      };
  };

  nginxTmpl = pkgs.writeText "prime.conf" ''
    upstream primes {
        {{range service "prime"}}
        server {{.Address}}:8080;{{end}}
    }
  '';

in
{
  containers.prime1 = mkContainer "10.50.0" "2";
  containers.prime2 = mkContainer "10.50.0" "3";
  containers.prime3 = mkContainer "10.50.0" "4";
  containers.prime4 = mkContainer "10.50.0" "5";

  services.consul = {
    enable = true;
    extraConfig = {
      bootstrap = true;
      server = true;
    };
  };

  services.nginx = {
    enable = true;
    httpConfig = ''
      include /etc/nginx/prime.conf;

      server {
        listen 80;

        location / {
          proxy_pass http://primes;
        }
      }
    '';
  };

  systemd.services.nginx = {
    preStart = ''
      mkdir -p /etc/nginx
      touch -a /etc/nginx/prime.conf
    '';

    serviceConfig = {
      Restart = "on-failure";
      RestartSec = "1s";
    };
  };

  # Start order: consul -> consul-template -> nginx
  systemd.services.consul-template = {
    wantedBy = [ "nginx.service" ];
    before = [ "nginx.service" ];
    wants = [ "consul.service" ];
    after = [ "consul.service" ];

    serviceConfig = {
      Restart = "on-failure";
      RestartSec = "1s";
      ExecStart = "${pkgs.consul-template}/bin/consul-template -template '${nginxTmpl}:/etc/nginx/prime.conf:systemctl kill -s SIGHUP nginx'";
    };
  };

  boot.kernel.sysctl."net.ipv4.ip_forward" = true;
}
Differences with the original post:
  • We only create 4 containers instead of 10. I was lazy here. If you are lazy too, you can still automatize the process with nix functions (for example map).
  • We define some ordering in how services start and how they restart with systemd.
  • For simplicity we include the prime.conf nginx config instead of rewriting the whole nginx config with consul-template.
  • We create a self-contained package for our python service, so that anywhere it runs the dependencies will be satisfied.
Finally import this config in your /etc/nixos/configuration.nix with imports = [ ./prime.nix ];.

Step 3: apply the configuration


Type nixos-rebuild switch and then curl http://localhost. You may have to wait some seconds before consul writes the nginx config. In the while, nginx may have failed to start. If it exceeded the StartTime conditions, you can systemctl start nginx manually.
Fixing this is about tweaking the systemd service values about the StartTime.

Each container consumes practically no disk space at base. Everything else is shared through the host nix store, except logs, consul state, ecc. of course.

Have fun!

Sunday, February 08, 2015

Developing in golang with Nix package manager

I've been using Go since several months. It's a pleasant language, even though it has its own drawbacks.

In our Nixpkgs repository we have support for several programming languages: perl, python, ruby, haskell, lua, ... We've merged a better support for Go.

What kind of support are we talking about? In Nix, you never install libraries. Instead, you define an environment in which to use a certain library compiled for a certain version of the language. The library will be available only within this environment.

Think of it like virtualenv for python, except for any language, and also being able to mix them.
On the other hand Nix requires the src url and the checksum of every dependency of your project. So before starting, make sure you are willing to write nix packages that are not currently present in nixpkgs.

Also you probably have to wait a couple of days before this PR will be available in the unstable channel, at the time of this writing (otherwise git clone https://github.com/NixOS/nixpkgs.git).

Using nix-shell -p

First a quick example using nix-shell -p for your own project:
$ nix-shell '<nixpkgs>' -p goPackages.go goPackages.net goPackages.osext
[nix-shell]$ echo $GOPATH
/nix/store/kw9dryid364ac038zmbzq72bnh3zsinz-go-1.4.1-go.net-3338d5f109e9/share/go:...
That's how nix mostly works, it's as simple as that. The GOPATH is set for every package that provides a directory share/go.

What is goPackages? Currently it's go14Packages, which is all the go packages we have, compiled with go 1.4. There's also go13Packages, you know some particular packages don't work with go 1.4 yet.

Writing a nix file

A more structured example by writing a default.nix file in your project:
with import <nixpkgs> {}; with goPackages;

buildGoPackage rec {
  name = "yourproject";
  buildInputs = [ net osext ];
  goPackagePath = "github.com/you/yourproject";
}
Then you can just run nix-shell in your project directory and have your dev environment ready to compile your code.
The goPackagePath is something needed by buildGoPackage, in case you are going to run nix-build. Ignore it for now.

Writing a dependency

But nixpkgs doesn't have listed all the possible go projects. What if you need to use a particular library?
Let's take for example github.com/kr/pty. Write something like this in a pty.nix file:
{ goPackages, fetchFromGitHub }:

goPackages.buildGoPackage rec {
  rev = "67e2db24c831afa6c64fc17b4a143390674365ef";
  name = "pty-${rev}";
  goPackagePath = "github.com/kr/pty";
  src = fetchFromGitHub {
    inherit rev;
    owner = "kr";
    repo = "pty";
    sha256 = "1l3z3wbb112ar9br44m8g838z0pq2gfxcp5s3ka0xvm1hjvanw2d";
  };
}
Then in your default.nix:
with import <nixpkgs> {}; with goPackages;

let
  pty = callPackage ./pty.nix {};
in
buildGoPackage rec {
  name = "yourproject";
  buildInputs = [ net osext pty ];
  goPackagePath = "github.com/you/yourproject";
}
Type nix-shell and now you will also have pty in your dev environment.
So as you can see, for each go package nix requires a name, the go path, where to fetch the sources, and the hash.

You may be wondering how do you get the sha256: a dirty trick is to write a wrong sha, then nix will tell you the correct sha.

Conclusion and references

Nix looks a little complex and boring due to writing a package for each dependency. On the other hand you get for free:
  • Exact build and runtime dependencies
  • Sharing build and runtime dependencies between multiple projects
  • Easily test newer or older versions of libraries, without messing with system-wide installations
  • Mix with other programming languages, using a similar approach
  • Packages using C libraries don't need to be compiled manually by you: define the nix package once, reuse everywhere
For installing nix, follow the manual. Make sure you read the entire document to learn the nix syntax.

For more examples on how to write dependencies, you can look at nixpkgs goPackages itself.

Drop by #nixos on irc.freenode.net for any doubts.