Thoughts about computer technologies
Sunday, April 21, 2024
Tuesday, August 14, 2018
OpenVPN configuration for server and multiple clients
This is a simple post showing a basic configuration for setting up OpenVPN server accepting multiple clients with TLS.
First of all generate self-signed server and client private key and certificates, and dh params. Make sure to write the Organization Name AND Common Name (CN) when asked, otherwise openvpn will fail to verify the certificates.
If you plan to use the server as gateway like configured below, don't forget to enable IP forwarding and masquerading:
Client configuration:
Have fun, tune as needed.
First of all generate self-signed server and client private key and certificates, and dh params. Make sure to write the Organization Name AND Common Name (CN) when asked, otherwise openvpn will fail to verify the certificates.
openssl req -newkey rsa:2048 -nodes -keyout serverkey.pem -x509 -days 365000 -out servercert.pem
openssl req -newkey rsa:2048 -nodes -keyout clientkey.pem -out client.csr
openssl x509 -req -days 365000 -in client.csr -CA servercert.pem -CAkey serverkey.pem -set_serial 01 -out clientcert.pem
openssl dhparam -outform PEM -out dh.pem 1024
Server configuration:
dev tun
mode server
tls-server
server 10.8.0.0 255.255.255.0
ca servercert.pem
cert servercert.pem
key serverkey.pem
dh dh.pem
duplicate-cn
topology subnet
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
If you plan to use the server as gateway like configured below, don't forget to enable IP forwarding and masquerading:
sysctl net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -j MASQUERADE
Client configuration:
client
redirect-gateway
tls-client
key clientkey.pem
cert clientcert.pem
ca servercert.pem
keepalive 10 60
dev tun
Have fun, tune as needed.
Monday, January 04, 2016
TypeScript and NodeJS, I'm sold
TypeScript is a typed superset of JavaScript that compiles to plain JavaScript, the way you expect it to be.
I’ve heard of it a long time ago, but recently with TypeScript 1.7 it got
async
functions, which means you can await
asynchronous function calls, similarly to C#, Vala, Go and other languages with syntax support for concurrency. That makes coroutines a pleasant experience compared to plain JavaScript. That’s also the main reason why I didn’t choose Dart.
I’m writing a NodeJS application so I decided to give it a go. Here’s my personal tour of TypeScript and why I’m sold to using it.
Does it complain when using undeclared variables?
console.log(foo);
Cannot find name 'foo'.
Sold!
Does it infer types?
var foo = 123;
foo = "bar";
Type 'string' is not assignable to type 'number'.
Sold!
Does it support async arrow functions?
async function foo() {
}
var bar = async () => { await (foo); };
Sold!
Does it support sum types?
var foo: number | string;
foo = 123;
foo = "bar";
Sold!
Does it play nice with CommonJS/AMD imports and external libraries?
Yes, it does very well. Sold!
Is it easy to migrate from and to JavaScript?
Yes. TypeScript makes use of latest ECMAScript features in its syntax when possible, so that JavaScript -> TypeScript is as painless as possible. Sold!
Also to go back from TypeScript to JavaScript, either use the generated code, or remove all the type annotations in the code by yourself.
Does it have non-nullable variables?
No. This is mostly due to the JavaScript nature though. But I’m sure the TypeScript community will come up with a nice solution throughout this topic.
I’m going to use TypeScript wherever I can in this new year instead of plain JavaScript. In particular, I’m rewriting my latest NodeJS application in TypeScript right now.
I hope it will be a great year for this language. The project is exceptionally active, and I hope to contribute back to it.
Etichette:
development,
javascript,
nodejs,
typescript
Monday, August 24, 2015
Nix pill 19: fundamentals of stdenv
Welcome to the 19th Nix pill. In the previous 18th pill we did dive into the algorithm used by Nix to compute the store paths, and also introduced fixed-output store paths.
This time we will instead look into nixpkgs, in particular one of its core derivation: stdenv .
The stdenv is not a special derivation, but it's very important for the nixpkgs repository. It serves as base for packaging software. It is used to pull in dependencies such as the GCC toolchain, GNU make, core utilities, patch and diff utilities, and so on. Basic tools needed to compile a huge pile of software currently present in nixpkgs.
First of all stdenv is a derivation. And it's a very simple one:
How can this simple derivation pull in all the toolchain and basic tools needed to compile packages? Let's look at the runtime dependencies:
Remember our generic
The stdenv setup file is exactly that. It sets up several environment variables like PATH and creates some helper bash functions to build a package. I invite you to read it, it's only 860 lines at the time of this writing.
The hardcoded toolchain and utilities are used to initially fill up the environment variables so that it's more pleasant to run common commands, similarly but not equal like we did with our builder with
The build with stdenv works in phases. Phases are like
What
Every phase has hooks to run commands before and after the phase has been executed. Phases can be overwritten, reordered, whatever, it's just bash code.
How to use this file? Like our old builder. To test it, we enter a fake empty derivation, source the stdenv setup, unpack the hello sources and build it:
So we ran the
Very little digression for completeness. The stdenv derivation is just that setup file. That setup file is just this setup.sh in nixpkgs plus some lines on top of it, put by this simple builder:
Until now we worked with plain bash scripts. What about the Nix side? The nixpkgs repository offers a useful function, like we did with our old builder. It is a wrapper around the raw
Note how
Let's write a
It builds, and runs fine:
Let's take a look at the builder used by
You can open default-builder.sh and see what it does:
To get a clear understanding of the environment variables, look at the .drv of the hello derivation:
Last bit, the
The stdenv is the core of the nixpkgs repository. All packages use the
The overall process is simple:
Really, take your time to read that file. Don't forget that juicy docs are also available in the nixpkgs manual.
...we will talk about how to add dependencies to our packages,
This time we will instead look into nixpkgs, in particular one of its core derivation: stdenv .
The stdenv is not a special derivation, but it's very important for the nixpkgs repository. It serves as base for packaging software. It is used to pull in dependencies such as the GCC toolchain, GNU make, core utilities, patch and diff utilities, and so on. Basic tools needed to compile a huge pile of software currently present in nixpkgs.
What is stdenv
First of all stdenv is a derivation. And it's a very simple one:
$ nix-build '<nixpkgs>' -A stdenv
/nix/store/k4jklkcag4zq4xkqhkpy156mgfm34ipn-stdenv
$ ls -R result/
result/:
nix-support/ setup
result/nix-support:
propagated-user-env-packages
It has just two files: /setup
and /nix-support/propagated-user-env-packages
. Don't care about the latter, it's even empty. The important file is /setup
.How can this simple derivation pull in all the toolchain and basic tools needed to compile packages? Let's look at the runtime dependencies:
$ nix-store -q --references result
/nix/store/3a45nb37s0ndljp68228snsqr3qsyp96-bzip2-1.0.6
/nix/store/a457ywa1haa0sgr9g7a1pgldrg3s798d-coreutils-8.24
/nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39
/nix/store/47sfpm2qclpqvrzijizimk4md1739b1b-gcc-wrapper-4.9.3
...
How can it be? The package must be referring to those package somehow. In fact, they are hardcoded in the /setup
file:$ head result/setup
export SHELL=/nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39/bin/bash
initialPath="/nix/store/a457ywa1haa0sgr9g7a1pgldrg3s798d-coreutils-8.24 ..."
defaultNativeBuildInputs="/nix/store/sgwq15xg00xnm435gjicspm048rqg9y6-patchelf-0.8 ..."
The setup file
Remember our generic
builder.sh
in Pill 8? It sets up a basic PATH, unpacks the source and runs the usual autotools commands for us.The stdenv setup file is exactly that. It sets up several environment variables like PATH and creates some helper bash functions to build a package. I invite you to read it, it's only 860 lines at the time of this writing.
The hardcoded toolchain and utilities are used to initially fill up the environment variables so that it's more pleasant to run common commands, similarly but not equal like we did with our builder with
baseInputs
and buildInputs
.The build with stdenv works in phases. Phases are like
unpackPhase
, configurePhase
, buildPhase
, checkPhase
, installPhase
, fixupPhase
. You can see the default list in the genericBuild
function.What
genericBuild
does is just run these phases. Default phases are just bash functions, you can easily read them.Every phase has hooks to run commands before and after the phase has been executed. Phases can be overwritten, reordered, whatever, it's just bash code.
How to use this file? Like our old builder. To test it, we enter a fake empty derivation, source the stdenv setup, unpack the hello sources and build it:
$ nix-shell -E 'derivation { name = "fake"; builder = "fake"; system = "x86_64-linux"; }'
nix-shell$ unset PATH
nix-shell$ source /nix/store/k4jklkcag4zq4xkqhkpy156mgfm34ipn-stdenv/setup
nix-shell$ tar -xf hello-2.9.tar.gz
nix-shell$ cd hello-2.9
nix-shell$ configurePhase
...
nix-shell$ buildPhase
...
I unset PATH
to further show that the stdenv is enough self-contained to build autotools packages that have no other dependencies.So we ran the
configurePhase
function and buildPhase
function and they worked. These bash functions should be self-explanatory, you can read the code in the setup file.How is the setup file built
Very little digression for completeness. The stdenv derivation is just that setup file. That setup file is just this setup.sh in nixpkgs plus some lines on top of it, put by this simple builder:
...
echo "export SHELL=$shell" > $out/setup
echo "initialPath=\"$initialPath\"" >> $out/setup
echo "defaultNativeBuildInputs=\"$defaultNativeBuildInputs\"" >> $out/setup
echo "$preHook" >> $out/setup
cat "$setup" >> $out/setup
...
Nothing much to say, but you can read the Nix code that pass $initialPath
and $defaultNativeBuildInputs
. Not much interesting to continue further in this pill.The stdenv.mkDerivation function
Until now we worked with plain bash scripts. What about the Nix side? The nixpkgs repository offers a useful function, like we did with our old builder. It is a wrapper around the raw
derivation
function which pulls in the stdenv for us, and runs genericBuild
. It's stdenv.mkDerivation
.Note how
stdenv
is a derivation but it's also an attribute set which contains some other attributes, like mkDerivation
. Nothing fancy here, just convenience.Let's write a
hello.nix
expression using this new discovered stdenv:with import <nixpkgs> {};
stdenv.mkDerivation {
name = "hello";
src = ./hello-2.9.tar.gz;
}
Don't be scared by the with
expression. It pulls the nixpkgs repository into scope, so we can directly use stdenv
. It looks very similar to the hello expression in Pill 8.It builds, and runs fine:
$ nix-build hello.nix
...
/nix/store/6flbdbpq6sc1dc79xjx01bz43zwgj3wc-hello
$ result/bin/hello
Hello, world!
The stdenv.mkDerivation builder
Let's take a look at the builder used by
mkDerivation
. You can read the code here in nixpkgs:{
...
builder = attrs.realBuilder or shell;
args = attrs.args or ["-e" (attrs.builder or ./default-builder.sh)];
stdenv = result;
...
}
Also take a look at our old derivation wrapper in previous pills! The builder
is bash (that shell
variable), the argument to the builder (bash) is default-builder.sh
, and then we add the environment variable $stdenv
in the derivation which is the stdenv
derivation.You can open default-builder.sh and see what it does:
source $stdenv/setup
genericBuild
It's what we did in Pill 10 to make the derivations nix-shell friendly. When entering the shell, the setup file only sets up the environment without building anything. When doing nix-build, it actually runs the build process.To get a clear understanding of the environment variables, look at the .drv of the hello derivation:
$ pp-aterm -i $(nix-instantiate hello.nix)
Derive(
[("out", "/nix/store/6flbdbpq6sc1dc79xjx01bz43zwgj3wc-hello", "", "")]
, [("/nix/store/8z4xw8a0ax1csa0l83zflsm4jw9c94w2-bash-4.3-p39.drv", ["out"]), ("/nix/store/j0905apmxw2qb4ng5j40d4ghpiwa3mi1-stdenv.drv", ["out"])]
, ["/nix/store/0q6pfasdma4as22kyaknk4kwx4h58480-hello-2.9.tar.gz", "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"]
, "x86_64-linux"
, "/nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39/bin/bash"
, ["-e", "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39/bin/bash")
, ("name", "hello")
, ("nativeBuildInputs", "")
, ("out", "/nix/store/6flbdbpq6sc1dc79xjx01bz43zwgj3wc-hello")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/nix/store/0q6pfasdma4as22kyaknk4kwx4h58480-hello-2.9.tar.gz")
, ("stdenv", "/nix/store/k4jklkcag4zq4xkqhkpy156mgfm34ipn-stdenv")
, ("system", "x86_64-linux")
]
)
So short I decided to paste it entirely above. The builder is bash, with -e default-builder.sh
arguments. Then you can see the src
and stdenv
environment variables.Last bit, the
unpackPhase
in the setup is used to unpack the sources and enter the directory, again like we did in our old builder.Conclusion
The stdenv is the core of the nixpkgs repository. All packages use the
stdenv.mkDerivation
wrapper instead of the raw derivation
. It does a bunch of operations for us and also sets up a pleasant build environment.The overall process is simple:
nix-build
bash -e default-builder.sh
source $stdenv/setup
genericBuild
Really, take your time to read that file. Don't forget that juicy docs are also available in the nixpkgs manual.
Next pill...
...we will talk about how to add dependencies to our packages,
buildInputs
, propagatedBuildInputs
and setup hooks. These three concepts are at the base of the current nixpkgs packages composition.
To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.
Thursday, February 19, 2015
NixOS, Consul, Nginx and containers
This is a follow up post on https://medium.com/@dan.ellis/you-dont-need-1mm-for-a-distributed-system-70901d4741e1 . I think the post was well written, so I decided to write a variant using NixOS.
We'll be using declarative nixos containers, which do not use docker but systemd-nspawn. Also systemd is started as init system inside containers.
Please note that this configuration can be applied to any nixos machine, and also the containers configuration could be applied to real servers or other kinds of virtualization, e.g. via nixops. That is, the same syntax and configuration can be reused anywhere else within the nix world.
For example, you could create docker containers with nixos, and keep running the host with another distribution.
However for simplicity we'll use a NixOS system.
Architecture: the host runs nginx and a consul server, then spawns several containers with a python service and a consul client. On the host, consul-template will rewrite the nginx configuration when the health check status of container services change.
Please use a recent unstable release of nixos at the time of this writing (19 Feb 2015, at least commit aec96d4), as it contains the recently packaged consul-template.
Let's write our python service in
Write the following in
Type
Fixing this is about tweaking the systemd service values about the StartTime.
Each container consumes practically no disk space at base. Everything else is shared through the host nix store, except logs, consul state, ecc. of course.
Have fun!
We'll be using declarative nixos containers, which do not use docker but systemd-nspawn. Also systemd is started as init system inside containers.
Please note that this configuration can be applied to any nixos machine, and also the containers configuration could be applied to real servers or other kinds of virtualization, e.g. via nixops. That is, the same syntax and configuration can be reused anywhere else within the nix world.
For example, you could create docker containers with nixos, and keep running the host with another distribution.
However for simplicity we'll use a NixOS system.
Architecture: the host runs nginx and a consul server, then spawns several containers with a python service and a consul client. On the host, consul-template will rewrite the nginx configuration when the health check status of container services change.
Please use a recent unstable release of nixos at the time of this writing (19 Feb 2015, at least commit aec96d4), as it contains the recently packaged consul-template.
Step 1: write the service
Let's write our python service in
/root/work.py
:#!/usr/bin/env python
import random
from flask import Flask
app = Flask(__name__)
def find_prime(start):
"""
Find a prime greater than `start`.
"""
current = start
while True:
for p in xrange(2, current):
if current % p == 0:
break
else:
return current
current += 1
@app.route("/")
def home():
return str(find_prime(random.randint(2 ** 25, 2 ** 26)))
if __name__ == "__main__":
app.run(host='0.0.0.0', port=8080, debug=True)
The only difference with the original post is that we explicitly set the port to 8080 (who knows if a day flask changes the default port).Step 2: write the nixos config
Write the following in
/etc/nixos/prime.nix
:{ lib, pkgs, config, ... }:
let
pypkgs = pkgs.python27Packages;
# Create a self-contained package for our service
work = pkgs.stdenv.mkDerivation {
name = "work";
unpackPhase = "true";
buildInputs = [ pkgs.makeWrapper pypkgs.python pypkgs.flask ];
installPhase = ''
mkdir -p $out/bin
cp ${/root/work.py} $out/bin/work.py
chmod a+rx $out/bin/work.py
wrapProgram $out/bin/work.py --prefix PYTHONPATH : $PYTHONPATH
'';
};
# Function which takes a network and the final octet, and returns a container
mkContainer = net: octet: {
privateNetwork = true;
hostAddress = "${net}.1";
localAddress = "${net}.${octet}";
autoStart = true;
config = { config, pkgs, ... }:
{
users.mutableUsers = false;
# Use this only for debugging, login the machine with machinectl
users.extraUsers.root.password = "root";
# Let consul run check scripts
users.extraUsers.consul.shell = "/run/current-system/sw/bin/bash";
environment.etc."consul/prime.json".text = builtins.toJSON {
service = {
name = "prime";
tags = [ "nginx" ];
port = 8080;
check = {
script = "${pkgs.curl}/bin/curl localhost:8080 >/dev/null 2>&1";
interval = "30s";
};
};
};
systemd.services.prime = {
wantedBy = [ "multi-user.target" ];
serviceConfig = {
ExecStart = "${work}/bin/work.py";
};
};
services.consul = {
enable = true;
extraConfig = {
server = false;
start_join = [ "${net}.1" ];
};
extraConfigFiles = [ "/etc/consul/prime.json" ];
};
networking.firewall = {
allowedTCPPorts = [ 8080 8400 ];
allowPing = true;
};
};
};
nginxTmpl = pkgs.writeText "prime.conf" ''
upstream primes {
{{range service "prime"}}
server {{.Address}}:8080;{{end}}
}
'';
in
{
containers.prime1 = mkContainer "10.50.0" "2";
containers.prime2 = mkContainer "10.50.0" "3";
containers.prime3 = mkContainer "10.50.0" "4";
containers.prime4 = mkContainer "10.50.0" "5";
services.consul = {
enable = true;
extraConfig = {
bootstrap = true;
server = true;
};
};
services.nginx = {
enable = true;
httpConfig = ''
include /etc/nginx/prime.conf;
server {
listen 80;
location / {
proxy_pass http://primes;
}
}
'';
};
systemd.services.nginx = {
preStart = ''
mkdir -p /etc/nginx
touch -a /etc/nginx/prime.conf
'';
serviceConfig = {
Restart = "on-failure";
RestartSec = "1s";
};
};
# Start order: consul -> consul-template -> nginx
systemd.services.consul-template = {
wantedBy = [ "nginx.service" ];
before = [ "nginx.service" ];
wants = [ "consul.service" ];
after = [ "consul.service" ];
serviceConfig = {
Restart = "on-failure";
RestartSec = "1s";
ExecStart = "${pkgs.consul-template}/bin/consul-template -template '${nginxTmpl}:/etc/nginx/prime.conf:systemctl kill -s SIGHUP nginx'";
};
};
boot.kernel.sysctl."net.ipv4.ip_forward" = true;
}
Differences with the original post:- We only create 4 containers instead of 10. I was lazy here. If you are lazy too, you can still automatize the process with nix functions (for example
map
). - We define some ordering in how services start and how they restart with systemd.
- For simplicity we include the
prime.conf
nginx config instead of rewriting the whole nginx config with consul-template. - We create a self-contained package for our python service, so that anywhere it runs the dependencies will be satisfied.
/etc/nixos/configuration.nix
with imports = [ ./prime.nix ];
.Step 3: apply the configuration
Type
nixos-rebuild switch
and then curl http://localhost
. You may have to wait some seconds before consul writes the nginx config. In the while, nginx may have failed to start. If it exceeded the StartTime conditions, you can systemctl start nginx
manually.Fixing this is about tweaking the systemd service values about the StartTime.
Each container consumes practically no disk space at base. Everything else is shared through the host nix store, except logs, consul state, ecc. of course.
Have fun!
Sunday, February 08, 2015
Developing in golang with Nix package manager
I've been using Go since several months. It's a pleasant language, even though it has its own drawbacks.
In our Nixpkgs repository we have support for several programming languages: perl, python, ruby, haskell, lua, ... We've merged a better support for Go.
What kind of support are we talking about? In Nix, you never install libraries. Instead, you define an environment in which to use a certain library compiled for a certain version of the language. The library will be available only within this environment.
Think of it like virtualenv for python, except for any language, and also being able to mix them.
On the other hand Nix requires the src url and the checksum of every dependency of your project. So before starting, make sure you are willing to write nix packages that are not currently present in nixpkgs.
Also you probably have to wait a couple of days before this PR will be available in the unstable channel, at the time of this writing (otherwise
What is
The
Let's take for example
So as you can see, for each go package nix requires a name, the go path, where to fetch the sources, and the hash.
You may be wondering how do you get the sha256: a dirty trick is to write a wrong sha, then nix will tell you the correct sha.
For more examples on how to write dependencies, you can look at nixpkgs goPackages itself.
Drop by #nixos on irc.freenode.net for any doubts.
In our Nixpkgs repository we have support for several programming languages: perl, python, ruby, haskell, lua, ... We've merged a better support for Go.
What kind of support are we talking about? In Nix, you never install libraries. Instead, you define an environment in which to use a certain library compiled for a certain version of the language. The library will be available only within this environment.
Think of it like virtualenv for python, except for any language, and also being able to mix them.
On the other hand Nix requires the src url and the checksum of every dependency of your project. So before starting, make sure you are willing to write nix packages that are not currently present in nixpkgs.
Also you probably have to wait a couple of days before this PR will be available in the unstable channel, at the time of this writing (otherwise
git clone https://github.com/NixOS/nixpkgs.git
).Using nix-shell -p
First a quick example usingnix-shell -p
for your own project:$ nix-shell '<nixpkgs>' -p goPackages.go goPackages.net goPackages.osext
[nix-shell]$ echo $GOPATH
/nix/store/kw9dryid364ac038zmbzq72bnh3zsinz-go-1.4.1-go.net-3338d5f109e9/share/go:...
That's how nix mostly works, it's as simple as that. The GOPATH
is set for every package that provides a directory share/go
.What is
goPackages
? Currently it's go14Packages
, which is all the go packages we have, compiled with go 1.4. There's also go13Packages
, you know some particular packages don't work with go 1.4 yet.Writing a nix file
A more structured example by writing adefault.nix
file in your project:with import <nixpkgs> {}; with goPackages;
buildGoPackage rec {
name = "yourproject";
buildInputs = [ net osext ];
goPackagePath = "github.com/you/yourproject";
}
Then you can just run nix-shell
in your project directory and have your dev environment ready to compile your code.The
goPackagePath
is something needed by buildGoPackage
, in case you are going to run nix-build
. Ignore it for now.Writing a dependency
But nixpkgs doesn't have listed all the possible go projects. What if you need to use a particular library?Let's take for example
github.com/kr/pty
. Write something like this in a pty.nix file:{ goPackages, fetchFromGitHub }:
goPackages.buildGoPackage rec {
rev = "67e2db24c831afa6c64fc17b4a143390674365ef";
name = "pty-${rev}";
goPackagePath = "github.com/kr/pty";
src = fetchFromGitHub {
inherit rev;
owner = "kr";
repo = "pty";
sha256 = "1l3z3wbb112ar9br44m8g838z0pq2gfxcp5s3ka0xvm1hjvanw2d";
};
}
Then in your default.nix:with import <nixpkgs> {}; with goPackages;
let
pty = callPackage ./pty.nix {};
in
buildGoPackage rec {
name = "yourproject";
buildInputs = [ net osext pty ];
goPackagePath = "github.com/you/yourproject";
}
Type nix-shell
and now you will also have pty in your dev environment.So as you can see, for each go package nix requires a name, the go path, where to fetch the sources, and the hash.
You may be wondering how do you get the sha256: a dirty trick is to write a wrong sha, then nix will tell you the correct sha.
Conclusion and references
Nix looks a little complex and boring due to writing a package for each dependency. On the other hand you get for free:- Exact build and runtime dependencies
- Sharing build and runtime dependencies between multiple projects
- Easily test newer or older versions of libraries, without messing with system-wide installations
- Mix with other programming languages, using a similar approach
- Packages using C libraries don't need to be compiled manually by you: define the nix package once, reuse everywhere
For more examples on how to write dependencies, you can look at nixpkgs goPackages itself.
Drop by #nixos on irc.freenode.net for any doubts.
Monday, January 12, 2015
Nix pill 18: nix store paths
Welcome to the 18th Nix pill. In the previous 17th pill we have scratched the surface of the nixpkgs repository structure. It is a set of packages, and it's possible to override such packages so that all other packages will use the overrides.
Before reading existing derivations, I'd like to talk about store paths and how they are computed. In particular we are interested in fixed store paths that depend on an integrity hash (e.g. a sha256), which is usually applied to source tarballs.
The way store paths are computed is a little contrived, mostly due to historical reasons. Our reference will be the Nix source code.
Source paths
Let's start simple. You know nix allows relative paths to be used, such that the file or directory is stored in the nix store, that is
./myfile
gets stored into /nix/store/......
. We want to understand how is the store path generated for such a file:$ echo mycontent > myfile
I remind you, the simplest derivation you can write has a name, a builder and the system:$ nix-repl
nix-repl> derivation { system = "x86_64-linux"; builder = ./myfile; name = "foo"; }
«derivation /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv»
Now inspect the .drv to see where is ./myfile
being stored:$ pp-aterm -i /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv
Derive(
[("out", "/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo", "", "")]
, []
, ["/nix/store/xv2iccirbrvklck36f1g7vldn5v58vck-myfile"]
, "x86_64-linux"
...
Great, how did nix decide to use xv2iccirbrvklck36f1g7vldn5v58vck
? Keep looking at the nix comments.Note: doing
nix-store --add myfile
will store the file in the same store path.Step 1, compute the hash of the file
The comments tell us to first compute the sha256 of the NAR serialization of the file. Can be done in two ways:
$ nix-hash --type sha256 myfile
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3
Or:
$ nix-store --dump myfile|sha256sum
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3 -
In general, Nix understands two contents: flat for regular files, or recursive for NAR serializations which can be anything.Step 2, build the string description
Then nix uses a special string which includes the hash, the path type and the file name. We store this in another file:
$ echo -n "source:sha256:2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3:/nix/store:myfile" > myfile.str
Step 3, compute the final hash
Finally the comments tell us to compute the base-32 representation of the first 160 bits (truncation) of a sha256 of the above string:
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
xv2iccirbrvklck36f1g7vldn5v58vck
Output paths
Output paths are usually generated for derivations. We use the above example because it's simple. Even if we didn't build the derivation, nix knows the out path
hs0yi5n5nw6micqhy8l1igkbhqdkzqa1
. This is because the out path only depends on inputs.It's computed in a similar way to source paths, except that the .drv is hashed and the type of derivation is
output:out
. In case of multiple outputs, we may have different output:<id>
.At the time nix computes the out path, the .drv contains an empty string for each out path. So what we do is getting our .drv and replacing the out path with an empty string:
$ cp -f /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv myout.drv
$ sed -i 's,/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo,,g' myout.drv
The myout.drv
is the .drv state in which nix is when computing the out path for our derivation:$ sha256sum myout.drv
1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5 myout.drv
$ echo -n "output:out:sha256:1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5:/nix/store:foo" > myout.str
$ nix-hash --type sha256 --truncate --base32 --flat myout.str
hs0yi5n5nw6micqhy8l1igkbhqdkzqa1
Then nix puts that out path in the .drv, and that's it.In case the .drv has input derivations, that is it references other .drv, then such .drv paths are replaced by this same algorithm which returns an hash.
In other words, you get a final .drv where every other .drv path is replaced by its hash.
Fixed-output paths
Finally, the other most used kind of path is when we know beforehand an integrity hash of a file. This is usual for tarballs.
A derivation can take three special attributes:
outputHashMode
, outputHash
and outputHashAlgo
which are well documented in the nix manual.The builder must create the out path and make sure its hash is the same as the one declared with
outputHash
.Let's say our builder should create a file whose contents is
mycontent
:$ echo mycontent > myfile
$ sha256sum myfile
f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb myfile
nix-repl> derivation { name = "bar"; system = "x86_64-linux"; builder = "none"; outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb"; }
«derivation /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv»
Inspect the .drv and see that it also stored the fact that it's a fixed-output derivation with sha256 algorithm, compared to the previous examples:$ pp-aterm -i /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv
Derive(
[("out", "/nix/store/a00d5f71k0vp5a6klkls0mvr1f7sx6ch-bar", "sha256", "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb")]
...
It doesn't matter which input derivations are being used, the final out path must only depend on the declared hash.What nix does is to create an intermediate string representation of the fixed-output content:
$ echo -n "fixed:out:sha256:f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb:" > mycontent.str
$ sha256sum mycontent.str
423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639 myfile.str
Then proceed as it was a normal derivation output path:$ echo -n "output:out:sha256:423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639:/nix/store:bar" > myfile.str
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
a00d5f71k0vp5a6klkls0mvr1f7sx6ch
Hence, the store path only depends on the declared fixed-output hash.Conclusion
There are other types of store paths, but you get the idea. Nix first hashes the contents, then creates a string description, and the final store path is the hash of this string.
Also we've introduced some fundamentals, in particular the fact that Nix knows beforehand the out path of a derivation since it only depends on the inputs. We've also introduced fixed-output derivations which are especially used by the nixpkgs repository for downloading and verifying source tarballs.
Also we've introduced some fundamentals, in particular the fact that Nix knows beforehand the out path of a derivation since it only depends on the inputs. We've also introduced fixed-output derivations which are especially used by the nixpkgs repository for downloading and verifying source tarballs.
Next pill
...we will introduce stdenv. In the previous pills we rolled our own mkDerivation convenience function for wrapping the builtin derivation, but the nixpkgs repository also has its own convenience functions for dealing with autotools projects and other build systems.
Pill 19 is available for
reading here.
To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.
To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.
Subscribe to:
Posts (Atom)