tag:blogger.com,1999:blog-320546522024-03-18T05:22:27.241-07:00Thoughts about computer technologiesLuca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.comBlogger158125tag:blogger.com,1999:blog-32054652.post-29314414925242590722018-08-14T12:16:00.001-07:002019-02-23T08:07:24.661-08:00OpenVPN configuration for server and multiple clientsThis is a simple post showing a basic configuration for setting up OpenVPN server accepting multiple clients with TLS.<br />
<br />
First of all generate self-signed server and client private key and certificates, and dh params. <b>Make sure to write the Organization Name AND Common Name (CN) when asked</b>, otherwise openvpn will fail to verify the certificates.<br />
<br />
<code>
openssl req -newkey rsa:2048 -nodes -keyout serverkey.pem -x509 -days 365000 -out servercert.pem<br />
openssl req -newkey rsa:2048 -nodes -keyout clientkey.pem -out client.csr<br />
openssl x509 -req -days 365000 -in client.csr -CA servercert.pem -CAkey serverkey.pem -set_serial 01 -out clientcert.pem<br />
openssl dhparam -outform PEM -out dh.pem 1024<br />
</code>
<br />
<div>
<br /></div>
<div>
Server configuration:<br />
<br />
<code>
dev tun<br />
mode server<br />
tls-server<br />
server 10.8.0.0 255.255.255.0<br />
ca servercert.pem<br />
cert servercert.pem<br />
key serverkey.pem<br />
dh dh.pem<br />
duplicate-cn<br />
topology subnet<br />
keepalive 10 60<br />
ping-timer-rem<br />
persist-tun<br />
persist-key<br />
</code>
</div>
<div>
<br />
If you plan to use the server as gateway like configured below, don't forget to enable IP forwarding and masquerading:<br />
<br />
<code>
sysctl net.ipv4.ip_forward=1<br />
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -j MASQUERADE<br />
</code>
<br />
Client configuration:<br />
<br />
<code>
client<br />
redirect-gateway<br />
tls-client<br />
key clientkey.pem<br />
cert clientcert.pem<br />
ca servercert.pem<br />
keepalive 10 60<br />
dev tun<br />
</code>
<br />
<br />
Have fun, tune as needed.</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com51tag:blogger.com,1999:blog-32054652.post-57591385544426109182016-04-15T01:42:00.001-07:002016-04-15T06:30:47.900-07:00Cheap Docker images with NixLet's talk about <a href="https://www.docker.com/">Docker</a> and <a href="http://nixos.org/nix/">Nix</a> today. Before explaining what Nix is, if you don't know yet, and before going into the details, I will show you a snippet similar to a Dockerfile for creating a Redis image equivalent to the one in <a href="https://github.com/docker-library/redis/blob/master/3.0/Dockerfile">docker hub</a>.<br />
<br />
The final image will be around <strong>42mb</strong> (or <b>25mb</b>) in size, compared to 177mb.<br />
<br />
EDIT: as mentioned on HN, <a href="https://github.com/docker-library/redis/blob/7dec62fe6de187165dce3f771efa57ce4e5d7a32/3.0/alpine/Dockerfile">alpine-based images</a> can even go around 15mb in size.<br />
<br />
If you want to try this, the first step is to <a href="http://nixos.org/nix/download.html">install Nix</a>.<br />
<br />
Here's the <code>redis.nix</code> snippet:<br />
<br />
<script src="https://gist.github.com/lethalman/7e39f8f38c6190dbf26b071e3bfd6188.js"></script><br />
Build it with: <code>nix-build redis.nix</code><br />
Load it with: <code>docker load < result</code><br />
<br />
Once loaded, you can see with <code>docker images</code> that it takes about 42mb of space.<br />
<h2 id="fundamental-differences-with-classic-docker-builds">
</h2>
<h2 id="fundamental-differences-with-classic-docker-builds">
Fundamental differences with classic docker builds</h2>
<ul>
<li>We do not use any base image, like it's done for most docker images including <a href="https://hub.docker.com/_/redis/">redis from the hub</a>. It starts from scratch. In fact, we set up some basic shadow-related files with the <code>shadowSetup</code> utility, enough to add the redis user and make <code>gosu</code> work.</li>
<li>The Redis package is not being compiled inside Docker. It's being done by Nix, just like any other package.</li>
<li>The built image has only one layer, compared to dozens usually spitted by a readable <code>Dockerfile</code>. In our case, having multiple layers is useless because caching is handled by Nix, and not by Docker.</li>
</ul>
<h2 id="a-smaller-image">
</h2>
<h2 id="a-smaller-image">
A smaller image</h2>
We can cut the size down to <strong>25mb</strong> by avoid using <span style="font-family: monospace;">id</span> from coreutils. As an example we'll always launch redis without the entrypoint:<br />
<br />
<script src="https://gist.github.com/lethalman/06f528dfe48ce866de69e66ae44b106b.js"></script><br />
You might ask: but coreutils is still needed for the <code>chown</code>, <code>mkdir</code> and other commands like that!<br />
<br />
The secret is that those commands are only used at build time and are not required at runtime in the container. Nix is able to detect that automatically for us.<br />
<br />
It means we don't need to manually remove packages after the container is built, like with other package managers! See <a href="https://github.com/docker-library/redis/blob/master/3.0/Dockerfile#L40">this line</a> in Redis <code>Dockerfile</code> for example.<br />
<h2 id="using-a-different-redis-version">
</h2>
<h2 id="using-a-different-redis-version">
Using a different redis version</h2>
Let's say we want to build a Docker image with Redis 2.8.23. First we want to write a package (or <em>derivation</em> in Nix land) for it, and then use that inside the image:<br />
<br />
<script src="https://gist.github.com/lethalman/4d2dabc1fe6342b621af108eb5f26905.js"></script><br />
Note we also added the tag 2.8.23 to the resulting image. And that's it. The beauty is that we reuse the same redis expression from nixpkgs, but we override only the version to build.<br />
<h2 id="a-generic-build">
</h2>
<h2 id="a-generic-build">
A generic build</h2>
There's more you can do with Nix. Being a language, it's possible to create a generic function for building Redis images given a specific package:<br />
<br />
<script src="https://gist.github.com/lethalman/67b3bf4475737873dc81831347a182d3.js"></script><br />
We created a "redisImage" function that takes a "redis" parameter as input, and returns a Docker image as output.<br />
<br />
Build it with:<br />
<ul>
<li><code>nix-build redis-generic.nix -A redisDocker_3_0_7</code> </li>
<li><code>nix-build redis-generic.nix -A redisDocker_2_8_23</code></li>
</ul>
<br />
<h2 id="building-off-a-base-image">
</h2>
<h2 id="building-off-a-base-image">
Building off a base image</h2>
One of the selling points of Docker is reusing an existing image to add more stuff on top of it.<br />
<br />
Nix comes with a completely different set of packages compared to other distros, with its own toolchain and glibc version. This doesn't mean it's not possible to base a new image off an existing Debian image for instance.<br />
<br />
By using <code>dockerTools.pullImage</code> it's also possible to pull images from the Docker hub.<br />
<br />
<script src="https://gist.github.com/lethalman/6fb7377b5c274683e70ebf497df1b827.js"></script><br />
Build it with: <code>nix-build redis-generic.nix -A redisOnDebian</code>.<br />
<br />
Note that we added a couple of things. We pass the base image (<code>debianImage</code>), to our generic <code>redisImage</code> function, and that we only initialize shadow-utils if the base image is null.<br />
<br />
The result is a Docker image based off latest Debian but running Redis compiled with nixpkgs toolchain and using nixpkgs glibc. It's about 150mb. It has all the layers from the base image, plus the new single layer for Redis.<br />
<br />
That said, it's as well possible to use one of the previously defined Redis images as base image. The result of `pullImage` and `buildImage` is a .tar.gz docker image in both cases.<br />
<br />
You realize it's possible to build something quite similar to <a href="https://github.com/docker-library">docker-library</a> using only Nix expressions. It might be an interesting project.<br />
<br />
<em>Be aware that things like PAM configurations, or other stuff, created to be suitable for Debian may not work with Nix programs that use a different glibc.</em><br />
<h2 id="other-random-details">
</h2>
<h2 id="other-random-details">
Other random details</h2>
The code above has been made possible by using nixpkgs commit 3ae4d2afe (2016-04-14) onwards, commit at which I've finally packaged <code>gosu</code> and since the size of the derivations have been notably reduced.<br />
<br />
Building the image is done without using any of the Docker commands. The way it works is as follows:<br />
<ol>
<li>Create a layer directory with all the produced contents inside. This includes the filesystem as well as the json metadata. This process will use certain build dependencies (like <code>coreutils</code>, <code>shadow-utils</code>, <code>bash</code>, <code>redis</code>, <code>gosu</code>, ...).</li>
<li>Ask Nix what are the runtime dependencies of the layer directory (like <code>redis</code>, <code>gosu</code>). Such dependencies will be always a subset of the build dependencies.</li>
<li>Add such runtime dependencies to the layer directory.</li>
<li>Pack the layer in a .tar.gz by following the <a href="https://github.com/docker/docker/blob/master/image/spec/v1.md">Docker specification</a>.</li>
</ol>
I'd like to state that <strong>Nix has a safer and easier caching</strong> of operations while building the image. <br />
As for Docker, great care has to be taken in order to use the layer cache correctly, because such caching is solely based on the <code>RUN</code> command string. This <a href="http://thenewstack.io/understanding-the-docker-cache-for-faster-builds/">blog post</a> explains it well.<br />
This is not the case for Nix, because every output depends on a set of exact inputs. If any of the inputs change, the output will be rebuilt.<br />
<h2 id="so-what-is-nix">
</h2>
<h2 id="so-what-is-nix">
So what is Nix?</h2>
<a href="http://nixos.org/nix/">Nix</a> is a language and deployment tool, often used as package manager or configuration builder and system provisioning. The operating system <a href="http://nixos.org/nixos">NixOS</a> is based on it. <br />
<br />
The code shown above is Nix. We have used the <a href="https://github.com/NixOS/nixpkgs">nixpkgs</a> repository which provides several reusable Nix expressions like <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/nosql/redis/default.nix">redis</a> and <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/docker/default.nix">dockerTools</a>.<br />
<br />
The Nix concept is simple: write a Nix expression, build it. This is how the building process works at a high-level:<br />
<ol>
<li>Read a Nix expression</li>
<li>Evaluate it and determine the thing (called <em>derivation</em>) to be built.</li>
<li>By evaluating the code, Nix is able to determine exactly the build inputs needed for such derivation.</li>
<li>Build (or fetch from cache) all the needed inputs.</li>
<li>Build (or fetch from the cache) the final derivation.</li>
</ol>
Nix stores all such derivations in a common nix store (usually <code>/nix/store</code>), identified by an hash. Each derivation may have dependencies to other paths in the same store. Each derivation is stored in a separate directory from other derivations.<br />
<br />
Won't go deeper as there's plenty of documentation about how Nix works and how its storage works.<br />
<br />
Hope you enjoyed the reading, and that you may give Nix a shot.<br />
<br />Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com412tag:blogger.com,1999:blog-32054652.post-73701720883000883982016-01-04T13:33:00.000-08:002016-01-04T13:42:14.687-08:00TypeScript and NodeJS, I'm sold<div class="wmd-preview-section preview-content" id="wmd-preview-section-72" style="background-color: #f6f6f6; box-sizing: border-box; color: #3f3f3f; font-family: 'Source Sans Pro', sans-serif; font-size: 17px; line-height: 19.72px; outline: none !important; text-align: justify;">
<div style="box-sizing: border-box; margin-bottom: 1.1em; outline: none !important;">
<a href="http://www.typescriptlang.org/" style="background: transparent; box-sizing: border-box; color: #4fa1db; outline: none !important; text-decoration: none; transition: background-color 0.15s ease-in-out, color 0.15s ease-in-out, border-color 0.15s ease-in-out;">TypeScript</a> is a typed superset of JavaScript that compiles to plain JavaScript, the way you expect it to be.</div>
<div style="box-sizing: border-box; margin-bottom: 1.1em; outline: none !important;">
I’ve heard of it a long time ago, but recently with <a href="https://github.com/Microsoft/TypeScript/wiki/What%27s-new-in-TypeScript#typescript-17" style="background: transparent; box-sizing: border-box; color: #4fa1db; outline: none !important; text-decoration: none; transition: background-color 0.15s ease-in-out, color 0.15s ease-in-out, border-color 0.15s ease-in-out;">TypeScript 1.7</a> it got <code style="background-color: rgba(128, 128, 128, 0.0745098); border-radius: 4px; box-sizing: border-box; font-family: 'Source Code Pro', monospace; font-size: 0.9em; outline: none !important; padding: 2px 4px;">async</code> functions, which means you can <code style="background-color: rgba(128, 128, 128, 0.0745098); border-radius: 4px; box-sizing: border-box; font-family: 'Source Code Pro', monospace; font-size: 0.9em; outline: none !important; padding: 2px 4px;">await</code>asynchronous function calls, similarly to C#, Vala, Go and other languages with syntax support for concurrency. That makes coroutines a pleasant experience compared to plain JavaScript. That’s also the main reason why I didn’t choose Dart.</div>
<div style="box-sizing: border-box; margin-bottom: 1.1em; outline: none !important;">
I’m writing a NodeJS application so I decided to give it a go. Here’s my personal tour of TypeScript and why I’m sold to using it.</div>
</div>
<div class="wmd-preview-section preview-content" id="wmd-preview-section-73" style="background-color: #f6f6f6; box-sizing: border-box; color: #3f3f3f; font-family: 'Source Sans Pro', sans-serif; line-height: 19.72px; outline: none !important; text-align: justify;">
<h3 style="box-sizing: border-box; color: inherit; font-family: inherit; line-height: 1.1; margin: 1.5em 0px; outline: none !important; text-align: start;">
<span style="font-size: large;">Does it complain when using undeclared variables?</span></h3>
</div>
<div class="wmd-preview-section preview-content" id="wmd-preview-section-1410" style="background-color: #f6f6f6; box-sizing: border-box; color: #3f3f3f; font-family: 'Source Sans Pro', sans-serif; line-height: 19.72px; outline: none !important; text-align: justify;">
<pre class="prettyprint" style="background-color: rgba(128, 128, 128, 0.0470588); border-image-outset: initial; border-image-repeat: initial; border-image-slice: initial; border-image-source: initial; border-image-width: initial; border-radius: 5px; border: 0px; box-sizing: border-box; color: #333333; font-family: 'Source Code Pro', monospace; line-height: 1.45; margin-bottom: 1.1em; outline: none !important; padding: 10px 20px; text-align: start; white-space: pre-wrap; word-break: break-word; word-wrap: break-word;"><code class="language-typescript hljs rust" style="background-attachment: initial; background-clip: initial; background-color: transparent !important; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: black; display: block; font-family: 'Source Code Pro', monospace; font-size: inherit; outline: none !important; padding: 0.5em;">console.<span class="hljs-keyword" style="box-sizing: border-box; font-weight: bold; outline: none !important;">log</span>(foo);</code></pre>
<pre style="background-color: rgba(128, 128, 128, 0.0470588); border-image-outset: initial; border-image-repeat: initial; border-image-slice: initial; border-image-source: initial; border-image-width: initial; border-radius: 5px; border: 0px; box-sizing: border-box; color: #333333; font-family: 'Source Code Pro', monospace; font-size: 0.9em; line-height: 1.45; margin-bottom: 1.1em; outline: none !important; padding: 10px 20px; text-align: start; white-space: pre-wrap; word-break: break-word; word-wrap: break-word;"><code style="background-color: transparent !important; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: 'Source Code Pro', monospace; font-size: inherit; outline: none !important; padding: 0px;">Cannot find name 'foo'.
</code></pre>
<div style="box-sizing: border-box; font-size: 17px; margin-bottom: 1.1em; outline: none !important;">
Sold!</div>
</div>
<div class="wmd-preview-section preview-content" id="wmd-preview-section-1411" style="background-color: #f6f6f6; box-sizing: border-box; color: #3f3f3f; font-family: 'Source Sans Pro', sans-serif; line-height: 19.72px; outline: none !important; text-align: justify;">
<h3 style="box-sizing: border-box; color: inherit; font-family: inherit; line-height: 1.1; margin: 1.5em 0px; outline: none !important; text-align: start;">
<span style="font-size: large;">Does it infer types?</span></h3>
</div>
<div class="wmd-preview-section preview-content" id="wmd-preview-section-1412" style="background-color: #f6f6f6; box-sizing: border-box; color: #3f3f3f; font-family: 'Source Sans Pro', sans-serif; line-height: 19.72px; outline: none !important; text-align: justify;">
<pre class="prettyprint" style="background-color: rgba(128, 128, 128, 0.0470588); border-image-outset: initial; border-image-repeat: initial; border-image-slice: initial; border-image-source: initial; border-image-width: initial; border-radius: 5px; border: 0px; box-sizing: border-box; color: #333333; font-family: 'Source Code Pro', monospace; line-height: 1.45; margin-bottom: 1.1em; outline: none !important; padding: 10px 20px; text-align: start; white-space: pre-wrap; word-break: break-word; word-wrap: break-word;"><code class="language-typescript hljs cs" style="background-attachment: initial; background-clip: initial; background-color: transparent !important; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: black; display: block; font-family: 'Source Code Pro', monospace; font-size: inherit; outline: none !important; padding: 0.5em;"><span class="hljs-keyword" style="box-sizing: border-box; font-weight: bold; outline: none !important;">var</span> foo = <span class="hljs-number" style="box-sizing: border-box; color: #008800; outline: none !important;">123</span>;
foo = <span class="hljs-string" style="box-sizing: border-box; color: #880000; outline: none !important;">"bar"</span>;</code></pre>
<pre style="background-color: rgba(128, 128, 128, 0.0470588); border-image-outset: initial; border-image-repeat: initial; border-image-slice: initial; border-image-source: initial; border-image-width: initial; border-radius: 5px; border: 0px; box-sizing: border-box; color: #333333; font-family: 'Source Code Pro', monospace; font-size: 0.9em; line-height: 1.45; margin-bottom: 1.1em; outline: none !important; padding: 10px 20px; text-align: start; white-space: pre-wrap; word-break: break-word; word-wrap: break-word;"><code style="background-color: transparent !important; border-radius: 0px; box-sizing: border-box; color: inherit; font-family: 'Source Code Pro', monospace; font-size: inherit; outline: none !important; padding: 0px;">Type 'string' is not assignable to type 'number'.
</code></pre>
<div style="box-sizing: border-box; font-size: 17px; margin-bottom: 1.1em; outline: none !important;">
Sold!</div>
</div>
<div class="wmd-preview-section preview-content" id="wmd-preview-section-1413" style="background-color: #f6f6f6; box-sizing: border-box; color: #3f3f3f; font-family: 'Source Sans Pro', sans-serif; line-height: 19.72px; outline: none !important; text-align: justify;">
<h3 style="box-sizing: border-box; color: inherit; font-family: inherit; line-height: 1.1; margin: 1.5em 0px; outline: none !important; text-align: start;">
<span style="font-size: large;">Does it support async arrow functions?</span></h3>
</div>
<div class="wmd-preview-section preview-content" id="wmd-preview-section-1414" style="background-color: #f6f6f6; box-sizing: border-box; color: #3f3f3f; font-family: 'Source Sans Pro', sans-serif; line-height: 19.72px; outline: none !important; text-align: justify;">
<pre class="prettyprint" style="background-color: rgba(128, 128, 128, 0.0470588); border-image-outset: initial; border-image-repeat: initial; border-image-slice: initial; border-image-source: initial; border-image-width: initial; border-radius: 5px; border: 0px; box-sizing: border-box; color: #333333; font-family: 'Source Code Pro', monospace; line-height: 1.45; margin-bottom: 1.1em; outline: none !important; padding: 10px 20px; text-align: start; white-space: pre-wrap; word-break: break-word; word-wrap: break-word;"><code class="language-typescript hljs php" style="background-attachment: initial; background-clip: initial; background-color: transparent !important; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: black; display: block; font-family: 'Source Code Pro', monospace; font-size: inherit; outline: none !important; padding: 0.5em;">async <span class="hljs-function" style="box-sizing: border-box; outline: none !important;"><span class="hljs-keyword" style="box-sizing: border-box; font-weight: bold; outline: none !important;">function</span> <span class="hljs-title" style="box-sizing: border-box; color: #880000; font-weight: bold; outline: none !important;">foo</span><span class="hljs-params" style="box-sizing: border-box; outline: none !important;">()</span> {</span>
}
<span class="hljs-keyword" style="box-sizing: border-box; font-weight: bold; outline: none !important;">var</span> bar = async () => { await (foo); };</code></pre>
<div style="box-sizing: border-box; font-size: 17px; margin-bottom: 1.1em; outline: none !important;">
Sold!</div>
</div>
<div class="wmd-preview-section preview-content" id="wmd-preview-section-1415" style="background-color: #f6f6f6; box-sizing: border-box; color: #3f3f3f; font-family: 'Source Sans Pro', sans-serif; line-height: 19.72px; outline: none !important; text-align: justify;">
<h3 style="box-sizing: border-box; color: inherit; font-family: inherit; line-height: 1.1; margin: 1.5em 0px; outline: none !important; text-align: start;">
<span style="font-size: large;">Does it support sum types?</span></h3>
</div>
<div class="wmd-preview-section preview-content" id="wmd-preview-section-1416" style="background-color: #f6f6f6; box-sizing: border-box; color: #3f3f3f; font-family: 'Source Sans Pro', sans-serif; line-height: 19.72px; outline: none !important; text-align: justify;">
<pre class="prettyprint" style="background-color: rgba(128, 128, 128, 0.0470588); border-image-outset: initial; border-image-repeat: initial; border-image-slice: initial; border-image-source: initial; border-image-width: initial; border-radius: 5px; border: 0px; box-sizing: border-box; color: #333333; font-family: 'Source Code Pro', monospace; line-height: 1.45; margin-bottom: 1.1em; outline: none !important; padding: 10px 20px; text-align: start; white-space: pre-wrap; word-break: break-word; word-wrap: break-word;"><code class="language-typescript hljs cs" style="background-attachment: initial; background-clip: initial; background-color: transparent !important; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: black; display: block; font-family: 'Source Code Pro', monospace; font-size: inherit; outline: none !important; padding: 0.5em;"><span class="hljs-keyword" style="box-sizing: border-box; font-weight: bold; outline: none !important;">var</span> foo: number | <span class="hljs-keyword" style="box-sizing: border-box; font-weight: bold; outline: none !important;">string</span>;
foo = <span class="hljs-number" style="box-sizing: border-box; color: #008800; outline: none !important;">123</span>;
foo = <span class="hljs-string" style="box-sizing: border-box; color: #880000; outline: none !important;">"bar"</span>;</code></pre>
<div style="box-sizing: border-box; font-size: 17px; margin-bottom: 1.1em; outline: none !important;">
Sold!</div>
</div>
<div class="wmd-preview-section preview-content" id="wmd-preview-section-81" style="background-color: #f6f6f6; box-sizing: border-box; color: #3f3f3f; font-family: 'Source Sans Pro', sans-serif; line-height: 19.72px; outline: none !important; text-align: justify;">
<h3 style="box-sizing: border-box; color: inherit; font-family: inherit; line-height: 1.1; margin: 1.5em 0px; outline: none !important; text-align: start;">
<span style="font-size: large;">Does it play nice with CommonJS/AMD imports and external libraries?</span></h3>
<div style="box-sizing: border-box; font-size: 17px; margin-bottom: 1.1em; outline: none !important;">
Yes, it does very well. Sold!</div>
</div>
<div class="wmd-preview-section preview-content" id="wmd-preview-section-82" style="background-color: #f6f6f6; box-sizing: border-box; color: #3f3f3f; font-family: 'Source Sans Pro', sans-serif; line-height: 19.72px; outline: none !important; text-align: justify;">
<h3 style="box-sizing: border-box; color: inherit; font-family: inherit; line-height: 1.1; margin: 1.5em 0px; outline: none !important; text-align: start;">
<span style="font-size: large;">Is it easy to migrate from and to JavaScript?</span></h3>
<div style="box-sizing: border-box; font-size: 17px; margin-bottom: 1.1em; outline: none !important;">
Yes. TypeScript makes use of latest ECMAScript features in its syntax when possible, so that JavaScript -> TypeScript is as painless as possible. Sold!</div>
<div style="box-sizing: border-box; font-size: 17px; margin-bottom: 1.1em; outline: none !important;">
Also to go back from TypeScript to JavaScript, either use the generated code, or remove all the type annotations in the code by yourself.</div>
</div>
<div class="wmd-preview-section preview-content" id="wmd-preview-section-1443" style="background-color: #f6f6f6; box-sizing: border-box; color: #3f3f3f; font-family: 'Source Sans Pro', sans-serif; line-height: 19.72px; outline: none !important; text-align: justify;">
<h3 style="box-sizing: border-box; color: inherit; font-family: inherit; line-height: 1.1; margin: 1.5em 0px; outline: none !important; text-align: start;">
<span style="font-size: large;">Does it have non-nullable variables?</span></h3>
<div style="box-sizing: border-box; font-size: 17px; margin-bottom: 1.1em; outline: none !important;">
No. This is mostly due to the JavaScript nature though. But I’m sure the TypeScript community will come up with a nice solution <a href="https://github.com/Microsoft/TypeScript/issues/185" style="background: transparent; box-sizing: border-box; color: #4fa1db; outline: none !important; text-decoration: none; transition: background-color 0.15s ease-in-out, color 0.15s ease-in-out, border-color 0.15s ease-in-out;">throughout this topic</a>.</div>
<hr style="border-bottom-width: 0px; border-left-width: 0px; border-right-width: 0px; border-top-color: rgba(128, 128, 128, 0.0980392); border-top-style: solid; box-sizing: content-box; font-size: 17px; height: 0px; margin: 2em 0px; outline: none !important;" />
<div style="box-sizing: border-box; font-size: 17px; margin-bottom: 1.1em; outline: none !important;">
I’m going to use TypeScript wherever I can in this new year instead of plain JavaScript. In particular, I’m rewriting my latest NodeJS application in TypeScript right now.</div>
<div style="box-sizing: border-box; font-size: 17px; margin-bottom: 1.1em; outline: none !important;">
I hope it will be a great year for this language. The project is exceptionally active, and I hope to contribute back to it.</div>
</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com14tag:blogger.com,1999:blog-32054652.post-29510192818826784552015-08-24T09:48:00.000-07:002015-08-25T03:25:02.873-07:00Nix pill 19: fundamentals of stdenvWelcome to the 19th Nix pill. In the previous <a href="http://lethalman.blogspot.it/2015/01/nix-pill-18-nix-store-paths.html">18th pill</a> we did dive into the algorithm used by Nix to compute the store paths, and also introduced fixed-output store paths.<br />
This time we will instead look into nixpkgs, in particular one of its core derivation: <strong>stdenv</strong> .<br />
<br />
The stdenv is not a special derivation, but it's very important for the nixpkgs repository. It serves as base for packaging software. It is used to pull in dependencies such as the GCC toolchain, GNU make, core utilities, patch and diff utilities, and so on. Basic tools needed to compile a huge pile of software currently present in nixpkgs.<br />
<br />
<h1 id="what-is-stdenv">
What is stdenv</h1>
<br />
First of all stdenv is a derivation. And it's a very simple one:<br />
<pre><code>$ nix-build '<nixpkgs>' -A stdenv
/nix/store/k4jklkcag4zq4xkqhkpy156mgfm34ipn-stdenv
$ ls -R result/
result/:
nix-support/ setup
result/nix-support:
propagated-user-env-packages
</nixpkgs></code></pre>
It has just two files: <code>/setup</code> and <code>/nix-support/propagated-user-env-packages</code>. Don't care about the latter, it's even empty. The important file is <code>/setup</code>.<br />
How can this simple derivation pull in all the toolchain and basic tools needed to compile packages? Let's look at the runtime dependencies:<br />
<pre><code>$ nix-store -q --references result
/nix/store/3a45nb37s0ndljp68228snsqr3qsyp96-bzip2-1.0.6
/nix/store/a457ywa1haa0sgr9g7a1pgldrg3s798d-coreutils-8.24
/nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39
/nix/store/47sfpm2qclpqvrzijizimk4md1739b1b-gcc-wrapper-4.9.3
...
</code></pre>
How can it be? The package must be referring to those package somehow. In fact, they are hardcoded in the <code>/setup</code> file:<br />
<pre><code>$ head result/setup
export SHELL=/nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39/bin/bash
initialPath="/nix/store/a457ywa1haa0sgr9g7a1pgldrg3s798d-coreutils-8.24 ..."
defaultNativeBuildInputs="/nix/store/sgwq15xg00xnm435gjicspm048rqg9y6-patchelf-0.8 ..."
</code></pre>
<h1 id="the-setup-file">
The setup file</h1>
<br />
Remember our generic <code>builder.sh</code> in <a href="http://lethalman.blogspot.it/2014/08/nix-pill-8-generic-builders.html">Pill 8</a>? It sets up a basic PATH, unpacks the source and runs the usual autotools commands for us.<br />
The stdenv setup file is exactly that. It sets up several environment variables like PATH and creates some helper bash functions to build a package. I invite you to read it, it's only 860 lines at the time of this writing.<br />
<br />
The hardcoded toolchain and utilities are used to initially fill up the environment variables so that it's more pleasant to run common commands, similarly but not equal like we did with our builder with <code>baseInputs</code> and <code>buildInputs</code>.<br />
<br />
The build with stdenv works in phases. Phases are like <code>unpackPhase</code>, <code>configurePhase</code>, <code>buildPhase</code>, <code>checkPhase</code>, <code>installPhase</code>, <code>fixupPhase</code>. You can see the default list in the <code>genericBuild</code> function.<br />
What <code>genericBuild</code> does is just run these phases. Default phases are just bash functions, you can easily read them.<br />
<br />
Every phase has hooks to run commands before and after the phase has been executed. Phases can be overwritten, reordered, whatever, it's just bash code.<br />
<br />
How to use this file? Like our old builder. To test it, we enter a fake empty derivation, source the stdenv setup, unpack the hello sources and build it:<br />
<pre><code>$ nix-shell -E 'derivation { name = "fake"; builder = "fake"; system = "x86_64-linux"; }'
nix-shell$ unset PATH
nix-shell$ source /nix/store/k4jklkcag4zq4xkqhkpy156mgfm34ipn-stdenv/setup
nix-shell$ tar -xf hello-2.9.tar.gz
nix-shell$ cd hello-2.9
nix-shell$ configurePhase
...
nix-shell$ buildPhase
...
</code></pre>
<i>I unset <code>PATH</code> to further show that the stdenv is enough self-contained to build autotools packages that have no other dependencies.</i><br />
<br />
So we ran the <code>configurePhase</code> function and <code>buildPhase</code> function and they worked. These bash functions should be self-explanatory, you can read the code in the setup file.<br />
<br />
<h1 id="how-is-the-setup-file-built">
How is the setup file built</h1>
<br />
Very little digression for completeness. The stdenv derivation is just that setup file. That setup file is just this <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh">setup.sh in nixpkgs</a> plus some lines on top of it, put by this <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/builder.sh">simple builder</a>:<br />
<pre><code>...
echo "export SHELL=$shell" > $out/setup
echo "initialPath=\"$initialPath\"" >> $out/setup
echo "defaultNativeBuildInputs=\"$defaultNativeBuildInputs\"" >> $out/setup
echo "$preHook" >> $out/setup
cat "$setup" >> $out/setup
...
</code></pre>
Nothing much to say, but you can read the Nix code that pass <code>$initialPath</code> and <code>$defaultNativeBuildInputs</code>. Not much interesting to continue further in this pill.<br />
<br />
<h1 id="the-stdenvmkderivation-function">
The stdenv.mkDerivation function</h1>
<br />
Until now we worked with plain bash scripts. What about the Nix side? The nixpkgs repository offers a useful function, like we did with our old builder. It is a wrapper around the raw <code>derivation</code> function which pulls in the stdenv for us, and runs <code>genericBuild</code>. It's <code>stdenv.mkDerivation</code>.<br />
<br />
Note how <code>stdenv</code> is a derivation but it's also an attribute set which contains some other attributes, like <code>mkDerivation</code>. Nothing fancy here, just convenience.<br />
<br />
Let's write a <code>hello.nix</code> expression using this new discovered stdenv:<br />
<pre><code>with import <nixpkgs> {};
stdenv.mkDerivation {
name = "hello";
src = ./hello-2.9.tar.gz;
}
</code></pre>
Don't be scared by the <code>with</code> expression. It pulls the nixpkgs repository into scope, so we can directly use <code>stdenv</code>. It looks very similar to the hello expression in <a href="http://lethalman.blogspot.it/2014/08/nix-pill-8-generic-builders.html">Pill 8</a>.<br />
It builds, and runs fine:<br />
<pre><code>$ nix-build hello.nix
...
/nix/store/6flbdbpq6sc1dc79xjx01bz43zwgj3wc-hello
$ result/bin/hello
Hello, world!
</code></pre>
<h1 id="the-stdenvmkderivation-builder">
The stdenv.mkDerivation builder</h1>
<br />
Let's take a look at the builder used by <code>mkDerivation</code>. You can read the code <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/default.nix#L155">here in nixpkgs</a>:<br />
<pre><code>{
...
builder = attrs.realBuilder or shell;
args = attrs.args or ["-e" (attrs.builder or ./default-builder.sh)];
stdenv = result;
...
}
</code></pre>
Also take a look at our old derivation wrapper in previous pills! The <code>builder</code> is bash (that <code>shell</code> variable), the argument to the builder (bash) is <code>default-builder.sh</code>, and then we add the environment variable <code>$stdenv</code> in the derivation which is the <code>stdenv</code> derivation.<br />
<br />
You can open <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/default-builder.sh">default-builder.sh</a> and see what it does:<br />
<pre><code>source $stdenv/setup
genericBuild
</code></pre>
It's what we did in <a href="http://lethalman.blogspot.it/2014/08/nix-pill-10-developing-with-nix-shell.html">Pill 10</a> to make the derivations nix-shell friendly. When entering the shell, the setup file only sets up the environment without building anything. When doing nix-build, it actually runs the build process.<br />
<br />
To get a clear understanding of the environment variables, look at the .drv of the hello derivation:<br />
<pre><code>$ pp-aterm -i $(nix-instantiate hello.nix)
Derive(
[("out", "/nix/store/6flbdbpq6sc1dc79xjx01bz43zwgj3wc-hello", "", "")]
, [("/nix/store/8z4xw8a0ax1csa0l83zflsm4jw9c94w2-bash-4.3-p39.drv", ["out"]), ("/nix/store/j0905apmxw2qb4ng5j40d4ghpiwa3mi1-stdenv.drv", ["out"])]
, ["/nix/store/0q6pfasdma4as22kyaknk4kwx4h58480-hello-2.9.tar.gz", "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"]
, "x86_64-linux"
, "/nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39<b>/bin/bash</b>"
, ["-e", "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-<b>default-builder.sh</b>"]
, [ ("buildInputs", "")
, ("builder", "/nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39/bin/bash")
, ("name", "hello")
, ("nativeBuildInputs", "")
, ("out", "/nix/store/6flbdbpq6sc1dc79xjx01bz43zwgj3wc-hello")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/nix/store/0q6pfasdma4as22kyaknk4kwx4h58480-<b>hello-2.9.tar.gz</b>")
, ("stdenv", "/nix/store/k4jklkcag4zq4xkqhkpy156mgfm34ipn-<b>stdenv</b>")
, ("system", "x86_64-linux")
]
)
</code></pre>
So short I decided to paste it entirely above. The builder is bash, with <code>-e default-builder.sh</code> arguments. Then you can see the <code>src</code> and <code>stdenv</code> environment variables.<br />
<br />
Last bit, the <code>unpackPhase</code> in the setup is used to unpack the sources and enter the directory, again like we did in our old builder.<br />
<br />
<h1 id="conclusion">
Conclusion</h1>
<br />
The stdenv is the core of the nixpkgs repository. All packages use the <code>stdenv.mkDerivation</code> wrapper instead of the raw <code>derivation</code>. It does a bunch of operations for us and also sets up a pleasant build environment.<br />
<br />
The overall process is simple:<br />
<ul>
<li><code>nix-build</code></li>
<li><code>bash -e default-builder.sh</code></li>
<li><code>source $stdenv/setup</code></li>
<li><code>genericBuild</code></li>
</ul>
That's it, everything you need to know about the stdenv phases is in the <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh">setup file</a>.<br />
<br />
Really, take your time to read that file. Don't forget that juicy docs are also available in the <a href="http://nixos.org/nixpkgs/manual/#chap-stdenv">nixpkgs manual</a>.<br />
<br />
<h1 id="next-pill">
Next pill...</h1>
<br />
...we will talk about how to add dependencies to our packages, <code>buildInputs</code>, <code>propagatedBuildInputs</code> and setup hooks. These three concepts are at the base of the current nixpkgs packages composition.<br />
<br />
<h3>
<div style="font-size: medium; font-weight: normal;">
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.<br />
<div>
<br /></div>
</div>
<div style="font-size: medium; font-weight: normal;">
</div>
</h3>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com15tag:blogger.com,1999:blog-32054652.post-29887971568853289192015-02-19T03:59:00.001-08:002015-02-19T04:12:22.730-08:00NixOS, Consul, Nginx and containersThis is a follow up post on <a href="https://medium.com/@dan.ellis/you-dont-need-1mm-for-a-distributed-system-70901d4741e1">https://medium.com/@dan.ellis/you-dont-need-1mm-for-a-distributed-system-70901d4741e1</a> . I think the post was well written, so I decided to write a variant using <a href="http://nixos.org/">NixOS</a>.<br />
<br />
We'll be using <a href="https://nixos.org/releases/nixos/14.12/nixos-14.12.374.61adf9e/manual/ch-containers.html#sec-declarative-containers">declarative nixos containers</a>, which do not use docker but <a href="http://www.freedesktop.org/software/systemd/man/systemd-nspawn.html">systemd-nspawn</a>. Also systemd is started as init system inside containers.<br />
<br />
Please note that this configuration can be applied to any nixos machine, and also the containers configuration could be applied to real servers or other kinds of virtualization, e.g. via <a href="http://nixos.org/nixops/">nixops</a>. That is, the same syntax and configuration can be reused anywhere else within the nix world.<br />
<br />
For example, you could create docker containers with nixos, and keep running the host with another distribution.<br />
<br />
However for simplicity we'll use a NixOS system.<br />
<br />
Architecture: the host runs nginx and a consul server, then spawns several containers with a python service and a consul client. On the host, <a href="https://github.com/hashicorp/consul-template">consul-template</a> will rewrite the nginx configuration when the health check status of container services change.<br />
<br />
Please use a recent unstable release of nixos at the time of this writing (19 Feb 2015, at least commit aec96d4), as it contains the recently packaged <a href="https://github.com/hashicorp/consul-template">consul-template</a>.<br />
<br />
<h3>
Step 1: write the service</h3>
<br />
Let's write our python service in <code>/root/work.py</code>:<br />
<pre><code>#!/usr/bin/env python
import random
from flask import Flask
app = Flask(__name__)
def find_prime(start):
"""
Find a prime greater than `start`.
"""
current = start
while True:
for p in xrange(2, current):
if current % p == 0:
break
else:
return current
current += 1
@app.route("/")
def home():
return str(find_prime(random.randint(2 ** 25, 2 ** 26)))
if __name__ == "__main__":
app.run(host='0.0.0.0', port=8080, debug=True)
</code></pre>
The only difference with the original post is that we explicitly set the port to 8080 (who knows if a day flask changes the default port).<br />
<br />
<h3>
Step 2: write the nixos config</h3>
<br />
Write the following in <code>/etc/nixos/prime.nix</code>:<br />
<pre><code>{ lib, pkgs, config, ... }:
let
pypkgs = pkgs.python27Packages;
# Create a self-contained package for our service
work = pkgs.stdenv.mkDerivation {
name = "work";
unpackPhase = "true";
buildInputs = [ pkgs.makeWrapper pypkgs.python pypkgs.flask ];
installPhase = ''
mkdir -p $out/bin
cp ${/root/work.py} $out/bin/work.py
chmod a+rx $out/bin/work.py
wrapProgram $out/bin/work.py --prefix PYTHONPATH : $PYTHONPATH
'';
};
# Function which takes a network and the final octet, and returns a container
mkContainer = net: octet: {
privateNetwork = true;
hostAddress = "${net}.1";
localAddress = "${net}.${octet}";</code></pre>
<pre><code> autoStart = true;
config = { config, pkgs, ... }:
{
users.mutableUsers = false;
# Use this only for debugging, login the machine with machinectl
users.extraUsers.root.password = "root";
# Let consul run check scripts
users.extraUsers.consul.shell = "/run/current-system/sw/bin/bash";
environment.etc."consul/prime.json".text = builtins.toJSON {
service = {
name = "prime";
tags = [ "nginx" ];
port = 8080;
check = {
script = "${pkgs.curl}/bin/curl localhost:8080 >/dev/null 2>&1";
interval = "30s";
};
};
};
systemd.services.prime = {
wantedBy = [ "multi-user.target" ];
serviceConfig = {
ExecStart = "${work}/bin/work.py";
};
};
services.consul = {
enable = true;
extraConfig = {
server = false;
start_join = [ "${net}.1" ];
};
extraConfigFiles = [ "/etc/consul/prime.json" ];
};
networking.firewall = {
allowedTCPPorts = [ 8080 8400 ];
allowPing = true;
};
};
};
nginxTmpl = pkgs.writeText "prime.conf" ''
upstream primes {
{{range service "prime"}}
server {{.Address}}:8080;{{end}}
}
'';
in
{
containers.prime1 = mkContainer "10.50.0" "2";
containers.prime2 = mkContainer "10.50.0" "3";
containers.prime3 = mkContainer "10.50.0" "4";
containers.prime4 = mkContainer "10.50.0" "5";
services.consul = {
enable = true;
extraConfig = {
bootstrap = true;
server = true;
};
};
services.nginx = {
enable = true;
httpConfig = ''
include /etc/nginx/prime.conf;
server {
listen 80;
location / {
proxy_pass http://primes;
}
}
'';
};
systemd.services.nginx = {
preStart = ''
mkdir -p /etc/nginx
touch -a /etc/nginx/prime.conf
'';
serviceConfig = {
Restart = "on-failure";
RestartSec = "1s";
};
};
# Start order: consul -> consul-template -> nginx
systemd.services.consul-template = {
wantedBy = [ "nginx.service" ];
before = [ "nginx.service" ];
wants = [ "consul.service" ];
after = [ "consul.service" ];
serviceConfig = {
Restart = "on-failure";
RestartSec = "1s";
ExecStart = "${pkgs.consul-template}/bin/consul-template -template '${nginxTmpl}:/etc/nginx/prime.conf:systemctl kill -s SIGHUP nginx'";
};
};
boot.kernel.sysctl."net.ipv4.ip_forward" = true;
}
</code></pre>
Differences with the original post:<br />
<ul>
<li>We only create 4 containers instead of 10. I was lazy here. If you are lazy too, you can still automatize the process with nix functions (for example <code>map</code>).</li>
<li>We define some ordering in how services start and how they restart with systemd.</li>
<li>For simplicity we include the <code>prime.conf</code> nginx config instead of rewriting the whole nginx config with consul-template.</li>
<li>We create a self-contained package for our python service, so that anywhere it runs the dependencies will be satisfied.</li>
</ul>
Finally import this config in your <code>/etc/nixos/configuration.nix</code> with <code>imports = [ ./prime.nix ];</code>.<br />
<br />
<h3>
Step 3: apply the configuration</h3>
<br />
Type <code>nixos-rebuild switch</code> and then <code>curl http://localhost</code>. You may have to wait some seconds before consul writes the nginx config. In the while, nginx may have failed to start. If it exceeded the StartTime conditions, you can <code>systemctl start nginx</code> manually.<br />
Fixing this is about tweaking the systemd service values about the StartTime.<br />
<br />
Each container consumes practically no disk space at base. Everything else is shared through the host nix store, except logs, consul state, ecc. of course.<br />
<br />
Have fun!Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com20tag:blogger.com,1999:blog-32054652.post-26809958350297097882015-02-08T03:33:00.001-08:002015-02-08T03:33:22.262-08:00Developing in golang with Nix package managerI've been using Go since several months. It's a pleasant language, even though it has its own drawbacks.<br />
<br />
In our <a href="https://github.com/NixOS/nixpkgs">Nixpkgs repository</a> we have support for several programming languages: perl, python, ruby, haskell, lua, ... We've <a href="https://github.com/NixOS/nixpkgs/pull/6119">merged</a> a better support for Go.<br />
<br />
What kind of support are we talking about? In <a href="http://nixos.org/nix/">Nix</a>, you <b>never install libraries</b>. Instead, you define an environment in which to use a certain library compiled for a certain version of the language. The library will be available only within this environment.<br />
<br />
Think of it like virtualenv for python, except for any language, and also being able to mix them.<br />
On the other hand Nix requires the src url and the checksum of every dependency of your project. So before starting, make sure you are willing to write nix packages that are not currently present in nixpkgs.<br />
<br />
Also you probably have to wait a couple of days before this PR will be available in the unstable channel, at the time of this writing (otherwise <code>git clone https://github.com/NixOS/nixpkgs.git</code>).<br />
<br />
<h2>
Using nix-shell -p</h2>
First a quick example using <code>nix-shell -p</code> for your own project:<br />
<pre><code>$ nix-shell '<nixpkgs>' -p goPackages.go goPackages.net goPackages.osext
[nix-shell]$ echo $GOPATH
/nix/store/kw9dryid364ac038zmbzq72bnh3zsinz-go-1.4.1-go.net-3338d5f109e9/share/go:...
</code></pre>
That's how nix mostly works, it's as simple as that. The <code>GOPATH</code> is set for every package that provides a directory <code>share/go</code>.<br />
<br />
What is <code>goPackages</code>? Currently it's <code>go14Packages</code>, which is all the go packages we have, compiled with go 1.4. There's also <code>go13Packages</code>, you know some particular packages don't work with go 1.4 yet.<br />
<br />
<h2 id="writing-a-nix-file">
Writing a nix file</h2>
A more structured example by writing a <code>default.nix</code> file in your project:<br />
<pre><code>with import <nixpkgs> {}; with goPackages;
buildGoPackage rec {
name = "yourproject";
buildInputs = [ net osext ];
goPackagePath = "github.com/you/yourproject";
}
</code></pre>
Then you can just run <code>nix-shell</code> in your project directory and have your dev environment ready to compile your code.<br />
The <code>goPackagePath</code> is something needed by <code>buildGoPackage</code>, in case you are going to run <code>nix-build</code>. Ignore it for now.<br />
<br />
<h2 id="writing-a-dependency">
Writing a dependency</h2>
But nixpkgs doesn't have listed all the possible go projects. What if you need to use a particular library?<br />
Let's take for example <code>github.com/kr/pty</code>. Write something like this in a pty.nix file:<br />
<pre><code>{ goPackages, fetchFromGitHub }:
goPackages.buildGoPackage rec {
rev = "67e2db24c831afa6c64fc17b4a143390674365ef";
name = "pty-${rev}";
goPackagePath = "github.com/kr/pty";
src = fetchFromGitHub {
inherit rev;
owner = "kr";
repo = "pty";
sha256 = "1l3z3wbb112ar9br44m8g838z0pq2gfxcp5s3ka0xvm1hjvanw2d";
};
}
</code></pre>
Then in your default.nix:<br />
<pre><code>with import <nixpkgs> {}; with goPackages;
let
pty = callPackage ./pty.nix {};
in
buildGoPackage rec {
name = "yourproject";
buildInputs = [ net osext pty ];
goPackagePath = "github.com/you/yourproject";
}
</code></pre>
Type <code>nix-shell</code> and now you will also have pty in your dev environment.<br />
So as you can see, for each go package nix requires a name, the go path, where to fetch the sources, and the hash.<br />
<br />
<em>You may be wondering how do you get the sha256: a dirty trick is to write a wrong sha, then nix will tell you the correct sha.</em><br />
<em><br /></em>
<h2 id="conclusion-and-references">
Conclusion and references</h2>
Nix looks a little complex and boring due to writing a package for each dependency. On the other hand you get for free:<br />
<ul>
<li>Exact build and runtime dependencies</li>
<li>Sharing build and runtime dependencies between multiple projects</li>
<li>Easily test newer or older versions of libraries, without messing with system-wide installations</li>
<li>Mix with other programming languages, using a similar approach</li>
<li>Packages using C libraries don't need to be compiled manually by you: define the nix package once, reuse everywhere</li>
</ul>
For installing nix, follow <a href="http://nixos.org/nix/manual/#chap-quick-start">the manual</a>. Make sure you read the entire document to learn the nix syntax.<br />
<br />
For more examples on how to write dependencies, you can look at nixpkgs <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/go-packages.nix">goPackages itself</a>.<br />
<br />
Drop by #nixos on <a href="http://webchat.freenode.net/">irc.freenode.net</a> for any doubts.<br />
<br />
<br />Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com29tag:blogger.com,1999:blog-32054652.post-19849641410354461732015-01-12T04:48:00.003-08:002015-08-24T09:53:41.269-07:00Nix pill 18: nix store paths<h3>
<div style="font-size: medium; font-weight: normal;">
Welcome to the 18th Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/11/nix-pill-17-nixpkgs-overriding-packages.html">17th pill</a> we have scratched the surface of the nixpkgs repository structure. It is a set of packages, and it's possible to override such packages so that all other packages will use the overrides.</div>
<div style="font-size: medium; font-weight: normal;">
<br /></div>
<div style="font-size: medium; font-weight: normal;">
Before reading existing derivations, I'd like to talk about store paths and how they are computed. In particular we are interested in fixed store paths that depend on an integrity hash (e.g. a sha256), which is usually applied to source tarballs.</div>
<div style="font-size: medium; font-weight: normal;">
<br /></div>
<div style="font-size: medium; font-weight: normal;">
The way store paths are computed is a little contrived, mostly due to historical reasons. Our reference will be the <a href="http://lxr.devzen.net/source/xref/nix/src/libstore/store-api.cc#97">Nix source code</a>.</div>
<div style="font-size: medium; font-weight: normal;">
<br /></div>
</h3>
<h3>
Source paths</h3>
<br />
Let's start simple. You know nix allows relative paths to be used, such that the file or directory is stored in the nix store, that is <code>./myfile</code> gets stored into <code>/nix/store/......</code>. We want to understand how is the store path generated for such a file:<br />
<pre><code>$ echo mycontent > myfile
</code></pre>
I remind you, the simplest derivation you can write has a name, a builder and the system:<br />
<pre><code>$ nix-repl
nix-repl> derivation { system = "x86_64-linux"; builder = ./myfile; name = "foo"; }
«derivation /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv»
</code></pre>
Now inspect the .drv to see where is <code>./myfile</code> being stored:<br />
<pre><code>$ pp-aterm -i /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv
Derive(
[("out", "/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo", "", "")]
, []
, ["/nix/store/xv2iccirbrvklck36f1g7vldn5v58vck-myfile"]
, "x86_64-linux"
...
</code></pre>
Great, how did nix decide to use <code>xv2iccirbrvklck36f1g7vldn5v58vck</code> ? Keep looking at the <a href="http://lxr.devzen.net/source/xref/nix/src/libstore/store-api.cc#97">nix comments</a>.<br />
<br />
<strong>Note</strong>: doing <code>nix-store --add myfile</code> will store the file in the same store path.<br />
<br />
<h4>
Step 1, compute the hash of the file</h4>
<br />
The comments tell us to first compute the sha256 of the <strong><em>NAR serialization</em></strong> of the file. Can be done in two ways:<br />
<pre><code>$ nix-hash --type sha256 myfile
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3</code></pre>
Or:
<br />
<pre><code>$ nix-store --dump myfile|sha256sum
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3 -
</code></pre>
In general, Nix understands two contents: flat for regular files, or recursive for NAR serializations which can be anything.<br />
<br />
<h4>
Step 2, build the string description</h4>
<br />
Then nix uses a special string which includes the hash, the path type and the file name. We store this in another file:<br />
<pre><code>$ echo -n "source:sha256:2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3:/nix/store:myfile" > myfile.str</code></pre>
<h4>
Step 3, compute the final hash</h4>
<br />
Finally the comments tell us to compute the base-32 representation of the first 160 bits (truncation) of a sha256 of the above string:<br />
<pre><code>$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
xv2iccirbrvklck36f1g7vldn5v58vck</code></pre>
<h3>
Output paths</h3>
<br />
Output paths are usually generated for derivations. We use the above example because it's simple. Even if we didn't build the derivation, nix knows the out path <code>hs0yi5n5nw6micqhy8l1igkbhqdkzqa1</code>. This is because the out path only depends on inputs.<br />
<br />
It's computed in a similar way to source paths, except that the .drv is hashed and the type of derivation is <code>output:out</code>. In case of multiple outputs, we may have different <code>output:<id></code>.<br />
<br />
At the time nix computes the out path, the .drv contains an empty string for each out path. So what we do is getting our .drv and replacing the out path with an empty string:<br />
<pre><code>$ cp -f /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv myout.drv
$ sed -i 's,/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo,,g' myout.drv
</code></pre>
The <code>myout.drv</code> is the .drv state in which nix is when computing the out path for our derivation:<br />
<pre><code>$ sha256sum myout.drv
1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5 myout.drv
$ echo -n "output:out:sha256:1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5:/nix/store:foo" > myout.str
$ nix-hash --type sha256 --truncate --base32 --flat myout.str
hs0yi5n5nw6micqhy8l1igkbhqdkzqa1
</code></pre>
Then nix puts that out path in the .drv, and that's it.<br />
<br />
In case the .drv has input derivations, that is it references other .drv, then such .drv paths are replaced by this same algorithm which returns an hash.<br />
<br />
In other words, you get a final .drv where every other .drv path is replaced by its hash.<br />
<br />
<h3>
Fixed-output paths</h3>
<br />
Finally, the other most used kind of path is when we know beforehand an integrity hash of a file. This is usual for tarballs.<br />
<br />
A derivation can take three special attributes: <code>outputHashMode</code>, <code>outputHash</code> and <code>outputHashAlgo</code> which are well documented in the <a href="http://nixos.org/nix/manual/#ssec-derivation">nix manual</a>.<br />
<br />
The builder must create the out path and make sure its hash is the same as the one declared with <code>outputHash</code>.<br />
<br />
Let's say our builder should create a file whose contents is <code>mycontent</code>:<br />
<pre><code>$ echo mycontent > myfile
$ sha256sum myfile
f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb myfile
nix-repl> derivation { name = "bar"; system = "x86_64-linux"; builder = "none"; outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb"; }
«derivation /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv»
</code></pre>
Inspect the .drv and see that it also stored the fact that it's a fixed-output derivation with sha256 algorithm, compared to the previous examples:<br />
<pre><code>$ pp-aterm -i /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv
Derive(
[("out", "/nix/store/a00d5f71k0vp5a6klkls0mvr1f7sx6ch-bar", "sha256", "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb")]
...
</code></pre>
It doesn't matter which input derivations are being used, the final out path must only depend on the declared hash.<br />
What nix does is to create an intermediate string representation of the fixed-output content:<br />
<pre><code>$ echo -n "fixed:out:sha256:f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb:" > mycontent.str
$ sha256sum mycontent.str
423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639 myfile.str
</code></pre>
Then proceed as it was a normal derivation output path:<br />
<pre><code>$ echo -n "output:out:sha256:423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639:/nix/store:bar" > myfile.str
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
a00d5f71k0vp5a6klkls0mvr1f7sx6ch
</code></pre>
Hence, the store path only depends on the declared fixed-output hash.<br />
<br />
<h3>
Conclusion</h3>
<h3>
<div style="font-size: medium; font-weight: normal;">
<br /></div>
<div style="font-size: medium; font-weight: normal;">
There are other types of store paths, but you get the idea. Nix first hashes the contents, then creates a string description, and the final store path is the hash of this string.<br />
<br />
Also we've introduced some fundamentals, in particular the fact that Nix knows beforehand the out path of a derivation since it only depends on the inputs. We've also introduced fixed-output derivations which are especially used by the nixpkgs repository for downloading and verifying source tarballs.</div>
<div style="font-size: medium; font-weight: normal;">
<br /></div>
</h3>
<h3>
Next pill</h3>
<div style="font-size: medium; font-weight: normal;">
<br /></div>
<div style="font-size: medium; font-weight: normal;">
...we will introduce stdenv. In the previous pills we rolled our own mkDerivation convenience function for wrapping the builtin derivation, but the nixpkgs repository also has its own convenience functions for dealing with autotools projects and other build systems.</div>
<div style="font-size: medium; font-weight: normal;">
<br /></div>
<div style="font-size: medium; font-weight: normal;">
Pill 19 is available for
<a href="http://lethalman.blogspot.it/2015/08/nix-pill-19-fundamentals-of-stdenv.html" style="background-color: white; color: #4d469c; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; line-height: 12.1333341598511px; text-decoration: none;">reading here</a>.<br />
<br />
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.
</div>Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com20tag:blogger.com,1999:blog-32054652.post-2867578332826992402014-11-10T02:00:00.000-08:002015-01-12T04:48:52.579-08:00Nix pill 17: nixpkgs, overriding packagesWelcome to the 17th Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/11/nix-pill-16-nixpkgs-parameters.html">16th pill</a> we have started to dive into the <a href="http://github.com/NixOS/nixpkgs">nixpkgs repository</a>. Nixpkgs is a function, and we've looked at some parameters like <code>system</code> and <code>config</code>.<br />
<br />
Today we'll talk about a special attribute: <code>config.packageOverrides</code>. Overriding packages in a set with fixed point can be considered another design pattern in nixpkgs.<br />
<br />
<h3>
Overriding a package</h3>
<br />
I recall the override design pattern from the <a href="http://lethalman.blogspot.it/2014/09/nix-pill-14-override-design-pattern.html">nix pill 14</a>. Instad of calling a function with parameters directly, we make the call (function + parameters) overridable.<br />
We put the override function in the returned attribute set of the original function call.<br />
<br />
Take for example <code>graphviz</code>. It has an input parameter <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/graphics/graphviz/default.nix#L2"><em>xlibs</em></a>. If it's null, then graphviz will build <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/graphics/graphviz/default.nix#L17">without X support</a>.<br />
<pre><code>$ nix-repl
nix-repl> :l <nixpkgs>
Added 4360 variables.
nix-repl> :b graphviz.override { xlibs = null; }
</code></pre>
This will build graphviz without X support, it's as simple as that.<br />
<br />
However let's say a package P depends on graphviz, how do we make P depend on the new graphviz without X support?<br />
<br />
<h3>
In an imperative world...</h3>
<br />
...you could do something like this:<br />
<pre><code>pkgs = import <nixpkgs> {};
pkgs.graphviz = pkgs.graphviz.override { xlibs = null; };
build(pkgs.P)
</code></pre>
Given pkgs.P depends on pkgs.graphviz, it's easy to build P with the replaced graphviz.
On a pure functional language it's not that easy because you can assign to variables only once.<br />
<br />
<h3>
Fixed point</h3>
<br />
The fixed point with lazy evaluation is crippling but about necessary in a language like Nix. It lets us achieve something similar to what we'd do imperatively.<br />
Follows the definition of fixed point in <a href="https://github.com/NixOS/nixpkgs/blob/master/lib/trivial.nix#L21">nixpkgs</a>:<br />
<pre><code># Take a function and evaluate it with its own returned value.
fix = f: let result = f result; in result;
</code></pre>
It's a function that accepts a function <code>f</code>, calls <code>f result</code> on the result just returned by <code>f result</code> and returns it.
In other words it's <code>f(f(f(...</code>.<br />
At first sight, it's an infinite loop. With lazy evaluation it isn't, because the call is done only when needed.<br />
<pre><code>nix-repl> fix = f: let result = f result; in result
nix-repl> pkgs = self: { a = 3; b = 4; c = self.a+self.b; }
nix-repl> fix pkgs
{ a = 3; b = 4; c = 7; }
</code></pre>
Without the <code>rec</code> keyword, we were able to refer to <code>a</code> and <code>b</code> of the same set.<br />
<ol>
<li>First pkgs gets called with an unevaluated thunk (<code>pkgs(pkgs(...</code>)</li>
<li>To set the value of <code>c</code> then <code>self.a</code> and <code>self.b</code> are evaluated.</li>
<li>The pkgs function gets called again to get the value of <code>a</code> and <code>b</code>.</li>
</ol>
The trick is that <code>c</code> is not needed to be evaluated in the inner call, thus it doesn't go in an infinite loop.<br />
<br />
Won't go further with the explanation here. A good post about fixed point and Nix can be <a href="http://r6.ca/blog/20140422T142911Z.html">found here</a>.<br />
<br />
<h4>
Overriding a set with fixed point</h4>
<br />
Given that <code>self.a</code> and <code>self.b</code> refer to the <em>passed</em> set and not to the <em>literal</em> set in the function, we're able to override both a and b and get a new value for c:<br />
<pre><code>nix-repl> overrides = { a = 1; b = 2; }
nix-repl> let newpkgs = pkgs (newpkgs // overrides); in newpkgs
{ a = 3; b = 4; c = 3; }
nix-repl> let newpkgs = pkgs (newpkgs // overrides); in newpkgs // overrides
{ a = 1; b = 2; c = 3; }
</code></pre>
In the first case we computed <code>pkgs</code> with the overrides, in the second case we also included the overriden attributes in the result.<br />
<br />
<h3>
Overriding nixpkgs packages</h3>
<br />
We've seen how to override attributes in a set such that they get recursively picked by dependant attributes.
This approach can be used for derivations too, after all nixpkgs is a giant set of attributes that depend on each other.<br />
<br />
To do this, nixpkgs offers <code>config.packageOverrides</code>. So nixpkgs returns a fixed point of the package set, and <code>packageOverrides</code> is used to inject the overrides.<br />
<br />
Create a <code>config.nix</code> file like this somewhere:<br />
<pre><code>{
packageOverrides = pkgs: {
graphviz = pkgs.graphviz.override { xlibs = null; };
};
}
</code></pre>
Now we can build e.g. <code>asciidocFull</code> and it will automatically use the overridden graphviz:<br />
<pre><code>nix-repl> pkgs = import <nixpkgs> { config = import ./config.nix; }
nix-repl> :b pkgs.asciidocFull
</code></pre>
Note how we pass the config with <code>packageOverrides</code> when importing nixpkgs. Then <code>pkgs.asciidocFull</code> is a derivation that has graphviz input (<code>pkgs.asciidoc</code> is the lighter version and doesn't use graphviz at all).<br />
<br />
Since there's no version of asciidoc with graphviz without X support in the binary cache, Nix will recompile the needed stuff for you.<br />
<br />
<h3>
The ~/.nixpkgs/config.nix file</h3>
<br />
In the previous pill we already talked about this file. The above <code>config.nix</code> that we just wrote could be the content of <code>~/.nixpkgs/config.nix</code>.<br />
<br />
Instead of passing it explicitly whenever we import nixpkgs, it will be automatically <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix#L54">imported by nixpkgs</a>.<br />
<br />
<h3>
Conclusion</h3>
<br />
We've learned about a new design pattern: using fixed point for overriding packages in a package set.<br />
<br />
Whereas in an imperative setting, like with other package managers, a library is installed replacing the old version and applications will use it, in Nix it's not that straight and simple. But it's more precise.<br />
<br />
Nix applications will depend on specific versions of libraries, hence the reason why we have to recompile asciidoc to use the new graphviz library.<br />
<br />
The newly built asciidoc will depend on the new graphviz, and old asciidoc will keep using the old graphviz undisturbed.<br />
<br />
<h3>
Next pill</h3>
<br />
...we will stop diving nixpkgs for a moment and talk about store paths. How does Nix compute the path in the store where to place the result of builds? How to add files to the store for which we have an integrity hash?<br />
<br />
Pill 18 is available for <a href="http://lethalman.blogspot.it/2015/01/nix-pill-18-nix-store-paths.html">reading here</a>.<br />
<br />
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com20tag:blogger.com,1999:blog-32054652.post-46642293702179922282014-11-04T03:36:00.001-08:002014-11-10T02:00:55.452-08:00Nix pill 16: nixpkgs, the parameters<div>
<div>
<div style="margin: 0px;">
Welcome to the 16th Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/09/nix-pill-15-nix-search-paths.html">15th pill</a> we've realized how nix finds expressions with the angular brackets syntax, so that we finally know where is <span style="font-family: Courier New, Courier, monospace;"><nixpkgs></span> located on our system.</div>
</div>
<div>
<div style="margin: 0px;">
<br />
We can start diving into the <a href="https://github.com/NixOS/nixpkgs">nixpkgs repository</a>, through all the various tools and design patterns. Please note that also nixpkgs has its own <a href="http://nixos.org/nixpkgs/manual/">manual</a>, underlying the difference between the general "nix" language and the "nixpkgs" repository.<br />
<br /></div>
</div>
</div>
<h3>
The default.nix expression</h3>
<div>
<div style="margin: 0px;">
<br />
We will not start inspecting packages at the beginning, rather the general structure of nixpkgs.<br />
In our custom repository we created a default.nix which composed the expressions of the various packages.<br />
Also nixpkgs has its own <a href="https://github.com/NixOS/nixpkgs/blob/master/default.nix">default.nix</a>, which is the one being loaded when referring to <span style="font-family: Courier New, Courier, monospace;"><nixpkgs></span>. It does a simple thing: check whether the nix version is at least 1.7 (at the time of writing this blog post). Then import <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix">pkgs/top-level/all-packages.nix</a>. From now on, we will refer to this set of packages as <b>pkgs</b>.<br />
<br />
The all-packages.nix is then the file that composes all the packages. Note the <span style="font-family: Courier New, Courier, monospace;">pkgs/</span> subdirectory, while nixos is in the <span style="font-family: Courier New, Courier, monospace;">nixos/</span> subdirectory.<br />
<br />
The all-packages.nix is a bit contrived. First of all, it's a function. It accepts a couple of interesting parameters:<br />
<ul>
<li>system: defaults to the current system</li>
<li>config: defaults to null</li>
<li>others...</li>
</ul>
<div>
The <b>system</b> parameter, as per comment in the expression, it's the system for which the packages will be built. It allows for example to install i686 packages on amd64 machines.</div>
<div>
<br /></div>
<div>
The <b>config</b> parameter is a simple attribute set. Packages can read some of its values and change the behavior of some derivations.</div>
</div>
</div>
<div>
<br />
<h3>
The system parameter</h3>
</div>
<div>
<br /></div>
You will find this parameter in many other .nix expressions (e.g. release expressions). The reason is that, given pkgs accepts a system parameter, then whenever you want to import pkgs you also want to pass through the value of system. E.g.:<br />
<br />
myrelease.nix:<br />
<pre>{ system ? builtins.currentSystem }:
let pkgs = import <nixpkgs> { inherit system; };
...
</pre>
Why is it useful? With this parameter it's very easy to select a set of packages for a particular system. For example:
<br />
<pre>nix-build -A psmisc --argstr system i686-linux
</pre>
This will build the psmisc derivation for i686-linux instead of x86_64-linux. This concept is very similar to multi-arch of Debian.<br />
<br />
The setup for cross compiling is also in nixpkgs, however it's a little contrived to talk about it and I don't know much of it either.<br />
<br />
<h3>
The config parameter</h3>
<div>
<br /></div>
<div>
I'm sure on the wiki or other manuals you've read about <span style="font-family: Courier New, Courier, monospace;">~/.nixpkgs/config.nix</span> and I'm sure you've wondered whether that's hardcoded in nix. It's not, <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix#L45">it's in nixpkgs</a>.</div>
<div>
<br /></div>
<div>
The all-packages.nix expression accepts the <b>config</b> parameter. If it's null, then it reads the <span style="font-family: Courier New, Courier, monospace;">NIXPKGS_CONFIG</span> environment variable. If not specified, nixpkgs will peek <span style="font-family: Courier New, Courier, monospace;">$HOME/.nixpkgs/config.nix</span> . </div>
<div>
<br /></div>
<div>
After determining config.nix, it will be imported as nix expression, and that will be the value of <b>config</b> (in case it hasn't been passed as parameter to <span style="font-family: Courier New, Courier, monospace;">import <nixpkgs></span>).</div>
<div>
<br /></div>
<div>
The config is available in the resulting repository:</div>
<pre>$ nix-repl
nix-repl> pkgs = import <nixpkgs> {}
nix-repl> pkgs.config
{ }
nix-repl> pkgs = import <nixpkgs> { config = { foo = "bar"; }; }
nix-repl> pkgs.config
{ foo = "bar"; }
</pre>
What attributes go in config is a matter of convenience and conventions.<br />
<br />
For example, <span style="font-family: Courier New, Courier, monospace;">config.allowUnfree</span> is an attribute that forbids building packages that have an unfree license by default. The <span style="font-family: Courier New, Courier, monospace;">config.pulseaudio</span><span style="font-family: inherit;"> setting tells whether to build packages with pulseaudio support or not where </span>applicable and when the derivation obeys to the setting.<br />
<br />
<h3>
About .nix functions</h3>
<br />
A .nix file contains a nix expression. Thus it can also be a function.<br />
I remind you that nix-build expects the expression to return a derivation. Therefore it's natural to return straight a derivation from a .nix file.<br />
However, it's also very natural for the .nix file to accept some parameters, in order to tweak the derivation being returned.<br />
<br />
In this case, nix does a trick:<br />
<ul>
<li>If the expression is a derivation, well build it.</li>
<li>If the expression is a function, call it and build the resulting derivation.</li>
</ul>
<div>
For example you can nix-build the .nix file below:</div>
<pre>{ pkgs ? import <nixpkgs> {} }:
pkgs.psmisc
</pre>
Nix is able to call the function because the pkgs parameter has a default value. This allows you to pass a different value for pkgs using the <span style="font-family: Courier New, Courier, monospace;">--arg</span> option.<br />
<br />
Does it work if you have a function returning a function that returns a derivation? No, Nix only calls the function it encounters once.<br />
<br />
<div>
<h3>
Conclusion</h3>
<div>
</div>
<div>
<br />
We've unleashed the <span style="font-family: Courier New, Courier, monospace;"><nixpkgs></span> repository. It's a function that accepts some parameters, and returns the set of all packages. Due to laziness, only the accessed derivations will be built.<br />
You can use this repository to build your own packages as we've seen in the previous pill when creating our own repository.<br />
<br />
Lately I'm a little busy with the NixOS 14.11 release and other stuff, and I'm also looking toward migrating from blogger to a more coder-oriented blogging platform. So sorry for the delayed and shorter pills :)<br />
<br /></div>
</div>
<h3>
Next pill</h3>
<div>
<br />
...we will talk about overriding packages in the nixpkgs repository. What if you want to change some options of a library and let all other packages pick the new library? One possibility is to use, like described above, the config parameter when applicable. The other possibility is to override derivations.<br />
<br />
Pill 17 is available for <a href="http://lethalman.blogspot.it/2014/11/nix-pill-17-nixpkgs-overriding-packages.html">reading here</a>.<br />
<br /></div>
<div>
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com4tag:blogger.com,1999:blog-32054652.post-90627115629933178122014-09-16T02:28:00.001-07:002014-12-21T04:57:01.055-08:00Nix pill 15: nix search paths<div>
<div>
<div style="margin: 0px;">
Welcome to the 15th Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/09/nix-pill-14-override-design-pattern.html">14th pill</a> we have introduced the "override" pattern, useful for writing variants of derivations by passing different inputs.</div>
</div>
<div>
<div style="margin: 0px;">
<br />
Assuming you followed the previous posts, I hope you are now ready to understand nixpkgs. But we have to find nixpkgs in our system first! So this is the step: introducing some options and environment variables used by nix tools.<br />
<br /></div>
</div>
</div>
<h3>
The NIX_PATH</h3>
<div>
<div style="margin: 0px;">
<br />
The <a href="http://nixos.org/nix/manual/#sec-common-env">NIX_PATH environment variable</a> is very important. It's very similar to the PATH environment variable. The syntax is similar, several paths are separated by a colon ":". Nix will then search for something in those paths from left to right.<br />
<br />
Who uses NIX_PATH? The nix expressions! Yes, NIX_PATH is not of much use by the nix tools themselves, rather it's used when writing nix expressions.<br />
<br />
In the shell for example, when you execute the command "ping", it's being searched in the PATH directories. The first one found is the one being used.<br />
<br />
In nix it's exactly the same, however the syntax is different. Instead of just typing "ping" you have to type <ping>. Yes, I know... you are already thinking of <nixpkgs>.<br />
However don't stop reading here, let's keep going.<br />
<br />
What's NIX_PATH good for? Nix expressions may refer to an "abstract" path such as <nixpkgs>, and it's possible to override it from the command line.<br />
<br />
For ease we will use <span style="font-family: Courier New, Courier, monospace;">nix-instantiate --eval</span> to do our tests. I remind you, <a href="http://nixos.org/nix/manual/#sec-nix-instantiate">nix-instantiate</a> is used to evaluate nix expressions and generate the .drv files. Here we are not interested in building derivations, so evaluation is enough. It can be used for one-shot expressions.<br />
<br />
<h3>
Fake it a little</h3>
<br />
It's useless from a nix view point, but I think it's useful for your own understanding. Let's use PATH itself as NIX_PATH, and try to locate ping (or another binary if you don't have it).<br />
<pre>$ nix-instantiate --eval -E '<ping>'
error: file `ping' was not found in the Nix search path (add it using $NIX_PATH or -I)
$ NIX_PATH=$PATH nix-instantiate --eval -E '<ping>'
/bin/ping
$ nix-instantiate -I /bin --eval -E '<ping>'
/bin/ping
</pre>
</div>
</div>
Great. At first attempt nix obviously said <span style="font-family: Courier New, Courier, monospace;"><ping></ping></span><span style="font-family: inherit;"> could not be found anywhere in the search path. Note that the -I option accepts a single directory. Paths added with -I take precedence over NIX_PATH.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">The NIX_PATH also accepts a different yet very handy syntax: "</span><span style="font-family: Courier New, Courier, monospace;">somename=somepath</span><span style="font-family: inherit;">". That is, instead of searching inside a directory for a name, we specify exactly the value of that name.</span><br />
<pre>$ NIX_PATH="ping=/bin/ping" nix-instantiate --eval -E '<ping>'
/bin/ping
$ NIX_PATH="ping=/bin/foo" nix-instantiate --eval -E '<ping>'
error: file `ping' was not found in the Nix search path (add it using $NIX_PATH or -I)
</pre>
Note in the second case how Nix checks whether the path exists or not.<br />
<br />
<h3>
The path to repository</h3>
<br />
You are out of curiosity, right?<br />
<pre>$ nix-instantiate --eval -E '<nixpkgs>'
/home/nix/.nix-defexpr/channels/nixpkgs
$ echo $NIX_PATH
nixpkgs=/home/nix/.nix-defexpr/channels/nixpkgs
</pre>
You may have a different path, depending on how you added channels etc.. Anyway that's the whole point. The <nixpkgs> stranger that we used in our nix expressions, is referring to a path in the filesystem specified by NIX_PATH.<br />
You can list that directory and realize it's simply a checkout of the <a href="https://github.com/NixOS/nixpkgs">nixpkgs</a> repository at a specific commit (hint: <span style="font-family: Courier New, Courier, monospace;">.version-suffix</span>).<br />
The NIX_PATH variable is exported by <span style="font-family: Courier New, Courier, monospace;">nix.sh</span>, and that's the reason why I always asked you to <a href="http://nixos.org/nix/manual/#idm47361539530016">source nix.sh</a> at the beginning of my posts.<br />
<br />
You may wonder: then I can also specify a different nixpkgs path to, e.g., a git checkout of <a href="https://github.com/NixOS/nixpkgs">nixpkgs</a>? Yes, you can and I encourage doing that. We'll talk about this in the next pill.<br />
<br />
Let's define a path for our repository, then! Let's say all the <span style="font-family: Courier New, Courier, monospace;">default.nix</span>, <span style="font-family: Courier New, Courier, monospace;">graphviz.nix</span> etc. are under <span style="font-family: Courier New, Courier, monospace;">/home/nix/mypkgs</span>:<br />
<pre>$ export NIX_PATH=mypkgs=/home/nix/mypkgs:$NIX_PATH
$ nix-instantiate --eval '<mypkgs>'
{ graphviz = <code>; graphvizCore = <code>; hello = <code>; mkDerivation = <code>; }
</pre>
As expected we got the set of our packages (well except the <span style="font-family: Courier New, Courier, monospace;">mkDerivation</span> utility), that's our repository.<br />
<br />
Until now we used <a href="http://nixos.org/nix/manual/#sec-nix-build">nix-build</a> directly in the directory of default.nix. However nix-build generally needs a .nix to be specified to the command line:<br />
<pre>$ nix-build /home/nix/mypkgs -A graphviz
/nix/store/dhps2c1fnsfxssdri3yxbdiqmnhzg6yr-graphviz
$ nix-build '<mypkgs>' -A graphviz
/nix/store/dhps2c1fnsfxssdri3yxbdiqmnhzg6yr-graphviz
</pre>
Yes, nix-build also accepts paths with angular brackets. We first evaluate the whole repository (default.nix) and then peek the graphviz attribute.<br />
<br />
<h3>
A big word about nix-env</h3>
<br />
The <a href="http://nixos.org/nix/manual/#sec-nix-env">nix-env</a> command is a little different than nix-instantiate and nix-build. Whereas nix-instantiate and nix-build require a starting nix expression, nix-env does not.<br />
<br />
You may be crippled by this concept at the beginning, you may think nix-env uses NIX_PATH to find the nixpkgs repository. But that's not it.<br />
<br />
The nix-env command uses ~/.nix-defexpr, which is also part of NIX_PATH by default, but that's only a coincidence. If you empty NIX_PATH, nix-env will still be able to find derivations because of ~/.nix-defexpr.<br />
<br />
So if you run <span style="font-family: Courier New, Courier, monospace;">nix-env -i graphviz</span> inside your repository, it will install the nixpkgs one. Same if you set NIX_PATH to point to your repository.<br />
<br />
In order to specify an alternative to <span style="font-family: Courier New, Courier, monospace;">~/.nix-defexpr</span> it's possible to use the -f option:<br />
<pre>$ nix-env -f '<mypkgs>' -i graphviz
warning: there are multiple derivations named `graphviz'; using the first one
replacing old `graphviz'
installing `graphviz'
</pre>
Oh why did it say there's another derivation named graphviz? Because both <span style="font-family: Courier New, Courier, monospace;">graphviz</span> and <span style="font-family: Courier New, Courier, monospace;">graphvizCore</span> attributes in our repository have the name "graphviz" for the derivation:<br />
<pre>$ nix-env -f '<mypkgs>' -qaP
graphviz graphviz
graphvizCore graphviz
hello hello
</pre>
By default nix-env parses all derivations and use the derivation names to interpret the command line. So in this case "graphviz" matched two derivations. Alternatively, like for nix-build, one can use -A to specify an attribute name instead of a derivation name:<br />
<pre>$ nix-env -f '<mypkgs>' -i -A graphviz
replacing old `graphviz'
installing `graphviz'
</pre>
This form, other than being more precise, it's also faster because nix-env does not have to parse all the derivations.<br />
<br />
For completeness: you may install graphvizCore with -A, since without the -A switch it's ambiguous.<br />
<br />
In summary, it may happen when playing with nix that nix-env<b> peeks a different derivation than nix-build.</b> In such case you probably specified NIX_PATH, but nix-env is instead looking into ~/.nix-defexpr.<br />
<br />
Why is nix-env having this different behavior? I don't know specifically by myself either, but the answers could be:
<br />
<ul>
<li>nix-env tries to be generic, thus it does not look for "nixpkgs" in NIX_PATH, rather it looks in ~/.nix-defexpr.</li>
<li>nix-env is able to merge multiple trees in ~/.nix-defexpr by looking at all the possible derivations</li>
</ul>
<div>
It may also happen to you that you <b>cannot</b> <b>match a derivation name when installing</b>, because of the derivation name vs -A switch described above. Maybe nix-env wanted to be more friendly in this case for default user setups.<br />
<br />
It may or may not make sense for you, or it's like that for historical reasons, but that's how it works currently, unless somebody comes up with a better idea.<br />
<br /></div>
<div>
<h3>
Conclusion</h3>
<div>
</div>
<div>
<br />
The NIX_PATH variable is the search path used by nix when using the angular brackets syntax. It's possible to refer to "abstract" paths inside nix expressions and define the "concrete" path by means of NIX_PATH, or the usual -I flag in nix tools.<br />
<br />
We've also explained some of the uncommon nix-env behaviors for newcomers. The nix-env tool does not use NIX_PATH to search for packages, but rather for ~/.nix-defexpr. Beware of that!<br />
<br />
In general do not abuse NIX_PATH, when possible use relative paths when writing your own nix expressions. Of course, in the case of <nixpkgs> in our repository, that's a perfectly fine usage of NIX_PATH. Instead, inside our repository itself, refer to expressions with relative paths like <span style="font-family: Courier New, Courier, monospace;">./hello.nix</span>.</div>
</div>
<br />
<br />
<h3>
Next pill</h3>
<div>
<br />
...we will finally dive into nixpkgs. Most of the techniques we have developed in this series are already in nixpkgs, like mkDerivation, callPackage, override, etc., but of course better. With time, those base utilities get enhanced by the community with more features in order to handle more and more use cases and in a more general way.<br />
<br />
Pill 16 is available for <a href="http://lethalman.blogspot.it/2014/11/nix-pill-16-nixpkgs-parameters.html">reading here</a>.<br />
<br /></div>
<div>
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com8tag:blogger.com,1999:blog-32054652.post-20442382491508360252014-09-10T04:38:00.000-07:002014-09-16T02:28:45.286-07:00Nix pill 14: the override design pattern<div>
<div>
<div style="margin: 0px;">
Welcome to the 14th Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/09/nix-pill-13-callpackage-design-pattern.html">13th pill</a> we have introduced the callPackage pattern, used to simplify the composition of software in a repository.</div>
</div>
<div>
<div style="margin: 0px;">
<br />
The next design pattern is less necessary but useful in many cases and it's a good exercise to learn more about Nix.<br />
<br /></div>
</div>
</div>
<h3>
About composability</h3>
<div>
<div style="margin: 0px;">
<br />
Functional languages are known for being able to compose functions. In particular, you gain a lot from functions that are able to manipulate the original value into a new value having the same structure. So that in the end we're able to call multiple functions to have the desired modifications.<br />
<br />
In Nix we mostly talk about <b>functions</b> that accept inputs in order to return <b>derivations</b>. In our world we want nice utility functions that are able to manipulate those structures. These utilities add some useful properties to the original value, and we must be able to apply more utilities on top of it.<br />
<br />
For example let's say we have an initial derivation drv and we want it to be a drv with debugging information and also to apply some custom patches:<br />
<pre>debugVersion (applyPatches [ ./patch1.patch ./patch2.patch ] drv)
</pre>
The final result will be still the original derivation plus some changes. That's both interesting and very different from other packaging approaches, which is a consequence of using a functional language to describe packages.<br />
<br />
Designing such utilities is not trivial in a functional language that is not statically typed, because understanding what can or cannot be composed is difficult. But we try to do the best.<br />
<br />
<h3>
The override pattern</h3>
<br />
In the <a href="http://lethalman.blogspot.it/2014/08/nix-pill-12-inputs-design-pattern.html">pill 12</a> we introduced the inputs design pattern. We do not return a derivation picking dependencies directly from the repository, rather we declare the inputs and let the callers pass the necessary arguments.<br />
<br />
In our repository we have a set of attributes that import the expressions of the packages and pass these arguments, getting back a derivation. Let's take for example the graphviz attribute:<br />
<pre>graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpeg bzip2; };
</pre>
If we wanted to produce a derivation of graphviz with a customized gd version, we would have to repeat most of the above plus specifying an alternative gd:<br />
<pre>mygraphviz = import ./graphviz.nix {
inherit mkDerivation fontconfig libjpeg bzip2;
gd = customgd;
};
</pre>
That's hard to maintain. Using callPackage it would be easier:<br />
<pre>mygraphviz = callPackage ./graphviz.nix { gd = customgd; };
</pre>
But we may still be diverging from the original graphviz in the repository.<br />
<br />
We would like to avoid specifying the nix expression again, instead reuse the original graphviz attribute in the repository and add our overrides like this:<br />
<pre>mygraphviz = graphviz.override { gd = customgd; };
</pre>
The difference is obvious, as well as the advantages of this approach.<br />
<br />
<u>Note</u>: that .override is not a "method" in the OO sense as you may think. Nix is a functional language. That .override is simply an attribute of a set.<br />
<br />
<h3>
The override implementation</h3>
<div>
<br /></div>
<div>
I remind you, the graphviz attribute in the repository is the derivation returned by the function imported from graphviz.nix. We would like to add a further attribute named "override" to the returned set.</div>
<div>
<br /></div>
<div>
Let's start simple by first creating a function "makeOverridable" that takes a function and a set of original arguments to be passed to the function.</div>
<div>
<br /></div>
<div>
<u>Contract</u>: the wrapped function must return a set.</div>
<div>
<br /></div>
<div>
Let's write a lib.nix:</div>
<pre>{
makeOverridable = f: origArgs:
let
origRes = f origArgs;
in
origRes // { override = newArgs: f (origArgs // newArgs); };
}
</pre>
<div>
So makeOverridable takes a function and a set of original arguments. It returns the original returned set, plus a new <span style="font-family: Courier New, Courier, monospace;">override</span> attribute.<br />
This override attribute is a function taking a set of new arguments, and returns the result of the original function called with the original arguments unified with the new arguments. What a mess.<br />
<br />
Let's try it with nix-repl:<br />
<pre>$ nix-repl
nix-repl> :l lib.nix
Added 1 variables.
nix-repl> f = { a, b }: { result = a+b; }
nix-repl> f { a = 3; b = 5; }
{ result = 8; }
nix-repl> res = makeOverridable f { a = 3; b = 5; }
nix-repl> res
{ override = «lambda»; result = 8; }
nix-repl> res.override { a = 10; }
{ result = 15; }
</pre>
</div>
<div>
Note that the function <span style="font-family: Courier New, Courier, monospace;">f</span> does not return the plain sum but a set, because of the contract. You didn't forget already, did you? :-)<br />
The variable <span style="font-family: Courier New, Courier, monospace;">res</span> is the result of the function call without any override. It's easy to see in the definition of makeOverridable. In addition you can see the new override attribute being a function.<br />
<br />
Calling that .override with a set will invoke the original function with the overrides, as expected.<br />
But: we can't override again! Because the returned set with result 15 does not have an override attribute!<br />
That's bad, it breaks further compositions.<br />
<br />
The solution is simple, the .override function should make the result overridable again:<br />
<pre>rec {
makeOverridable = f: origArgs:
let
origRes = f origArgs;
in
origRes // { override = newArgs: makeOverridable f (origArgs // newArgs); };
}
</pre>
Please note the <span style="font-family: Courier New, Courier, monospace;">rec</span> keyword. It's necessary so that we can refer to makeOverridable from makeOverridable itself.<br />
<br />
Now let's try overriding twice:<br />
<pre>nix-repl> :l lib.nix
Added 1 variables.
nix-repl> f = { a, b }: { result = a+b; }
nix-repl> res = makeOverridable f { a = 3; b = 5; }
nix-repl> res2 = res.override { a = 10; }
nix-repl> res2
{ override = «lambda»; result = 15; }
nix-repl> res2.override { b = 20; }
{ override = «lambda»; result = 30; }
</pre>
<br />
Success! The result is 30, as expected because <span style="font-family: Courier New, Courier, monospace;">a</span> is overridden to 10 in the first override, and <span style="font-family: Courier New, Courier, monospace;">b</span> to 20.<br />
<br />
Now it would be nice if callPackage made our derivations overridable. That was the goal of this pill after all. This is an exercise for the reader.</div>
</div>
</div>
<br />
<div>
<h3>
Conclusion</h3>
<div>
<br /></div>
<div>
The "override" pattern simplifies the way we customize packages starting from an existing set of packages. This opens a world of possibilities about using a central repository like nixpkgs, and defining overrides on our local machine without even modifying the original package.<br />
<br />
Dream of a custom isolated nix-shell environment for testing graphviz with a custom gd:<br />
<pre>debugVersion (graphviz.override { gd = customgd; })
</pre>
Once a new version of the overridden package comes out in the repository, the customized package will make use of it automatically.<br />
<br />
The key in Nix is to find powerful yet simple abstractions in order to let the user customize his environment with highest consistency and lowest maintenance time, by using predefined composable components.<br />
<br /></div>
<h3>
Next pill</h3>
<div>
<br />
...we will talk about Nix search paths. By search path I mean a place in the file system where Nix looks for expressions. You may have wondered, where does that holy <nixpkgs> come from?</div>
</div>
<br />
Pill 15 is available for <a href="http://lethalman.blogspot.it/2014/09/nix-pill-15-nix-search-paths.html">reading here</a>.<br />
<br />
<div>
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com9tag:blogger.com,1999:blog-32054652.post-26076825446740070222014-09-04T06:01:00.001-07:002014-09-24T05:29:06.588-07:00Nix pill 13: the callPackage design pattern<div>
<div>
<div style="margin: 0px;">
Welcome to the 13th Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/08/nix-pill-12-inputs-design-pattern.html">12th pill</a> we have introduced the first basic design pattern for organizing a repository of software. In addition we packaged graphviz to have at least another package for our little repository.</div>
</div>
<div>
<div style="margin: 0px;">
<br />
The next design pattern worth noting is what I'd like to call the <b>callPackage</b> pattern. This technique is extensively used in <a href="https://github.com/NixOS/nixpkgs">nixpkgs</a>, it's the current standard for importing packages in a repository.<br />
<br /></div>
</div>
</div>
<h3>
The callPackage convenience</h3>
<div>
<div style="margin: 0px;">
<br />
In the previous pill, we underlined the fact that the inputs pattern is great to decouple packages from the repository, in that we can pass manually the inputs to the derivation. The derivation declares its inputs, and the caller passes the arguments.<br />
However as with usual programming languages, we declare parameter names, and then we have to pass arguments. We do the job twice.<br />
With package management, we often see common patterns. In the case of nixpkgs it's the following.<br />
<br />
Some package derivation:<br />
<pre>{ input1, input2, ... }:
...
</pre>
Repository derivation:<br />
<pre>rec {
lib1 = import package1.nix { inherit input1 input2 ...; };
program2 = import package1.nix { inherit inputX inputY lib1 ...; };
}
</pre>
Where inputs may even be packages in the repository itself (note the <a href="http://lethalman.blogspot.it/2014/07/nix-pill-4-basics-of-language.html">rec keyword</a>).
The pattern here is clear, often inputs have the same name of the attributes in the repository itself.
Our desire is to pass those inputs from the repository automatically, and in case be able to specify a particular argument (that is, override the automatically passed default argument).<br />
<br />
To achieve this, we will define a callPackage function with the following synopsis:
<br />
<pre>{
lib1 = callPackage package1.nix { };
program2 = callPackage package2.nix { someoverride = overriddenDerivation; };
}
</pre>
What should it do?<br />
<ul>
<li>Import the given expression, which in turn returns a function.</li>
<li>Determine the name of its arguments.</li>
<li>Pass default arguments from the repository set, and let us override those arguments.</li>
</ul>
<div>
<br /></div>
<h3>
Implementing callPackage</h3>
<div>
<br /></div>
<div>
First of all, we need a way to introspect (reflection or whatever) at runtime the argument names of a function. That's because we want to automatically pass such arguments.</div>
<div>
<br /></div>
<div>
Then callPackage requires access to the whole packages set, because it needs to find the packages to pass automatically. <br />
<br />
We start off simple with nix-repl:</div>
<pre>nix-repl> add = { a ? 3, b }: a+b
nix-repl> builtins.functionArgs add
{ a = true; b = false; }
</pre>
<div>
Nix provides a builtin function to introspect the names of the arguments of a function. In addition, for each argument, it tells whether the argument has a default value or not. We don't really care about default values in our case. We are only interested in the argument names.<br />
<br />
Now we need a set with all the values, let's call it <span style="font-family: Courier New, Courier, monospace;">values</span>. And a way to intersect the attributes of <span style="font-family: Courier New, Courier, monospace;">values</span> with the function arguments:<br />
<pre>nix-repl> values = { a = 3; b = 5; c = 10; }
nix-repl> builtins.intersectAttrs values (builtins.functionArgs add)
{ a = true; b = false; }
nix-repl> builtins.intersectAttrs (builtins.functionArgs add) values
{ a = 3; b = 5; }
</pre>
Perfect, note from the example above that the <span style="font-family: Courier New, Courier, monospace;">intersectAttrs</span> returns a set whose names are the intersection, and the attribute values are taken from the second set.<br />
<br />
We're done, we have a way to get argument names from a function, and match with an existing set of attributes. This is our simple implementation of callPackage:<br />
<pre>nix-repl> callPackage = set: f: f (builtins.intersectAttrs (builtins.functionArgs f) set)
nix-repl> callPackage values add
8
nix-repl> with values; add { inherit a b; }
8
</pre>
Clearing up the syntax:<br />
<ul>
<li>We define a callPackage variable which is a function.</li>
<li>First it accepts a set, and it returns another function accepting another parameter. In other words, let's simplify by saying it accepts two parameters.</li>
<li>The second parameter is the function to "autocall".</li>
<li>We take the argument names of the function and intersect with the set of all values.</li>
<li>Finally we call the passed function <span style="font-family: Courier New, Courier, monospace;">f</span> with the resulting intersection.</li>
</ul>
</div>
<div>
In the code above, I've also shown that the callPackage call is equivalent to directly calling <span style="font-family: Courier New, Courier, monospace;">add a b</span><span style="font-family: inherit;">.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">We achieved what we wanted. <b>Automatically call functions given a set of possible arguments</b>. If an argument is not found in the set, that's nothing special. It's a function call with a missing parameter, and that's an error (unless the function has varargs </span><span style="font-family: Courier New, Courier, monospace;">...</span><span style="font-family: inherit;"> as explained in the <a href="http://lethalman.blogspot.it/2014/07/nix-pill-5-functions-and-imports.html">5th pill</a>).</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Or not. We missed something. Being able to override some of the parameters. We may not want to always call functions with values taken from the big set.</span><br />
Then we add a further parameter, which takes a set of overrides:<br />
<pre>nix-repl> callPackage = set: f: overrides: f ((builtins.intersectAttrs (builtins.functionArgs f) set) // overrides)
nix-repl> callPackage values add { }
8
nix-repl> callPackage values add { b = 12; }
15
</pre>
Apart from the increasing number of parenthesis, it should be clear that we simply do a set union between the default arguments, and the overriding set.<br />
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;">Use callPackage to simplify the repository</span></h3>
</div>
</div>
</div>
<div>
<br />
Given our brand new tool, we can simplify the repository expression (<span style="font-family: Courier New, Courier, monospace;">default.nix</span>).<br />
Let me write it down first:<br />
<pre>let
nixpkgs = import <nixpkgs> {};
allPkgs = nixpkgs // pkgs;
callPackage = path: overrides:
let f = import path;
in f ((builtins.intersectAttrs (builtins.functionArgs f) allPkgs) // overrides);
pkgs = with nixpkgs; {
mkDerivation = import ./autotools.nix nixpkgs;
hello = callPackage ./hello.nix { };
graphviz = callPackage ./graphviz.nix { };
graphvizCore = callPackage ./graphviz.nix { gdSupport = false; };
};
in pkgs
</pre>
</div>
Wow, there's a lot to say here:<br />
<ul>
<li>We renamed the old pkgs of the previous pill to nixpkgs. Our package set is now instead named pkgs. Sorry for the confusion.</li>
<li>We needed a way to pass pkgs to callPackage somehow. Instead of returning the set of packages directly from default.nix, we first assign it to a let variable and reuse it in callPackage.</li>
<li>For convenience, in callPackage we first import the file, instead of calling it directly. Otherwise for each package we would have to write the <span style="font-family: Courier New, Courier, monospace;">import</span>.</li>
<li>Since our expressions use packages from nixpkgs, in callPackage we use allPkgs, which is the union of nixpkgs and our packages.</li>
<li>We moved mkDerivation in pkgs itself, so that it gets also passed automatically.</li>
</ul>
<div>
Note how easy is to override arguments in the case of graphviz without gd. But most importantly, how easy it was to merge two repositories: nixpkgs and our pkgs!<br />
<br />
The reader should notice a magic thing happening. We're defining pkgs in terms of callPackage, and callPackage in terms of pkgs. That magic is possible thanks to lazy evaluation. </div>
<br />
<div>
<h3>
Conclusion</h3>
<div>
<br /></div>
<div>
The "callPackage" pattern has simplified a lot our repository. We're able to import packages that require some named arguments and call them automatically, given the set of all packages.<br />
<br />
We've also introduced some useful builtin functions that allows us to introspect Nix functions and manipulate attributes. These builtin functions are not usually used when packaging software, rather to provide tools for packaging. That's why they are not documented in the <a href="http://nixos.org/nix/manual/">nix manual</a>.<br />
<br />
Writing a repository in nix is an evolution of writing convenient functions for combining the packages. This demonstrates even more how nix is a generic tool to build and deploy something, and how suitable it is to create software repositories with your own conventions.<br />
<br /></div>
<h3>
Next pill</h3>
<div>
<br />
...we will talk about the "override" design pattern. The graphvizCore seems straightforward. It <i>starts from graphviz.nix</i> and builds it without gd. Now I want to give you another point of view: what if we instead wanted to <i>start from pkgs.graphviz</i> and disable gd?<br />
<br />
Pill 14 is available for <a href="http://lethalman.blogspot.it/2014/09/nix-pill-14-override-design-pattern.html">reading here</a>.<br />
<br /></div>
<div>
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com6tag:blogger.com,1999:blog-32054652.post-86926290484127216632014-08-28T08:08:00.001-07:002014-09-04T06:01:46.317-07:00Nix pill 12: the inputs design pattern<div>
<div>
<div style="margin: 0px;">
Welcome to the 12th Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/08/nix-pill-11-garbage-collector.html">11th pill</a> we stopped packaging and cleaned up the system with the garbage collector.</div>
</div>
<div>
<div style="margin: 0px;">
<br />
We restart our packaging, but we will improve a different aspect. We only packaged an hello world program so far, what if we want to create a repository of multiple packages?<br />
<br /></div>
</div>
</div>
<h3>
Repositories in Nix</h3>
<div>
<div style="margin: 0px;">
<br />
Nix is a tool for build and deployment, it does not enforce any particular repository format. A repository of packages is the main usage for Nix, but not the only possibility. See it more like a consequence due to the need of organizing packages.<br />
Nix is a language, and it is powerful enough to let you choose the format of your own repository. In this sense, it is not declarative, but functional.<br />
There is no preset directory structure or preset packaging policy. It's all about you and Nix.<br />
<br />
The <a href="http://github.com/NixOS/nixpkgs">nixpkgs</a> repository has a certain structure, which evolved and evolves with the time. Like other languages, Nix has its own history and therefore I'd like to say that it also has its own design patterns. Especially when packaging, you often do the same task again and again except for different software. It's inevitable to identify patterns during this process. Some of these patterns get reused if the community thinks it's a good way to package the software.<br />
<br />
Some of the patterns I'm going to show do not apply only to Nix, but to other systems of course.<br />
<br />
<h3>
The single repository pattern</h3>
<br />
Before introducing the "inputs" pattern, we can start talking about another pattern first which I'd like to call "single repository" pattern.<br />
Systems like Debian scatter packages in several small repositories. Personally, this makes it hard to track interdependent changes and to contribute to new packages.<br />
Systems like Gentoo instead, put package descriptions all in a single repository.<br />
<br />
The nix reference for packages is <a href="https://github.com/NixOS/nixpkgs">nixpkgs</a>, a single repository of all descriptions of all packages. I find this approach very natural and attractive for new contributions.<br />
<br />
From now on, we will adopt this technique. The natural implementation in Nix is to create a top-level Nix expression, and one expression for each package. The top-level expression imports and combines all expressions in a giant attribute set with <span style="font-family: Courier New, Courier, monospace;">name -> package</span> pairs.<br />
<br />
But isn't that heavy? It isn't, because Nix is a <a href="http://en.wikipedia.org/wiki/Lazy_evaluation">lazy</a> language, it evaluates only what's needed! And that's why nixpkgs is able to maintain such a big software repository in a giant attribute set.<br />
<br />
<h3>
Packaging graphviz</h3>
<br />
We have packaged GNU hello world, I guess you would like to package something else for creating at least a repository of two projects :-) . I chose graphviz, which uses the standard autotools build system, requires no patching and dependencies are optional.<br />
<br />
Download graphviz from <a href="http://www.graphviz.org/pub/graphviz/stable/SOURCES/graphviz-2.38.0.tar.gz">here</a>. The graphviz.nix expression is straightforward:<br />
<pre>let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
}
</pre>
</div>
</div>
<div>
Build with <span style="font-family: Courier New, Courier, monospace;">nix-build graphviz.nix</span> and you will get runnable binaries under result/bin. Notice how we did reuse the same autotools.nix of hello.nix. Let's create a simple png:<br />
<pre>$ echo 'graph test { a -- b }'|result/bin/dot -Tpng -o test.png
Format: "png" not recognized. Use one of: canon cmap [...]
</pre>
Oh of course... graphviz can't know about png. It built only the output formats it supports natively, without using any extra library.<br />
<br />
I remind you, in autotools.nix there's a buildInputs variablewhich gets concatenated to baseInputs. That would be the perfect place to add a build dependency. We created that variable exactly for this reason to be overridable from package expressions.<br />
<br />
This 2.38 version of graphviz has several plugins to output png. For simplicity, we will use libgd.<br />
<br />
<h3>
Digression about gcc and ld wrappers</h3>
<br />
The gd, jpeg, fontconfig and bzip2 libraries (dependencies of gd) don't use pkg-config to specify which flags to pass to the compiler. Since there's no global location for libraries, we need to tell gcc and ld where to find includes and libs.<br />
<br />
The nixpkgs provides gcc and binutils, and we are using them for our packaging. Not only, it also <a href="http://nixos.org/nixpkgs/manual/#ssec-setup-hooks">provides wrappers</a> for them which allow passing extra arguments to gcc and ld, bypassing the project build systems:<br />
<ul>
<li><span style="font-family: Courier New, Courier, monospace;">NIX_CFLAGS_COMPILE</span>: extra flags to gcc at compile time</li>
<li><span style="font-family: Courier New, Courier, monospace;">NIX_LDFLAGS</span>: extra flags to ld</li>
</ul>
What can we do about it? We can employ the same trick we did for PATH: automatically filling the variables from buildInputs. This is the relevant snippet of setup.sh:<br />
<pre>for p in $baseInputs $buildInputs; do
if [ -d $p/bin ]; then
export PATH="$p/bin${PATH:+:}$PATH"
fi
if [ -d $p/include ]; then
export NIX_CFLAGS_COMPILE="-I $p/include${NIX_CFLAGS_COMPILE:+ }$NIX_CFLAGS_COMPILE"
fi
if [ -d $p/lib ]; then
export NIX_LDFLAGS="-rpath $p/lib -L $p/lib${NIX_LDFLAGS:+ }$NIX_LDFLAGS"
fi
done
</pre>
Now by adding derivations to buildInputs, will add the lib, include and bin paths automatically in setup.sh.<br />
<br />
The -rpath flag in ld is needed because at runtime, the executable must use exactly that version of the library.<br />
If unneeded paths are specified, the fixup phase will shrink the rpath for us!<br />
<br />
<h3>
Completing graphviz with gd</h3>
<br />
Finish the expression for graphviz with gd support (note the use of the <b>with</b> expression in <b>buildInputs</b> to avoid repeating pkgs):<br />
<pre>let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
buildInputs = with pkgs; [ gd fontconfig libjpeg bzip2 ];
}
</pre>
</div>
Now you can create the png! Ignore any error from fontconfig, especially if you are in a chroot.<br />
<br />
<h3>
The repository expression</h3>
<br />
Now that we have two packages, what's a good way to put them together in a single repository? We do something like nixpkgs does. With nixpkgs, we import it and then we peek derivations by accessing the giant attribute set.<br />
For us nixers, this a good technique, because it abstracts from the file names. We don't refer to a package by <span style="font-family: Courier New, Courier, monospace;">REPO/some/sub/dir/package.nix</span> but by <span style="font-family: Courier New, Courier, monospace;">importedRepo.package</span> (or <span style="font-family: Courier New, Courier, monospace;">pkgs.package</span> in our examples).<br />
<br />
Create a default.nix in the current directory:
<br />
<pre>{
hello = import ./hello.nix;
graphviz = import ./graphviz.nix;
}
</pre>
Ready to use! Try it with nix-repl:
<br />
<pre>$ nix-repl
nix-repl> :l default.nix
Added 2 variables.
nix-repl> hello
«derivation /nix/store/dkib02g54fpdqgpskswgp6m7bd7mgx89-hello.drv»
nix-repl> graphviz
«derivation /nix/store/zqv520v9mk13is0w980c91z7q1vkhhil-graphviz.drv»
</pre>
With nix-build:
<br />
<pre>$ nix-build default.nix -A hello
[...]
$ result/bin/hello
Hello, world!
</pre>
The -A argument is used to access an attribute of the set from the given .nix expression.<br />
<b><u>Important</u></b>: why did we choose the default.nix? Because when a directory (by default the current directory) has a default.nix, that default.nix will be used (see import <a href="http://nixos.org/nix/manual/#ssec-builtins">here</a>). In fact you can run <span style="font-family: Courier New, Courier, monospace; white-space: normal;">nix-build -A hello</span><span style="font-family: 'Times New Roman'; white-space: normal;"> without specifying default.nix. </span><br />
For pythoners, it is similar to <span style="font-family: Courier New, Courier, monospace; white-space: normal;">__init__.py</span>.<br />
<br />
With nix-env, to install the package in your user environment:
<br />
<pre>$ nix-env -f . -iA graphviz
[...]
$ dot -V
</pre>
The -f option is used to specify the expression to use, in this case the current directory, therefore <span style="font-family: Courier New, Courier, monospace; white-space: normal;">./default.nix</span>.<br />
<div>
The -i stands for installation.</div>
<div>
The -A is the same as above for nix-build.<br />
<br />
We reproduced the very basic behavior of nixpkgs.<br />
<br />
<h3>
The inputs pattern</h3>
<br />
After a long preparation, we finally arrived. I know you have a big doubt in this moment. It's about the hello.nix and graphviz.nix. They are very, very dependent on nixpkgs:<br />
<ul>
<li>First big problem: they import nixpkgs directly. In autotools.nix instead we pass nixpkgs as an argument. That's a much better approach.</li>
<li>Second problem: what if we want a variant of graphviz without libgd support?</li>
<li>Third problem: what if we want to test graphviz with a particular libgd version?</li>
</ul>
<div>
The current answer to the above questions is: change the expression to match your needs (or change the callee to match your needs).</div>
<div>
With the inputs pattern, we choose to give another answer: let the user change the inputs of the expression (or change the caller to pass different inputs).</div>
<div>
<br /></div>
<div>
By inputs of an expression, we refer to the set of derivations needed to build that expression. In this case:</div>
<div>
<ul>
<li>mkDerivation from autotools. Recall that mkDerivation has an implicit dependency on the toolchain.</li>
<li>libgd and its dependencies.</li>
</ul>
<div>
The src is also an input but it's pointless to change the source from the caller. For version bumps, in nixpkgs we prefer to write another expression (e.g. because patches are needed or different inputs are needed).</div>
</div>
<div>
<br /></div>
<div>
<b><u>Goal</u></b>: make package expressions independent of the repository.<br />
<br /></div>
<div>
How do we achieve that? The answer is simple: <b>use functions to declare inputs for a derivation</b>. Doing it for graphviz.nix, will make the derivation independent of the repository and customizable:</div>
<pre>{ mkDerivation, gdSupport ? true, gd, fontconfig, libjpeg, bzip2 }:
mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
buildInputs = if gdSupport then [ gd fontconfig libjpeg bzip2 ] else [];
}
</pre>
<div>
I recall that "<span style="font-family: Courier New, Courier, monospace;">{...}: ...</span>" is the syntax for defining <a href="http://nixos.org/nix/manual/#idm47361539166560">functions</a> accepting an attribute set as argument.<br />
<br />
We made gd and its dependencies optional. If gdSupport is true (by default), we will fill buildInputs and thus graphviz will be built with gd support, otherwise it won't.
<br />
Now back to default.nix:<br />
<pre>let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in with pkgs; {
hello = import ./hello.nix { inherit mkDerivation; };
graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpeg bzip2; };
graphvizCore = import ./graphviz.nix {
inherit mkDerivation gd fontconfig libjpeg bzip2;
gdSupport = false;
};
}
</pre>
</div>
</div>
So we factorized the import of nixpkgs and mkDerivation, and also added a variant of graphviz with gd support disabled. The result is that both hello.nix (exercise for the reader) and graphviz.nix are independent of the repository and customizable by passing specific inputs.<br />
If you wanted to build graphviz with a specific version of gd, it would suffice to pass <span style="font-family: Courier New, Courier, monospace;">gd = ...</span>;.<br />
If you wanted to change the toolchain, you may pass a different mkDerivation function.<br />
<br />
Clearing up the syntax:<br />
<ul>
<li>In the end we return an attribute set from default.nix. With "let" we define some local variables.</li>
<li>We bring pkgs into the scope when defining the packages set, which is very convenient instead of typing everytime "pkgs".</li>
<li>We import hello.nix and graphviz.nix, which will return a function, and call it with a set of inputs to get back the derivation.</li>
<li>The "<span style="font-family: Courier New, Courier, monospace;">inherit x</span>" syntax is equivalent to "<span style="font-family: Courier New, Courier, monospace;">x = x</span>". So "<span style="font-family: Courier New, Courier, monospace;">inherit gd</span>" here, combined to the above "<span style="font-family: Courier New, Courier, monospace;">with pkgs;</span>" is equivalent to "<span style="font-family: Courier New, Courier, monospace;">x = pkgs.gd</span>".</li>
</ul>
<div>
You can find the whole repository at the <a href="https://gist.github.com/lethalman/734b168a0258b8a38ca2">pill 12 gist</a>.</div>
<div>
<br /></div>
<div>
<h3>
Conclusion</h3>
<div>
<br /></div>
<div>
The "inputs" pattern allows our expressions to be easily customizable through a set of arguments. These arguments could be flags, derivations, or whatelse. Our package expressions are functions, don't think there's any magic in there.<br />
<br />
It also makes the expressions independent of the repository. Given that all the needed information is passed through arguments, it is possible to use that expression in any other context.<br />
<br /></div>
<h3>
Next pill</h3>
<div>
<br />
...we will talk about the "callPackage" design pattern. It is tedious to specify the names of the inputs twice, once in the top-level default.nix, and once in the package expression. With callPackage, we will implicitly pass the necessary inputs from the top-level expression.<br />
<br />
Pill 13 is available for <a href="http://lethalman.blogspot.it/2014/09/nix-pill-13-callpackage-design-pattern.html">reading here</a>.<br />
<br /></div>
<div>
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com6tag:blogger.com,1999:blog-32054652.post-48429481758238387562014-08-21T02:40:00.001-07:002014-09-16T12:53:39.508-07:00Nix pill 11: the garbage collector<div>
<div>
<div style="margin: 0px;">
Welcome to the 11th Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/08/nix-pill-10-developing-with-nix-shell.html">10th pill</a> we managed to obtain a self-contained environment for developing a project. The concept is that whereas <span style="font-family: Courier New, Courier, monospace;">nix-build</span> is able to build a derivation in isolation, <span style="font-family: Courier New, Courier, monospace;">nix-shell</span> is able to drop us in a shell with (almost) the same environment used by <span style="font-family: Courier New, Courier, monospace;">nix-build</span>. This allows us to debug, modify and manually build software.</div>
</div>
<div>
<div style="margin: 0px;">
<br />
Today we stop packaging and look at a mandatory nix component, the garbage collector. When using nix tools, often derivations are built. This include both .drv files and out paths. These artifacts go in the nix store, and we never cared about deleting them until now.</div>
</div>
<div>
<div style="margin: 0px;">
<br /></div>
</div>
</div>
<h3>
How does it work</h3>
<div>
<div style="margin: 0px;">
<br />
Other package managers, like dpkg, have somehow a way to remove unused software. However, nix is much more precise compared to other systems.<br />
I bet with dpkg, rpm or anything else, you end up with having some unnecessary package installed or dangling files. With nix this does not happen.<br />
<br />
How do we determine whether a store path is still needed? The same way programming languages with a garbage collector decide whether an object is still alive.<br />
<br />
Programming languages with a garbage collector have an important concept in order to keep track of live objects: GC roots. A GC root is an object that is always alive (unless explicitly removed as GC root). All objects recursively referred to by a GC root are live.<br />
<br />
Therefore, the garbage collection process starts from GC roots, and recursively mark referenced objects as live. All other objects can be collected and deleted.<br />
<br />
In Nix there's this same concept. Instead of being objects, of course, <a href="http://nixos.org/nix/manual/#ssec-gc-roots">GC roots are store paths</a>. The implementation is very simple and transparent to the user. GC roots are stored under <span style="font-family: Courier New, Courier, monospace;">/nix/var/nix/gcroots</span>. If there's a symlink to a store path, then that store path is a GC root.<br />
Nix allows this directory to have subdirectories: it will simply recurse directories in search of symlinks to store paths.<br />
<br />
So we have a list of GC roots. At this point, deleting dead store paths is as easy as you can imagine. We have the list of all live store paths, hence the rest of the store paths are dead.<br />
In particular, Nix first moves dead store paths to <span style="font-family: Courier New, Courier, monospace;">/nix/store/trash</span> which is an atomic operation. Afterwards, the trash is emptied.<br />
<br />
<h3>
Playing with the GC</h3>
<br />
Before playing with the GC, first run the <a href="http://nixos.org/nix/manual/#sec-nix-collect-garbage">nix garbage collector</a> once, so that we have a cleaned up playground for our experiments:<br />
<pre>$ nix-collect-garbage
finding garbage collector roots...
[...]
deleting unused links...
note: currently hard linking saves -0.00 MiB
1169 store paths deleted, 228.43 MiB freed
</pre>
Perfect, if you run it again it won't find anything new to delete, as expected.<br />
What's left in the nix store is everything being referenced from the GC roots.<br />
<br />
Let's install for a moment bsd-games:<br />
<pre>$ nix-env -iA nixpkgs.bsdgames
$ readlink -f `which fortune`
/nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-<b>bsd-games-2.17</b>/bin/fortune
$ nix-store -q --roots `which fortune`
/nix/var/nix/profiles/default-9-link
$ nix-env --list-generations
[...]
9 2014-08-20 12:44:14 (current)
</pre>
The nix-store command can be used to query the GC roots that refer to a given derivation. In this case, our current user environment does refer to bsd-games.<br />
<br />
Now remove it, collect garbage and note that bsd-games is still in the nix store:<br />
<pre>$ nix-env -e bsd-games
uninstalling `bsd-games-2.17'
$ nix-collect-garbage
[...]
$ ls /nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-<b>bsd-games-2.17</b>
bin share
</pre>
That's because the old generation is still in the nix store because it's a GC root. As we'll see below, all profiles and their generations are GC roots.<br />
<br />
Removing a GC root is simple. Let's try deleting the generation that refers to bsd-games, collect garbage, and note that now bsd-games is no longer in the nix store:<br />
<pre>$ rm /nix/var/nix/profiles/default-9-link
$ nix-env --list-generations
[...]
8 2014-07-28 10:23:24
10 2014-08-20 12:47:16 (current)
$ nix-collect-garbage
[...]
$ ls /nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-<b>bsd-games-2.17</b>
ls: cannot access /nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-<b>bsd-games-2.17</b>: No such file or directory
</pre>
<u>Note</u>: <span style="font-family: Courier New, Courier, monospace;">nix-env --list-generations</span> does not rely on any particular metadata. It is able to list generations based solely on the file names under the profiles directory.<br />
<br />
However we removed the link from <span style="font-family: Courier New, Courier, monospace;">/nix/var/nix/profiles</span>, not from <span style="font-family: Courier New, Courier, monospace;">/nix/var/nix/gcroots</span>. Turns out, that <span style="font-family: Courier New, Courier, monospace;">/nix/var/nix/gcroots/profiles</span> is a symlink to <span style="font-family: Courier New, Courier, monospace;">/nix/var/nix/profiles</span>. That is very handy. It means any profile and its generations are GC roots.<br />
<br />
It's as simple as that, anything under <span style="font-family: Courier New, Courier, monospace;">/nix/var/nix/gcroots</span> is a GC root. And anything not being garbage collected is because it's referred from one of the GC roots.<br />
<br />
<h3>
Indirect roots</h3>
<br />
I remind you that building the GNU hello world package with nix-build produces a <span style="font-family: Courier New, Courier, monospace;">result</span> symlink in the current directory. Despite the collected garbage done above, the hello program is still working: therefore it has not been garbage collected. Clearly, since there's no other derivation that depends upon the GNU hello world package, it must be a GC root.<br />
<br />
In fact, nix-build automatically adds the <span style="font-family: Courier New, Courier, monospace;">result</span> symlink as a GC root. Yes, not the built derivation, but the symlink. These GC roots are added under <span style="font-family: Courier New, Courier, monospace;">/nix/var/nix/gcroots/auto</span> .<br />
<pre>$ ls -l /nix/var/nix/gcroots/auto/
total 8
drwxr-xr-x 2 nix nix 4096 Aug 20 10:24 ./
drwxr-xr-x 3 nix nix 4096 Jul 24 10:38 ../
lrwxrwxrwx 1 nix nix 16 Jul 31 10:51 xlgz5x2ppa0m72z5qfc78b8wlciwvgiz -> <b>/home/nix/result/</b>
</pre>
Don't care about the name of the symlink. What's important is that a symlink exists that point to <span style="font-family: Courier New, Courier, monospace;">/home/nix/result</span>. This is called an <b>indirect GC root</b>. That is, the GC root is effectively specified outside of <span style="font-family: Courier New, Courier, monospace;">/nix/var/nix/gcroots</span>. Whatever <span style="font-family: Courier New, Courier, monospace;">result</span> points to, it will not be garbage collected.<br />
<br />
How do we remove the derivation then? There are two possibilities:<br />
<ul>
<li>Remove the indirect GC root from<span style="font-family: Courier New, Courier, monospace;"> /nix/var/nix/gcroots/auto</span>.</li>
<li>Remove the <span style="font-family: Courier New, Courier, monospace;">result</span> symlink.</li>
</ul>
<div>
In the first case, the derivation will be deleted from the nix store, and <span style="font-family: Courier New, Courier, monospace;">result</span> becomes a dangling symlink. In the second case, the derivation is removed as well as the indirect root in <span style="font-family: Courier New, Courier, monospace;">/nix/var/nix/gcroots/auto</span> .</div>
<div>
Running <span style="font-family: Courier New, Courier, monospace;">nix-collect-garbage</span> after deleting the GC root or the indirect GC root, will remove the derivation from the store.</div>
<br />
<h3>
Cleanup everything</h3>
<br />
What's the main source of software duplication in the nix store? Clearly, GC roots due to nix-build and profile generations. Doing a nix-build results in a GC root for a build that somehow will refer to a specific version of glibc, and other libraries. After an upgrade, if that build is not deleted by the user, it will not be garbage collected. Thus the old dependencies referred to by the build will not be deleted either.<br />
<br />
Same goes for profiles. Manipulating the nix-env profile will create further generations. Old generations refer to old software, thus increasing duplication in the nix store after an upgrade.<br />
<br />
What are the basic steps for upgrading and removing everything old, including old generations? In other words, do an upgrade similar to other systems, where they forget everything about the older state:<br />
<pre>$ nix-channel --update
$ nix-env -u --always
$ rm /nix/var/nix/gcroots/auto/*
$ nix-collect-garbage -d
</pre>
First, we download a new version of the nixpkgs channel, which holds the description of all the software. Then we upgrade our installed packages with <span style="font-family: Courier New, Courier, monospace;">nix-env -u</span>. That will bring us into a fresh new generation with all updated software.<br />
<br />
Then we remove all the indirect roots generated by nix-build: beware, this will result in dangling symlinks. You may be smarter and also remove the target of those symlinks.<br />
<br />
Finally, the <span style="font-family: Courier New, Courier, monospace;">-d</span> option of <span style="font-family: Courier New, Courier, monospace;">nix-collect-garbage</span> is used to delete old generations of all profiles, then collect garbage. After this, you lose the ability to rollback to any previous generation. So make sure the new generation is working well before running the command.<br />
<br /></div>
</div>
<div>
<h3>
Conclusion</h3>
<div>
<br /></div>
<div>
Garbage collection in Nix is a powerful mechanism to cleanup your system. The nix-store commands allows us to know why a certain derivation is in the nix store.<br />
<br />
Cleaning up everything down to the oldest bit of software after an upgrade seems a bit contrived, but that's the price of having multiple generations, multiple profiles, multiple versions of software, thus rollbacks etc.. The price of having many possibilities.<br />
<div>
<br /></div>
</div>
<h3>
Next pill</h3>
<div>
<br />
...we will package another project and introduce what I call the "inputs" design pattern. We only played with a single derivation until now, however we'd like to start organizing a small repository of software. The "inputs" pattern is widely used in nixpkgs; it allows us to decouple derivations from the repository itself and increase customization opportunities.<br />
<br />
Pill 12 is available for <a href="http://lethalman.blogspot.it/2014/08/nix-pill-12-inputs-design-pattern.html">reading here</a>.<br />
<br /></div>
<div>
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com7tag:blogger.com,1999:blog-32054652.post-18624516854681069452014-08-19T03:21:00.001-07:002015-01-16T03:30:47.010-08:00Nix pill 10: developing with nix-shell<div style="font-size: medium;">
<div style="font-weight: normal;">
<div style="margin: 0px;">
Welcome to the 10th Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/08/nix-pill-9-automatic-runtime.html">9th pill</a> we have seen one of the powerful features of nix, automatic discovery of runtime dependencies and finalized the GNU hello world package.</div>
</div>
<div style="font-weight: normal;">
<div style="margin: 0px;">
<br /></div>
</div>
<div style="font-weight: normal;">
<div style="margin: 0px;">
In return from vacation, we want to hack a little the GNU hello world program. The nix-build tool creates an isolated environment for building the derivation, we want to do the same in order to modify some source files of the project.</div>
<div style="margin: 0px;">
<br /></div>
</div>
</div>
<h3>
What's nix-shell</h3>
<div style="font-size: medium;">
<div style="font-weight: normal;">
<div style="margin: 0px;">
<br /></div>
<div style="margin: 0px;">
The <a href="http://nixos.org/nix/manual/#sec-nix-shell">nix-shell</a> tool drops us in a shell by setting up the necessary environment variables to hack a derivation. It does not build the derivation, it only serves as a preparation so that we can run the build steps manually.</div>
<div style="margin: 0px;">
I remind you, in a nix environment you don't have access to libraries and programs unless you install them with nix-env. However installing libraries with nix-env is not good practice. We prefer to have isolated environments for development.</div>
<pre>$ nix-shell hello.nix
[nix-shell]$ make
bash: make: command not found
[nix-shell]$ echo $baseInputs
/nix/store/jff4a6zqi0yrladx3kwy4v6844s3swpc-<b>gnutar-1.27.1</b> [...]
</pre>
<div style="margin: 0px;">
First thing to notice, we call nix-shell on a nix expression which returns a derivation. We then enter a new bash shell, but it's really useless. We expected to have the GNU hello world build inputs available in PATH, including GNU make, but it's not the case.<br />
But, we have the environment variables that we set in the derivation, like $baseInputs, $buildInputs, $src and so on.<br />
<br />
That means we can source our builder.sh, and it will build the derivation. You may get an error in the installation phase, because the user may not have the permission to write to /nix/store:</div>
<pre>[nix-shell]$ source builder.sh
...
</pre>
<div style="margin: 0px;">
It didn't install, but it built. Things to notice:<br />
<ul>
<li>We sourced builder.sh, therefore it ran all the steps including setting up the PATH for us.</li>
<li>The working directory is no more a temp directory created by nix-build, but the current directory. Therefore, hello-2.9 has been unpacked there.</li>
</ul>
<div>
We're able to cd into hello-2.9 and type make, because now it's available.</div>
<div>
<br /></div>
<div>
In other words, nix-shell drops us in a shell with the same (or almost) environment used to run the builder!</div>
<br />
<h3>
A builder for nix-shell</h3>
</div>
</div>
</div>
<div>
<br />
The previous steps are a bit annoying of course, but we can improve our builder to be more nix-shell friendly.<br />
<br />
First of all, we were able to source builder.sh because it was in our current directory, but that's not nice. We want the builder.sh that is stored in the nix store, the one that would be used by nix-build. To do so, the right way is to pass the usual environment variable through the derivation.<br />
<u>Note</u>: $builder is already defined, but it's the bash executable, not our builder.sh. Our builder.sh is an argument to bash.<br />
<br />
Second, we don't want to run the whole builder, we only want it to setup the necessary environment for manually building the project. So we'll write two files, one for setting up the environment, and the real builder.sh that runs with nix-build.<br />
<br />
Additionally, we'll wrap the phases in functions, it may be useful, and move the <span style="font-family: Courier New, Courier, monospace;">set -e</span> to the builder instead of the setup. The <span style="font-family: Courier New, Courier, monospace;">set -e</span> is annoying in nix-shell.<br />
<br />
The codebase is becoming a little long. You can find all the files in this <a href="https://gist.github.com/lethalman/97fae227329b442267bc">nixpill10 gist</a>.<br />
Noteworthy is the <span style="font-family: Courier New, Courier, monospace;">setup = ./setup.sh; </span>attribute in the derivation, which adds setup.sh to the nix store and as usual, adds a $setup environment variable in the builder.<br />
Thanks to that, we can split builder.sh into setup.sh and builder.sh. What builder.sh does is sourcing $setup and calling the genericBuild function. Everything else is just some bash changes.<br />
<br />
Now back to nix-shell:<br />
<pre>$ nix-shell hello.nix
[nix-shell]$ source $setup
[nix-shell]$
</pre>
Now you can run, for example, unpackPhase which unpacks $src and enters the directory. And you can run commands like ./configure, make etc. manually, or run phases with their respective functions.<br />
<br />
It's all that straight, nix-shell builds the .drv file and its input dependencies, then drops into a shell by setting up the environment variables necessary to build the .drv, in particular those passed to the derivation function.<br />
<br />
<h3>
Conclusion</h3>
<div>
<br /></div>
<div>
With nix-shell we're able to drop into an isolated environment for developing a project, with the necessary dependencies just like nix-build does, except we can build and debug the project manually, step by step like you would do in any other operating system.<br />
Note that we did never install gcc, make, etc. system-wide. These tools and libraries are available per-build.<br />
<br /></div>
<h3>
<br class="Apple-interchange-newline" />Next pill</h3>
<div>
<br />
...we will clean up the nix store. We wrote and built derivations, added stuff to nix store, but until now we never worried about cleaning up the used space in the store. It's time to collect some garbage.<br />
<br />
Pill 11 is available for <a href="http://lethalman.blogspot.it/2014/08/nix-pill-11-garbage-collector.html">reading here</a>.<br />
<br /></div>
<div>
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com7tag:blogger.com,1999:blog-32054652.post-32001482143078142442014-08-08T01:11:00.003-07:002014-08-19T03:22:27.572-07:00Nix pill 9: automatic runtime dependencies<h3>
<div style="font-size: medium;">
<div style="font-weight: normal;">
Welcome to the 9th Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/08/nix-pill-8-generic-builders.html">8th pill</a> we wrote a generic builder for autotools projects. We feed build dependencies, a source tarball, and we get a Nix derivation as a result.</div>
<div style="font-weight: normal;">
<br /></div>
<div style="font-weight: normal;">
Today we stop by the GNU hello world program to analyze build and runtime dependencies, and enhance the builder in order to avoid unnecessary runtime dependencies.<br />
<br /></div>
<h3>
Build dependencies</h3>
<div style="font-weight: normal;">
<br /></div>
<div style="font-weight: normal;">
Let's start analyzing build dependencies for our GNU hello world package:</div>
<pre><span style="font-weight: normal;">$ nix-instantiate hello.nix
/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-</span>hello.drv<span style="font-weight: normal;">
$ nix-store -q --references /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-</span>hello.drv<span style="font-weight: normal;">
/nix/store/0q6pfasdma4as22kyaknk4kwx4h58480-</span>hello-2.9.tar.gz<span style="font-weight: normal;">
/nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-</span>coreutils-8.21.drv<span style="font-weight: normal;">
/nix/store/2h4b30hlfw4fhqx10wwi71mpim4wr877-</span>gnused-4.2.2.drv<span style="font-weight: normal;">
/nix/store/39bgdjissw9gyi4y5j9wanf4dbjpbl07-</span>gnutar-1.27.1.drv<span style="font-weight: normal;">
/nix/store/7qa70nay0if4x291rsjr7h9lfl6pl7b1-</span>builder.sh<span style="font-weight: normal;">
/nix/store/g6a0shr58qvx2vi6815acgp9lnfh9yy8-</span>gnugrep-2.14.drv<span style="font-weight: normal;">
/nix/store/jdggv3q1sb15140qdx0apvyrps41m4lr-</span>bash-4.2-p45.drv<span style="font-weight: normal;">
/nix/store/pglhiyp1zdbmax4cglkpz98nspfgbnwr-</span>gnumake-3.82.drv<span style="font-weight: normal;">
/nix/store/q9l257jn9lndbi3r9ksnvf4dr8cwxzk7-</span>gawk-4.1.0.drv<span style="font-weight: normal;">
/nix/store/rgyrqxz1ilv90r01zxl0sq5nq0cq7v3v-</span>binutils-2.23.1.drv<span style="font-weight: normal;">
/nix/store/qzxhby795niy6wlagfpbja27dgsz43xk-</span>gcc-wrapper-4.8.3.drv<span style="font-weight: normal;">
/nix/store/sk590g7fv53m3zp0ycnxsc41snc2kdhp-</span>gzip-1.6.drv<span style="font-weight: normal;">
</span></pre>
<div style="font-weight: normal;">
It has exactly the derivations referenced in the <span style="font-family: Courier New, Courier, monospace;">derivation</span> function, nothing more, nothing less. Some of them might not be used at all, however given that our generic mkDerivation function always pulls such dependencies (think of it like <a href="https://packages.debian.org/unstable/build-essential">build-essential</a> of Debian), for every package you build from now on, you will have these packages in the nix store.</div>
<div style="font-weight: normal;">
<br /></div>
<div style="font-weight: normal;">
Why are we looking at .drv files? Because the hello.drv file is the representation of the build action to perform in order to build the hello out path, and as such it also contains the input derivations needed to be built before building hello.</div>
</div>
<div style="font-size: medium; font-weight: normal;">
<br />
<h3>
Digression about NAR files</h3>
<br />
NAR is the Nix ARchive. First question: why not tar? Why another archiver? Because commonly used archivers are not deterministic. They add padding, they do not sort files, they add timestamps, etc.. Hence NAR, a very simple deterministic archive format being used by Nix for deployment.<br />
NARs are also used extensively within Nix itself as we'll see below.<br />
<br />
For the rationale and implementation details you can find more in the <a href="http://nixos.org/~eelco/pubs/phd-thesis.pdf">Dolstra's PhD Thesis</a>.<br />
<br />
To create NAR archives, it's possible to use <span style="font-family: Courier New, Courier, monospace;">nix-store --dump</span> and <span style="font-family: Courier New, Courier, monospace;">nix-store --restore</span>. Those two commands work regardless of /nix/store.<br />
<br />
<h3>
Runtime dependencies</h3>
</div>
</h3>
<div>
<br /></div>
<div>
Something is different for runtime dependencies however. Build dependencies are automatically recognized by Nix once they are used in any <span style="font-family: Courier New, Courier, monospace;">derivation</span> call, but we never specify what are the runtime dependencies for a derivation.<br />
<br />
There's really black magic involved. It's something that at first glance makes you think "no, this can't work in the long term", but at the same it works so well that a whole operating system is built on top of this magic.<br />
<br />
In other words, Nix automatically computes all the runtime dependencies of a derivation, and it's possible thanks to the hash of the store paths.<br />
<br />
Steps:<br />
<ol>
<li>Dump the derivation as NAR, a serialization of the derivation output. Works fine whether it's a single file or a directory.</li>
<li>For each build dependency .drv and its relative out path, search the contents of the NAR for this out path.</li>
<li>If found, then it's a runtime dependency.</li>
</ol>
<br />
You get really all the runtime dependencies, and that's why Nix deployments are so easy.<br />
<pre>$ nix-instantiate hello.nix
/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-<b>hello.drv</b>
$ nix-store -r /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-<b>hello.drv</b>
/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-<b>hello</b>
$ nix-store -q --references /nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-<b>hello</b>
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-<b>glibc-2.19</b>
/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-<b>gcc-4.8.3</b>
/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-<b>hello</b>
</pre>
Ok glibc and gcc. Well, gcc really should not be a runtime dependency!<br />
<pre>$ strings result/bin/hello|grep gcc
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-<b>glibc-2.19</b>/lib:/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-<b>gcc-4.8.3</b>/lib64
</pre>
Oh Nix added gcc because its out path is mentioned in the "hello" binary. Why is that? That's the <a href="http://en.wikipedia.org/wiki/Rpath">ld rpath</a>. It's the list of directories where libraries can be found at runtime. In other distributions, this is usually not abused. But in Nix, we have to refer to particular versions of libraries, thus the rpath has an important role.<br />
<br />
The build process adds that gcc lib path thinking it may be useful at runtime, but really it's not. How do we get rid of it? Nix authors have written another magical tool called <a href="https://nixos.org/patchelf.html">patchelf</a>, which is able to reduce the rpath to the paths that are really used by the binary.<br />
<br />
Not only, even after reducing the rpath the hello binary would still depend upon gcc. Because of debugging information. For that, the well known <a href="http://unixhelp.ed.ac.uk/CGI/man-cgi?strip">strip</a> can be used.<br />
<br />
<h3>
Another phase in the builder</h3>
<br />
We will add a new phase to our autotools builder. The builder has these phases already:<br />
<ol>
<li>First the environment is set up</li>
<li>Unpack phase: we unpack the sources in the current directory (remember, Nix changes dir to a temporary directory first)</li>
<li>Change source root to the directory that has been unpacked</li>
<li>Configure phase: <span style="font-family: Courier New, Courier, monospace;">./configure</span></li>
<li>Build phase: <span style="font-family: Courier New, Courier, monospace;">make</span></li>
<li>Install phase: <span style="font-family: Courier New, Courier, monospace;">make install</span></li>
</ol>
<div>
We add a new phase after the installation phase, which we call <b>fixup</b> phase. At the end of the builder.sh follows:</div>
<pre>find $out -type f -exec patchelf --shrink-rpath '{}' \; -exec strip '{}' \; 2>/dev/null
</pre>
That is, for each file we run <span style="font-family: Courier New, Courier, monospace;">patchelf --shrink-rpath</span> and <span style="font-family: Courier New, Courier, monospace;">strip</span><span style="font-family: inherit;">. Note that we used two new commands here, find and patchelf. These two deserve a place in </span><span style="font-family: Courier New, Courier, monospace;">baseInputs</span><span style="font-family: inherit;"> of autotools.nix as </span><span style="font-family: Courier New, Courier, monospace;">findutils</span><span style="font-family: inherit;"> and </span><span style="font-family: Courier New, Courier, monospace;">patchelf</span><span style="font-family: inherit;">.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Rebuild hello.nix and...:</span></div>
<pre>$ nix-build hello.nix
[...]
$ nix-store -q --references result
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19
/nix/store/md4a3zv0ipqzsybhjb8ndjhhga1dj88x-hello
</pre>
...only glibc is the runtime dependency. Exactly what we wanted.<br />
<br />
The package is self-contained, copy its closure on another machine and you will be able to run it. I remind you the very few components under the /nix/store necessary to run nix <a href="http://lethalman.blogspot.it/2014/07/nix-pill-2-install-on-your-running.html">when we installed it</a>. The hello binary will use that exact version of glibc library and interpreter, not the system one:<br />
<pre>$ ldd result/bin/hello
linux-vdso.so.1 (0x00007fff11294000)
libc.so.6 => /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/libc.so.6 (0x00007f7ab7362000)
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/ld-linux-x86-64.so.2 (0x00007f7ab770f000)
</pre>
Of course, the executable runs fine as long as everything is under the /nix/store path.<br />
<div>
<br />
<h3>
Conclusion</h3>
<div>
<br /></div>
<div>
Short post compared to previous ones as I'm still on vacation, but I hope you enjoyed it. Nix provides tools with cool features. In particular, Nix is able to compute all runtime dependencies automatically for us. Not only shared libraries, but also referenced executables, scripts, Python libraries etc..</div>
<div>
<br />
This makes packages self-contained, because we're sure (apart data and configuration) that copying the runtime closure on another machine is sufficient to run the program. That's why Nix has <a href="http://nixos.org/nix/manual/#sec-one-click">one-click install</a>, or <a href="http://nixos.org/nixops/manual/#chap-introduction">reliable deployment in the cloud</a>. All with one tool.</div>
<h3>
<br class="Apple-interchange-newline" />Next pill</h3>
<div>
<br />
...we will introduce nix-shell. With nix-build we build derivations always from scratch: the source gets unpacked, configured, built and installed. But this may take a long time, think of WebKit. What if we want to apply some small changes and compile incrementally instead, yet keeping a self-contained environment similar to nix-build?<br />
<br />
Pill 10 is available for <a href="http://lethalman.blogspot.it/2014/08/nix-pill-10-developing-with-nix-shell.html">reading here</a>.<br />
<br /></div>
<div>
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com8tag:blogger.com,1999:blog-32054652.post-31953225246043438312014-08-01T08:28:00.001-07:002014-09-23T05:28:48.738-07:00Nix pill 8: generic buildersWelcome to the 8th Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/07/nix-pill-7-working-derivation.html">7th pill</a> we successfully built a derivation. We wrote a builder script that compiled a C file and installed the binary under the nix store.<br />
<br />
In this post, we will generalize the builder script, write a Nix expression for <a href="http://www.gnu.org/software/hello/">GNU hello world</a> and create a wrapper around the derivation built-in function.<br />
<div>
<br /></div>
<h3>
Packaging GNU hello world</h3>
<div>
<div>
<br />
In the previous pill we packaged a simple .c file, which was being compiled with a raw gcc call. That's not a good example of project. Many use autotools, and since we're going to generalize our builder, better do it with the most used build system.<br />
<br />
<a href="http://www.gnu.org/software/hello/">GNU hello world</a>, despite its name, is a simple yet complete project using autotools. Fetch the latest tarball here: <a href="http://ftp.gnu.org/gnu/hello/hello-2.9.tar.gz">http://ftp.gnu.org/gnu/hello/hello-2.9.tar.gz</a> .<br />
<br />
Let's create a builder script for GNU hello world:<br />
<pre>hello_builder.sh
export PATH="$gnutar/bin:$gcc/bin:$gnumake/bin:$coreutils/bin:$gawk/bin:$gzip/bin:$gnugrep/bin:$gnused/bin:$binutils/bin"
tar -xzf $src
cd hello-2.9
./configure --prefix=$out
make
make install
</pre>
And the derivation hello.nix:<br />
<pre>with (import <nixpkgs> {});
derivation {
name = "hello";
builder = "${bash}/bin/bash";
args = [ ./hello_builder.sh ];
inherit gnutar gzip gnumake gcc binutils coreutils gawk gnused gnugrep;
src = ./hello-2.9.tar.gz;
system = builtins.currentSystem;
}
</pre>
</div>
</div>
Now build it with <span style="font-family: Courier New, Courier, monospace;">nix-build hello.nix</span> and you can launch <span style="font-family: Courier New, Courier, monospace;">result/bin/hello</span><span style="font-family: inherit;">. Nothing easier, but do we have to create a builder.sh for each package? Do we always have to pass the dependencies to the </span><span style="font-family: Courier New, Courier, monospace;">derivation</span><span style="font-family: inherit;"> function?</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Please note the </span><span style="font-family: Courier New, Courier, monospace;">--prefix=$out</span><span style="font-family: inherit;"> we were talking about in the <a href="http://lethalman.blogspot.it/2014/07/nix-pill-7-working-derivation.html">previous pill</a>.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
A generic builder</h3>
<span style="font-family: inherit;"><br /></span>
Let's a create a generic builder.sh for autotools projects:<br />
<pre>set -e
unset PATH
for p in $buildInputs; do
export PATH=$p/bin${PATH:+:}$PATH
done
tar -xf $src
for d in *; do
if [ -d "$d" ]; then
cd "$d"
break
fi
done
./configure --prefix=$out
make
make install
</pre>
What do we do here?
<br />
<ol>
<li>Exit the build on any error with set -e.</li>
<li>First unset PATH, because it's initially set to a non-existant path.</li>
<li>We'll see this below in detail, however for each path in $buildInputs, we append bin to PATH.</li>
<li>Unpack the source.</li>
<li>Find a directory where the source has been unpacked and cd into it.</li>
<li>Once we're set up, compile and install.</li>
</ol>
As you can see, there's no reference to "hello" in the builder anymore. It still does several assumptions, but it's certainly more generic.<br />
Now let's rewrite hello.nix:<br />
<pre>with (import <nixpkgs> {});
derivation {
name = "hello";
builder = "${bash}/bin/bash";
args = [ ./builder.sh ];
buildInputs = [ gnutar gzip gnumake gcc binutils coreutils gawk gnused gnugrep ];
src = ./hello-2.9.tar.gz;
system = builtins.currentSystem;
}
</pre>
All clear, except that buildInputs. However it's easier than any black magic you are thinking in this moment.<br />
<br />
Nix is able to convert a list to a string. It first converts the elements to strings, and then concatenates them separated by a space:<br />
<pre>nix-repl> builtins.toString 123
"123"
nix-repl> builtins.toString [ 123 456 ]
"123 456"
</pre>
Recall that derivations can be converted to a string, hence:<br />
<pre>nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> builtins.toString gnugrep
"/nix/store/g5gdylclfh6d224kqh9sja290pk186xd-<b>gnugrep-2.14</b>"
nix-repl> builtins.toString [ gnugrep gnused ]
"/nix/store/g5gdylclfh6d224kqh9sja290pk186xd-<b>gnugrep-2.14</b> /nix/store/krgdc4sknzpw8iyk9p20lhqfd52kjmg0-<b>gnused-4.2.2</b>"
</pre>
Simple! The buildInputs variable is a string with out paths separated by space, perfect for bash usage in a for loop.<br />
<br />
<h3>
A more convenient derivation function</h3>
<br />
We managed to write a builder that can be used for multiple autotools projects. But in the hello.nix expression we are specifying tools that are common to more projects; we don't want to pass them everytime.<br />
A natural approach would be to create a function that accepts an attribute set, similar to the one used by the derivation function, and merge it with another attribute set containing values common to many projects.<br />
<br />
Create autotools.nix:<br />
<pre>pkgs: attrs:
with pkgs;
let defaultAttrs = {
builder = "${bash}/bin/bash";
args = [ ./builder.sh ];
baseInputs = [ gnutar gzip gnumake gcc binutils coreutils gawk gnused gnugrep ];
buildInputs = [];
system = builtins.currentSystem;
};
in
derivation (defaultAttrs // attrs)
</pre>
<br />
Ok now we have to remember a little about <a href="http://lethalman.blogspot.it/2014/07/nix-pill-5-functions-and-imports.html">Nix functions</a>. The whole nix expression of this autotools.nix file will evaluate to a function. This function accepts a parameter <span style="font-family: Courier New, Courier, monospace;">pkgs</span>, then returns a function which accepts a parameter <span style="font-family: Courier New, Courier, monospace;">attrs</span>.<br />
The body of the function is simple, yet at first sight it might be hard to grasp:<br />
<ol>
<li>First drop in the scope the magic <span style="font-family: Courier New, Courier, monospace;">pkgs</span> attribute set.</li>
<li>Within a let expression we define an helper variable, <span style="font-family: Courier New, Courier, monospace;">defaultAttrs</span>, which serves as a set of common attributes used in derivations.</li>
<li>Finally we create the derivation with that strange expression, (<span style="font-family: Courier New, Courier, monospace;">defaultAttrs // attrs</span>).</li>
</ol>
<div>
The <a href="http://nixos.org/nix/manual/#idm47361539098656">// operator</a> is an operator between two sets. The result is the union of the two sets. In case of conflicts between attribute names, the value on the right set is preferred.</div>
<div>
So we use <span style="font-family: Courier New, Courier, monospace;">defaultAttrs</span> as base set, and add (or override) the attributes from <span style="font-family: Courier New, Courier, monospace;">attrs</span>.</div>
<div>
<br /></div>
<div>
A couple of examples ought to be enough to clear out the behavior of the operator:</div>
<pre>nix-repl> { a = "b"; } // { c = "d"; }
{ a = "b"; c = "d"; }
nix-repl> { a = "b"; } // { a = "c"; }
{ a = "c"; }
</pre>
<div>
<br />
Complete the new builder.sh by adding <span style="font-family: Courier New, Courier, monospace;">$baseInputs</span> in the <span style="font-family: Courier New, Courier, monospace;">for</span> loop together with <span style="font-family: Courier New, Courier, monospace;">$buildInputs</span>. As you noticed, we passed that new variable in the derivation. Instead of merging buildInputs with the base ones, we prefer to preserve buildInputs as seen by the caller, so we keep them separated. Just a matter of choice.<br />
<br />
Then we rewrite hello.nix as follows:<br />
<pre>let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
name = "hello";
src = ./hello-2.9.tar.gz;
}
</pre>
Finally! We got a very simple description of a package! A couple of remarks that you may find useful to keep understanding the nix language:<br />
<ul>
<li>We assigned to pkgs the import that we did in the previous expressions in the "with", don't be afraid. It's that straightforward.</li>
<li>The mkDerivation variable is a nice example of partial application, look at it as <span style="font-family: Courier New, Courier, monospace;">(import ./autotools.nix) pkgs</span><span style="font-family: inherit;">. First we import the expression, then we apply the </span><span style="font-family: Courier New, Courier, monospace;">pkgs</span><span style="font-family: inherit;"> parameter. That will give us a function that accepts the attribute set </span><span style="font-family: Courier New, Courier, monospace;">attrs</span><span style="font-family: inherit;">.</span></li>
<li><span style="font-family: inherit;">We create the derivation specifying only name and src. If the project eventually needed other dependencies to be in PATH, then we would simply add those to buildInputs (not specified in hello.nix because empty).</span></li>
</ul>
Note we didn't use any other library. Special C flags may be needed to find include files of other libraries at compile time, and ld flags at link time.<br />
<br />
<h3>
Conclusion</h3>
</div>
<div>
<br /></div>
<div>
Nix gives us the bare metal tools for creating derivations, setting up a build environment and storing the result in the nix store.</div>
<div>
<br /></div>
<div>
Out of this we managed to create a generic builder for autotools projects, and a function <span style="font-family: Courier New, Courier, monospace;">mkDerivation</span><span style="font-family: inherit;"> that composes by default the common components used in autotools projects instead of repeating them in all the packages we would write.</span></div>
<div>
<br /></div>
<div>
We are feeling the way a Nix system grows up: it's about creating and composing derivations with the Nix language.</div>
<div>
<br /></div>
<div>
<u>Analogy</u>: in C you create objects in the heap, and then you compose them inside new objects. Pointers are used to refer to other objects.</div>
<div>
In Nix you create derivations stored in the nix store, and then you compose them by creating new derivations. Store paths are used to refer to other derivations.</div>
<div>
<br /></div>
<h3>
Next pill</h3>
<div>
<br />
...we will talk a little about runtime dependencies. Is the GNU hello world package self-contained? What are its runtime dependencies? We only specified build dependencies by means of using other derivations in the "hello" derivation.<br />
<br />
I'm going on vacation right now, so I may not be able to keep up with new pills for a few weeks. Have fun with Nix in the while :-)<br />
<br />
Pill 9 available for <a href="http://lethalman.blogspot.it/2014/08/nix-pill-9-automatic-runtime.html">reading here</a>.</div>
<div>
<br />
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com10tag:blogger.com,1999:blog-32054652.post-18343230060722734972014-07-31T01:58:00.000-07:002014-08-01T08:29:08.017-07:00Nix pill 7: a working derivationWelcome to the seventh Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/07/nix-pill-6-our-first-derivation.html">sixth pill</a> we introduced the notion of derivation in the Nix language. How to define a raw derivation and how to (try) to build it.<br />
<br />
In this post, we will continue along the path, by creating a derivation that successfully builds something.<br />
<div>
Then we try to package a real program: we compile a simple C file and create a derivation out of it, given a blessed toolchain.</div>
<div>
<br /></div>
<div>
I remind you how to enter the Nix environment: <span style="font-family: Courier New, Courier, monospace;">source ~/.nix-profile/etc/profile.d/nix.sh</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<h3>
Using a script as builder</h3>
<div>
<div>
<div>
<br /></div>
<div>
What's the easiest way to run a sequence of commands for building something? A bash script. We write a custom bash script, and we want it to be our builder.</div>
<div>
Given a builder.sh, we want the derivation to run <span style="font-family: Courier New, Courier, monospace;">bash builder.sh</span><span style="font-family: inherit;">.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">We don't use hash bangs in builder.sh, because at the time we are writing builder.sh we do not know the path to bash in the nix store. </span>Yes, even bash is in the nix store, everything is in there.</div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">We don't even use </span><span style="font-family: Courier New, Courier, monospace;">/usr/bin/env</span><span style="font-family: inherit;">, because then we lose the cool stateless property of Nix. Not to say </span><span style="font-family: Courier New, Courier, monospace;">PATH</span><span style="font-family: inherit;"> gets cleared when building therefore it wouldn't work anyway.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
In summary: we want the builder to be bash, and pass it an argument, builder.sh . Turns out the <span style="font-family: Courier New, Courier, monospace;">derivation</span> function accepts an optional <span style="font-family: Courier New, Courier, monospace;">args</span> attribute that is exactly used to pass arguments to the builder executable.</div>
<div>
<br /></div>
<div>
First of all, let's write our builder.sh in the current directory:</div>
<pre>declare -xp
echo foo > $out
</pre>
<div>
Ok, let's get the hang of it. I remind you from the previous pill, Nix creates the out path (not physically, you know the path) of the derivation. In the .drv there's a list of environment variables passed to the builder. One of them is $out.</div>
<div>
What we have to do is to create something in $out, be it a file or a directory. In this case we are creating a file.</div>
<div>
<br /></div>
<div>
In addition, we also debug the environment variables during the build process. We cannot use <span style="font-family: Courier New, Courier, monospace;">env</span>, because <span style="font-family: Courier New, Courier, monospace;">env</span> is part of coreutils and we don't have a dependency to it. Not yet. It's plain bash, only bash.</div>
<div>
<br /></div>
<div>
Like for coreutils in the previous pill, we get a blessed bash for free from our magic nixpkgs stuff:</div>
<pre>nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> "${bash}"
"/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-<b>bash-4.2-p45</b>"
</pre>
</div>
</div>
<div>
<br /></div>
<div>
Great, with the usual trick we can then refer to bin/bash and create our derivation:<br />
<pre>nix-repl> d = derivation { name = "foo"; builder = "${bash}/bin/bash"; args = [ ./builder.sh ]; system = builtins.currentSystem; }
nix-repl> :b d
these derivations will be built:
/nix/store/ybnysdh5k6cjznhg4afjgbhr6czbwb4s-<b>foo.drv</b>
building path(s) `/nix/store/72v14vk4li47n8sx3z2ibd802ihpqyvx-<b>foo</b>'
these derivations will be built:
/nix/store/ibwr68l3rjlx02kgz61dkkkrlpgljfxd-simple.drv
[...]
this derivation produced the following outputs:
out -> /nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-<b>foo</b>
</pre>
<br />
What? We did it! The contents of <span style="font-family: Courier New, Courier, monospace;">/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-<b>foo</b></span> is really foo. We built our first derivation.<br />
<br />
Note: we used <span style="font-family: Courier New, Courier, monospace;">./builder.sh</span>, not <span style="font-family: Courier New, Courier, monospace;">"./builder.sh"</span>. This way it gets parsed as path and Nix does wonders with it as we'll see later. Try using the string version, it will say it cannot find ./builder.sh , because that would be relative to the temporary build directory.<br />
<br />
<h3>
The builder environment</h3>
<br />
Let's inspect those debugged environment variables during the build process.<br />
<ul>
<li><span style="font-family: Courier New, Courier, monospace;">$HOME</span> is not your home, and /homeless-shelter doesn't exist at all. We force packages to not depend upon <span style="font-family: Courier New, Courier, monospace;">$HOME</span> during the build process.</li>
<li><span style="font-family: Courier New, Courier, monospace;">$PATH</span> plays the same game of <span style="font-family: Courier New, Courier, monospace;">$HOME</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">$NIX_BUILD_CORES</span> and <span style="font-family: Courier New, Courier, monospace;">$NIX_STORE</span> are <a href="http://nixos.org/nix/manual/#sec-conf-file">nix configurations</a></li>
<li><span style="font-family: Courier New, Courier, monospace;">$PWD</span> and <span style="font-family: Courier New, Courier, monospace;">$TMP</span> clearly shows nix created a temporary build directory.</li>
<li>Then builder, name, out and system are variables set due to the .drv contents.</li>
</ul>
</div>
<div>
And that's how we used the <span style="font-family: Courier New, Courier, monospace;">$out</span> variable in our derivation, put stuff inside it. It's like Nix reserved a slot in the nix store for us, and we must fill it.<br />
<br />
In terms of autotools, that will be the --prefix path. Yes, not the make DESTDIR, but the --prefix. That's a big difference between Nix and other package managers. That's the essence of stateless packaging. You don't install the package in a global common path under <span style="font-family: Courier New, Courier, monospace;">/</span>, you install it in a local isolated path under your nix store slot.<br />
<br />
<h3>
The .drv contents</h3>
<br />
We added something else this time to the derivation. The <span style="font-family: Courier New, Courier, monospace;">args</span> attribute. Let's see how this changed the .drv compared to the previous pill:<br />
<pre>$ pp-aterm -i /nix/store/g6jj1mjzq68i66rbqyb3gpx3k0x606af-<b>foo.drv</b>
Derive(
[("out", "/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-<b>foo</b>", "", "")]
, [("/nix/store/jdggv3q1sb15140qdx0apvyrps41m4lr-<b>bash-4.2-p45.drv</b>", ["out"])]
, ["/nix/store/5d1i99yd1fy4wkyx85iz5bvh78j2j96r-<b>builder.sh</b>"]
, "x86_64-linux"
, "/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-<b>bash-4.2-p45</b>/bin/bash"
, ["/nix/store/5d1i99yd1fy4wkyx85iz5bvh78j2j96r-<b>builder.sh</b>"]
, [ ("builder", "/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-<b>bash-4.2-p45</b>/bin/bash")
, ("name", "foo")
, ("out", "/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-<b>foo</b>")
, ("system", "x86_64-linux")
]
)
</pre>
<br />
Perfect, much like the usual .drv, except there's a list of arguments in there passed to the builder (bash), with the builder.sh... builder.sh... what? It's not pointing to my home's builder.sh .<br />
Nix automatically copies files or directories needed for the build in the nix store, to ensure, for example, that they do not get changed during the build process. Also to ensure the deployment to be stateless and independent of the building machine.<br />
<br />
Not only builder.sh is in the arguments passed to the builder, it's also in the input derivations.<br />
<br />
Being builder.sh a plain file, it has no .drv associated with it. The store path will be computed based on the hash of its contents, and the name itself. We will talk about store paths at some point, in a dedicated pill.<br />
<br />
<h3>
Packaging a simple C executable</h3>
<div>
<br /></div>
<div>
Start off writing a simple.c file:</div>
<pre>void main () {
puts ("Simple!");
}
</pre>
<div>
And its simple_builder.sh:<br />
<pre>export PATH="$coreutils/bin:$gcc/bin"
mkdir $out
gcc -o $out/simple $src
</pre>
Don't spend time understanding where those variables come from. Let's write the derivation and build it:<br />
<pre>nix-repl> :l <nixpkgs>
nix-repl> simple = derivation { name = "simple"; builder = "${bash}/bin/bash"; args = [ ./simple_builder.sh ]; gcc = gcc; coreutils = coreutils; src = ./simple.c; system = builtins.currentSystem; }
nix-repl> :b simple
</pre>
<pre>this derivation produced the following outputs:</pre>
out -> /nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-<b>simple</b><br />
<br />
Perfect, now you can run <span style="font-family: Courier New, Courier, monospace;">/nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-<b>simple</b>/simple</span> in your shell.<br />
<br />
<b>Explanation</b><br />
<br />
We added two new attributes to the derivation call, gcc and coreutils. Please, don't get an headache by reading "<span style="font-family: Courier New, Courier, monospace;">gcc = gcc</span><span style="font-family: inherit;">"</span>. On the left, it's the attribute name of the set. On the right, there's an expression, it's the gcc derivation. Same goes for coreutils.<br />
<br />
We also added the src attribute, nothing magic it's just a name with the <span style="font-family: Courier New, Courier, monospace;">./simple.c</span> path. Like for simple_builder.sh, simple.c will be added to the store.<br />
<br />
The trick: every attribute in the set will be converted to a string and passed as environment variable to the builder. Now it's all clear. $coreutils and $gcc are then the out paths of the derivations, and of course appending "/bin" will point to their binaries.<br />
<br />
Same goes for the src variable, $src is the path to simple.c in the nix store. As an exercise, pretty print the .drv file. You'll see in the input derivations simple_builder.sh and simple.c files, then bash, gcc and coreutils .drv files. Plus the new environment variables described above.<br />
<br />
In simple_builder.sh we set the PATH for gcc and coreutils binaries, so that gcc can find the necessary binaries like "cat", "readlink", ecc. .<br />
Then we create $out as a directory and inside it we put the binary.<br />
<br />
Note: instead of running plain gcc (or mkdir), it would have been equivalent to run <span style="font-family: Courier New, Courier, monospace;">$gcc/bin/gcc</span> (or <span style="font-family: Courier New, Courier, monospace;">$coreutils/bin/mkdir</span>).<br />
<br />
<h3>
Enough with nix-repl</h3>
</div>
<br />
Drop out of nix-repl, write a simple.nix file:<br />
<pre>with (import <nixpkgs> {});
derivation {
name = "simple";
builder = "${bash}/bin/bash";
args = [ ./simple_builder.sh ];
inherit gcc coreutils;
src = ./simple.c;
system = builtins.currentSystem;
}
</pre>
</div>
Now you can build it with <span style="font-family: Courier New, Courier, monospace;">nix-build simple.nix</span><span style="font-family: inherit;">. It will create a symlink "result" in the current directory, pointing to the out path of the derivation.</span><br />
<br />
The <a href="http://nixos.org/nix/manual/#sec-nix-build">nix-build</a> tool does two main jobs:<br />
<ul>
<li><a href="http://nixos.org/nix/manual/#sec-nix-instantiate">nix-instantiate</a>: parse simple.nix and return the .drv file relative to the parsed derivation set</li>
<li><a href="http://nixos.org/nix/manual/#rsec-nix-store-realise">nix-store -r</a>: realise the .drv, which actually builds the derivation.</li>
</ul>
Finally creates the symlink.<br />
<br />
Look the first line of the .nix file. We have an "<span style="font-family: Courier New, Courier, monospace;">import</span>" function call nested in a "<span style="font-family: Courier New, Courier, monospace;">with</span>" expression. I recall <span style="font-family: Courier New, Courier, monospace;">import</span> accepts one argument, a nix file to parse. In this case it parsed a function out of the file.<br />
Afterwards we call the parsed function with the empty set. We saw this already in <a href="http://lethalman.blogspot.it/2014/07/nix-pill-5-functions-and-imports.html">nix pill 5</a>.<br />
<br />
Let me underline it: "<span style="font-family: Courier New, Courier, monospace;">import <nixpkgs> {}</span>" are two function calls, not one. Read it like "<span style="font-family: Courier New, Courier, monospace;">(import <nixpkgs>) {}</span>".<br />
<br />
The final returned value of that import is a set. To simplify it: it's a set of derivations. Using the "with" expression we drop them into the scope. We basically simulated what <span style="font-family: Courier New, Courier, monospace;">:l</span> does in nix-repl, so we can easily access derivations such as bash, gcc and coreutils.<br />
<br />
Then we meet the <a href="http://nixos.org/nix/manual/#idm47361539166560"><span style="font-family: Courier New, Courier, monospace;">inherit</span> keyword</a>. Doing <span style="font-family: Courier New, Courier, monospace;">inherit foo</span>, is the same as doing <span style="font-family: Courier New, Courier, monospace;">foo = foo</span>. Doing <span style="font-family: Courier New, Courier, monospace;">inherit foo bar</span>, is the same as doing <span style="font-family: Courier New, Courier, monospace;">foo = foo; bar = bar</span>. Literally.<br />
This syntax only makes sense inside sets. Don't think it's black magic, it's just a convenience to avoid repeating the same name twice, once for the attribute name, once for the variable in the scope.<br />
<br />
<h3>
Next pill</h3>
<div>
<br />
...we will generalize the builder. If you have noticed, we have written two builder.sh files in this post. We would like to have a generic builder script instead, especially since every builder script goes in the nix store: that's a waste.<br />
<br />
Again. <b>Is it really that hard to package stuff in Nix? No</b>, here we're studying the fundamentals of Nix.<br />
<br />
Pill 8 is available for <a href="http://lethalman.blogspot.it/2014/08/nix-pill-8-generic-builders.html">reading here</a>.</div>
<div>
<br />
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com11tag:blogger.com,1999:blog-32054652.post-42268201811776793522014-07-29T08:41:00.002-07:002014-07-31T01:59:42.499-07:00Nix pill 6: our first derivationWelcome to the sixth Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/07/nix-pill-5-functions-and-imports.html">fifth pill</a> we introduced functions and imports. Functions and imports are very simple concepts that allows for building complex abstractions and composition of modules to build a flexible Nix system.<br />
<br />
In this post we finally arrived to writing a derivation. Derivations are the building blocks of a Nix system, from a file system view point. The Nix language is used to describe such derivations.<br />
<br />
I remind you how to enter the Nix environment: <span style="font-family: Courier New, Courier, monospace;">source ~/.nix-profile/etc/profile.d/nix.sh</span><br />
<br />
<h3>
The derivation function</h3>
<div>
<br /></div>
<div>
The <a href="http://nixos.org/nix/manual/#ssec-derivation">derivation built-in function</a> is used to create derivations. I invite you to read the link in the Nix manual about the derivation built-in. A derivation from a Nix language view point is simply a set, with some attributes. Therefore you can pass the derivation around with variables like anything else.</div>
<div>
That's where the real power comes in. </div>
<div>
<br /></div>
<div>
The <span style="font-family: Courier New, Courier, monospace;">derivation</span> function receives a set as first argument. This set requires at least the following three attributes:</div>
<div>
<ul>
<li><span style="font-family: Courier New, Courier, monospace;">name</span>: the name of the derivation. In the nix store the format is <span style="font-family: Courier New, Courier, monospace;">hash-name</span>, that's the name.</li>
<li><span style="font-family: Courier New, Courier, monospace;">system</span>: is the name of the system in which the derivation can be built. For example, x86_64-linux.</li>
<li><span style="font-family: Courier New, Courier, monospace;">builder</span>: it is the binary program that builds the derivation.</li>
</ul>
<div>
First of all, what's the name of our system as seen by nix?</div>
</div>
<pre>nix-repl> builtins.currentSystem
"x86_64-linux"
</pre>
<div>
Let's try to fake the name of the system:</div>
<pre>nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> d
«derivation /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv»
</pre>
<div>
Oh oh, what's that? Did it build the derivation? No it didn't, but it <b>did create the .drv file</b>. nix-repl does not build derivations unless you tell to do so.</div>
<div>
<br /></div>
<h3>
Digression about .drv files</h3>
<div>
<br /></div>
<div>
What's that .drv file? It is the specification of how to build the derivation, without all the Nix language fuzz.</div>
<div>
Before continuing, some analogies with the C language:</div>
<div>
<ul>
<li>.nix files are like .c files</li>
<li>.drv files are intermediate files like .o files. The .drv describes how to build a derivation, it's the bare minimum information.</li>
<li>out paths are then the product of the build</li>
</ul>
<div>
Both drv paths and out paths are stored in the nix store as you can see.</div>
</div>
<div>
<br /></div>
<div>
What's in that .drv file? You can read it, but it's better to pretty print it.</div>
<pre>$ nix-env -i strategoxt
</pre>
<div>
If you feel the above command being too slow (ignore the meaning of the command below, just do it):</div>
<pre>$ nix-env -iA strategoPackages018.strategoxt -f '<nixpkgs>'
</pre>
<div>
The installed <span style="font-family: Courier New, Courier, monospace;">pp-aterm</span> program can be used to pretty print .drv files:</div>
<pre>$ pp-aterm -i /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
Derive(
[("out", "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname", "", "")]
, []
, []
, "mysystem"
, "mybuilder"
, []
, [ ("builder", "mybuilder")
, ("name", "myname")
, ("out", "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname")
, ("system", "mysystem")
]
</pre>
<div>
Ok we can see there's an out path, but it does not exist yet. We never told Nix to build it, but we know beforehand where the build output will be. Why?<br />
<br />
Think, if Nix ever built the derivation just because we accessed it in Nix, we would have to wait a long time if it was, say, Firefox. That's why Nix let us know the path beforehand and keep evaluating the Nix expressions, but it's still empty because no build was ever made.<br />
<br />
<u>Important</u>: the hash of the out path is based solely on the input derivations in the current version of Nix, not on the contents of the build product. It's possible however to have <a href="http://en.wikipedia.org/wiki/Content-addressable_storage">content-addressable</a> derivations for e.g. tarballs as we'll see later on.<br />
<br /></div>
<div>
Many things are empty in that .drv, however I write a summary of the <a href="http://nixos.org/~eelco/pubs/phd-thesis.pdf">.drv format</a> for you:</div>
<div>
<ol>
<li>The output paths (they can be multiple ones). By default nix creates one out path called "out".</li>
<li>The list of input derivations. It's empty because we are not referring to any other derivation. Otherwise, there would a list of other .drv files.</li>
<li>The system and the builder executable (yes, it's a fake one).</li>
<li>Then a list of environment variables passed to the builder.</li>
</ol>
<div>
That's it, the minimum necessary information to build our derivation.<br />
<br />
<u>Important note</u>: the environment variables passed to the builder are just those you see in the .drv plus some other Nix related configuration (number of cores, temp dir, ...). The builder will not inherit any variable from your running shell, otherwise builds would suffer from <a href="https://wiki.debian.org/ReproducibleBuilds">non-determinism</a>.</div>
</div>
<div>
<br /></div>
<h3>
Back to our fake derivation</h3>
<div>
<br /></div>
<div>
<div>
Let's build our really fake derivation:</div>
<pre>nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> :b d
[...]
these derivations will be built:
/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-<b>myname.drv</b>
building path(s) `/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-<b>myname</b>'
error: a `mysystem' is required to build `/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-<b>myname.drv</b>', but I am a `x86_64-linux'
</pre>
<div>
The :b is a nix-repl specific command to build a derivation. You can see more commands with :? .</div>
<div>
So in the output you can see that it takes the .drv as information on how to build the derivation. Then it says it's trying to produce our out path. Finally the error we were waiting for: that derivation can't be built on our system.</div>
<div>
<br /></div>
<div>
We're doing the build inside nix-repl, but what if we don't want to use nix-repl?</div>
<div>
You can <b>realise</b> a .drv with:</div>
<pre>$ nix-store -r /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-<b>myname.drv</b>
</pre>
<div>
You will get the same output as before.</div>
<div>
<br /></div>
<div>
Let's fix the system attribute:</div>
<pre>nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = builtins.currentSystem; }
nix-repl> :b d
[...]
build error: invalid file name `mybuilder'
</pre>
<div>
A step forward: of course, that "mybuilder" executable does not really exist. Stop for a moment.</div>
</div>
<div>
<br /></div>
<h3>
What's in a derivation set</h3>
<div>
<br /></div>
<div>
I find useful to inspect the returned value from the derivation function for you.</div>
<div>
First of all, the returned value is a plain set:</div>
<pre>nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> builtins.isAttrs d
true
nix-repl> builtins.attrNames d
[ "all" "builder" "drvAttrs" "drvPath" "name" "out" "outPath" "outputName" "system" "type" ]
</pre>
<div>
You can guess what <span style="font-family: Courier New, Courier, monospace;">builtins.isAttrs</span> does, it returns true if the argument is a set. While <span style="font-family: Courier New, Courier, monospace;">builtins.attrNames</span> returns a list of keys of the given set. Some kind of reflection, you might say.</div>
<div>
<br /></div>
<div>
Start from drvAttrs:</div>
<pre>nix-repl> d.drvAttrs
{ builder = "mybuilder"; name = "myname"; system = "mysystem"; }
</pre>
<div>
That's basically the input we gave to the derivation function. Also d.name, d.system and d.builder attributes are straight the ones we gave as input.</div>
<pre>nix-repl> (d == d.out)
true
</pre>
<div>
So <span style="font-family: Courier New, Courier, monospace;">out</span> is just the derivation itself, it seems weird but the reason is that we only have one output from the derivation. That's also the reason why <span style="font-family: Courier New, Courier, monospace;">d.all</span> is a singleton. We'll see multiple outputs later.</div>
<div>
<br /></div>
<div>
The d.drvPath is the path of the .drv file: /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-<b>myname.drv </b>.</div>
<div>
<br /></div>
<div>
Something interesting is the <span style="font-family: Courier New, Courier, monospace;">type</span> attribute. It's "derivation". Nix does add a little of magic to sets with type derivation, but not that much. To let you understand, you can create yourself a set with that type, it's a simple set:</div>
<pre>nix-repl> { type = "derivation"; }
«derivation ???»
</pre>
<div>
Of course it has no other information, so Nix doesn't know what to say :-) But you get it, the type = "derivation" is just a convention for Nix and for us to understand the set is a derivation.</div>
<div>
<b><br /></b></div>
<div>
When writing packages, we are interested in the outputs. The other metadata is needed for Nix to know how to create the drv path and the out path.</div>
<div>
<div>
The outPath attribute is the build path in the nix store: /nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-<b>myname </b>.</div>
</div>
<div>
<b><br /></b></div>
<h3>
Referring to other derivations</h3>
<div>
<br /></div>
<div>
Just like dependencies in other package managers, how do we refer to other packages? How do we refer to other derivations in terms of files on the disk?</div>
<div>
We use the outPath. The outPath tells where the files are of that derivation. To make it more convenient, Nix is able to do a conversion from a derivation set to a string.</div>
<pre>nix-repl> d.outPath
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-<b>myname</b>"
nix-repl> builtins.toString d
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-<b>myname</b>"
</pre>
<div>
Nix does the "set to string conversion" as long as there is the outPath attribute (much like a toString method in other languages):</div>
<pre>nix-repl> builtins.toString { outPath = "foo"; }
"foo"
nix-repl> builtins.toString { a = "b"; }
error: cannot coerce a set to a string, at (string):1:1
</pre>
<div>
Say we want to use binaries from coreutils (ignore the nixpkgs etc.):</div>
<pre>nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> coreutils
«derivation /nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-<b>coreutils-8.21.drv</b>»
nix-repl> builtins.toString coreutils
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-<b>coreutils-8.21</b>"
</pre>
<div>
Apart the nixpkgs stuff, just think we added to the scope a series of variables. One of them is coreutils. It is the derivation of the coreutils package you all know of from other Linux distributions. </div>
<div>
It contains basic binaries for GNU/Linux systems (you may have multiple derivations of coreutils in the nix store, no worries):</div>
<pre>$ ls /nix/store/*coreutils*/bin
[...]
</pre>
<div>
I remind you, inside strings it's possible to interpolate Nix expressions with ${...}:</div>
<pre>nix-repl> "${d}"
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-<b>myname</b>"
nix-repl> "${coreutils}"
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-<b>coreutils-8.21</b>"
</pre>
<div>
That's very convenient, because then we could refer to e.g. the bin/true binary like this:</div>
<pre>nix-repl> "${coreutils}/bin/true"
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-<b>coreutils-8.21</b>/bin/true"
</pre>
<h3>
An almost working derivation</h3>
<div>
<br /></div>
<div>
In the previous attempt we used a fake builder, "mybuilder" which obviously does not exist. But we can use for example bin/true, which always exits with 0 (success).</div>
<pre>nix-repl> :l <nixpkgs>
nix-repl> d = derivation { name = "myname"; builder = "${coreutils}/bin/true"; system = builtins.currentSystem; }
nix-repl> :b d
[...]
builder for `/nix/store/d4xczdij7xazjfm5kn4nmphx63mpv676-<b>myname.drv</b>' failed to produce output path `/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-<b>myname</b>'
</pre>
<div>
Another step forward, it executed the builder (bin/true), but the builder did not create the out path of course, it just exited with 0.</div>
<div>
<br /></div>
<div>
<u>Obvious note</u>: everytime we change the derivation, a new hash is created.</div>
<div>
<br /></div>
<div>
Let's examine the new .drv now that we referred to another derivation:</div>
<div>
<br /></div>
<div>
<span style="font-family: inherit;">$ pp-aterm -i /nix/store/d4xczdij7xazjfm5kn4nmphx63mpv676-<b>myname.drv</b></span></div>
<div>
<div>
<span style="font-family: inherit;">Derive(</span></div>
<div>
<span style="font-family: inherit;"> [("out", "</span>/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-<b>myname</b><span style="font-family: inherit;">", "", "")]</span></div>
<div>
<span style="font-family: inherit;">, [("/nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-<b>coreutils-8.21.drv</b>", ["out"])]</span></div>
<div>
<span style="font-family: inherit;">, []</span></div>
<div>
<span style="font-family: inherit;">, "x86_64-linux"</span></div>
<div>
<span style="font-family: inherit;">, "/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-<b>coreutils-8.21</b>/bin/true"</span></div>
<div>
<span style="font-family: inherit;">, []</span></div>
<div>
<span style="font-family: inherit;">, [ ("builder", "/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-<b>coreutils-8.21</b>/bin/true")</span></div>
<div>
<span style="font-family: inherit;"> , ("name", "myname")</span></div>
<div>
<span style="font-family: inherit;"> , ("out", "/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-<b>myname</b>")</span></div>
<div>
<span style="font-family: inherit;"> , ("system", "x86_64-linux")</span></div>
<div>
<span style="font-family: inherit;"> ]</span></div>
<div style="font-weight: bold;">
<br /></div>
</div>
<div>
Aha! Nix added a dependency to our myname.drv, it's the coreutils.drv. Before doing our build, Nix should build the coreutils.drv. But since coreutils is already in our nix store, no build is needed, </div>
<div>
it's already there with out path /nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-<b>coreutils-8.21 </b>.<br />
<br />
<h3>
When is the derivation built</h3>
<br />
Nix does not build derivations <b>during evaluation</b> of Nix expressions. In fact, that's why we have to do ":b drv" in nix-repl, or use nix-store -r in the first place.<br />
<br />
An important separation is made in Nix:<br />
<ul>
<li><b>Instantiate/Evaluation time</b>: the Nix expression is parsed, interpreted and finally returns a derivation set. During evaluation, you can refer to other derivations because Nix will create .drv files and we will know out paths beforehand. This is achieved with <a href="http://nixos.org/nix/manual/#sec-nix-instantiate">nix-instantiate</a>.</li>
<li><b>Realise/Build time</b>: the .drv from the derivation set is built, first building .drv inputs (build dependencies). This is achieved with <a href="http://nixos.org/nix/manual/#rsec-nix-store-realise">nix-store -r</a>.</li>
</ul>
<div>
Think of it as of compile time and link time like with C/C++ projects. You first compile all source files to object files. Then link object files in a single executable.</div>
<div>
<br /></div>
<div>
In Nix, first the Nix expression (usually in a .nix file) is compiled to .drv, then each .drv is built and the product is installed in the relative out paths.</div>
<div>
<br /></div>
</div>
<h3>
Conclusion</h3>
<div>
<br /></div>
<div>
<b>Is that complicated to create a package for Nix? No it's not.</b><br />
<br />
We're walking through the fundamentals of Nix derivations, to understand how they work, how they are represented.</div>
<div>
Packaging in Nix is certainly easier than that, but we're not there yet in this post. More Nix pills are needed.</div>
<div>
<br /></div>
<div>
With the derivation function we provide a set of information on how to build a package, and we get back the information about where the package was built.</div>
<div>
Nix converts a set to a string when there's an outPath, that's very convenient. With that, it's easy to refer to other derivations.<br />
<br /></div>
<div>
When Nix builds a derivation, it first creates a .drv file from a derivation expression, and uses it to build the output. It does so recursively for all the dependencies (inputs). It "executes" the .drv files like a machine. Not much magic after all.</div>
<div>
<div>
<h3>
<br class="Apple-interchange-newline" />Next pill</h3>
</div>
<div>
<br /></div>
<div>
...we will finally write our first <b>working</b> derivation. Yes, this post is about "our first derivation", but I never said it was a working one
<img border="0" src="http://www.russianmachineneverbreaks.com/wp-content/uploads/2013/11/Troll_Face.png" style="height: 1em;" />
<br />
<br />
Pill 7 is available for <a href="http://lethalman.blogspot.it/2014/07/nix-pill-7-working-derivation.html">reading here</a>.<br />
<br />
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com25tag:blogger.com,1999:blog-32054652.post-43802900731462077472014-07-28T06:36:00.001-07:002014-07-29T08:41:51.507-07:00Nix pill 5: functions and importsWelcome to the fifth Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/07/nix-pill-4-basics-of-language.html">fourth pill</a> we touched the Nix language for a moment. We introduced basic types and values of the Nix language, and basic expressions such as "<span style="font-family: Courier New, Courier, monospace;">if</span>", "<span style="font-family: Courier New, Courier, monospace;">with</span>" and "<span style="font-family: Courier New, Courier, monospace;">let</span>". I invite you to re-read about these expressions and play with them in the repl.<br />
<div>
<br /></div>
<div>
Functions help to build reusable components in a big repository like <a href="https://github.com/NixOS/nixpkgs/">nixpkgs</a>. The Nix manual has a <a href="http://nixos.org/nix/manual/#idm47361539166560">great explanation of functions</a>. Let's go: pill on one hand, Nix manual on the other hand.</div>
<div>
<br />
I remind you how to enter the Nix environment: <span style="font-family: Courier New, Courier, monospace;">source ~/.nix-profile/etc/profile.d/nix.sh</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<h3>
Nameless and single parameter</h3>
<div>
<br /></div>
<div>
Functions are anonymous (lambdas), and only have a single parameter. The syntax is extremely simple. Type the parameter name, then "<span style="font-family: Courier New, Courier, monospace;">:</span>", then the body of the function.</div>
<pre>nix-repl> x: x*2
«lambda»
</pre>
<div>
So here we defined a function that takes a parameter <span style="font-family: Courier New, Courier, monospace;">x</span>, and returns <span style="font-family: Courier New, Courier, monospace;">x*2</span>. The problem is that we cannot use it in any way, because it's unnamed... joke!</div>
<div>
We can store functions in variables.</div>
<pre>nix-repl> double = x: x*2
nix-repl> double
«lambda»
nix-repl> double 3
6
</pre>
<div>
As usual, please ignore the special syntax for assignments inside nix-repl.</div>
<div>
So, we defined a function <span style="font-family: Courier New, Courier, monospace;">x: x*2</span> that takes one parameter <span style="font-family: Courier New, Courier, monospace;">x</span>, and returns <span style="font-family: Courier New, Courier, monospace;">x*2</span>. This function is then assigned to the variable <span style="font-family: Courier New, Courier, monospace;">double</span>.</div>
<div>
Finally we did our first function call: <span style="font-family: Courier New, Courier, monospace;">double 3</span>.</div>
<div>
<u>Big note</u>: it's not like many other programming languages where you write <span style="font-family: Courier New, Courier, monospace;">double(3)</span>. It really is <span style="font-family: Courier New, Courier, monospace;">double 3</span>.</div>
<div>
<br /></div>
<div>
In summary: to call a function, name the variable, then space, then the argument. Nothing else to say, it's as easy as that.</div>
<div>
<br /></div>
<h3>
More than one parameter</h3>
<div>
<br /></div>
<div>
How do we create a function that accepts more than one parameter? For people not used to functional programming, this may take a while to grasp. Let's do it step by step.</div>
<pre>nix-repl> mul = a: (b: a*b)
nix-repl> mul
«lambda»
nix-repl> mul 3
«lambda»
nix-repl> (mul 3) 4
12
</pre>
<div>
We defined a function that takes the parameter "<span style="font-family: Courier New, Courier, monospace;">a</span>", the body returns another function. This other function takes a parameter "<span style="font-family: Courier New, Courier, monospace;">b</span>" and returns <span style="font-family: Courier New, Courier, monospace;">a*b</span>.</div>
<div>
Therefore, calling "<span style="font-family: Courier New, Courier, monospace;">mul 3</span>" returns this kind of function: <span style="font-family: Courier New, Courier, monospace;">b: 3*b</span><span style="font-family: inherit;">. In turn, we call the returned function with 4, and get the expected result.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
You don't have to use parenthesis at all, Nix has sane priorities when parsing the code:</div>
<pre>nix-repl> mul = a: b: a*b
nix-repl> mul
«lambda»
nix-repl> mul 3
«lambda»
nix-repl> mul 3 4
12
nix-repl> mul (6+7) (8+9)
221
</pre>
<div>
Much more readable, you don't even notice that functions only receive one argument.</div>
<div>
Since the argument is separated by a space, to pass more complex expressions you need parenthesis. In other common languages you would write <span style="font-family: Courier New, Courier, monospace;">mul(6+7, 8+9)</span>.</div>
<div>
<br /></div>
<div>
Given that functions have only one parameter, it is straightforward to use <b>partial application</b>:</div>
<pre>nix-repl> foo = mul 3
nix-repl> foo 4
12
nix-repl> foo 5
15
</pre>
<div>
We stored the function returned by <span style="font-family: Courier New, Courier, monospace;">mul 3</span> into a variable <span style="font-family: Courier New, Courier, monospace;">foo</span>, then reused it.</div>
<div>
<br /></div>
<h3>
Arguments set</h3>
<div>
<br /></div>
<div>
Now this is a very cool feature of Nix. It is possible to pattern match over a set in the parameter. We write an alternative version of <span style="font-family: Courier New, Courier, monospace;">mul = a: b: a*b</span> first by using a set as argument, then using pattern matching.</div>
<pre>nix-repl> mul = s: s.a*s.b
nix-repl> mul { a = 3; b = 4; }
12
nix-repl> mul = { a, b }: a*b
nix-repl> mul { a = 3; b = 4; }
12
</pre>
<div>
In the first case we defined a function that accepts a single parameter. We then access attributes "<span style="font-family: Courier New, Courier, monospace;">a</span>" and "<span style="font-family: Courier New, Courier, monospace;">b</span>" from the given set.</div>
<div>
Note how the parenthesis-less syntax for function calls is very elegant in this case, instead of doing <span style="font-family: Courier New, Courier, monospace;">mul({ a=3; b=4; })</span><span style="font-family: inherit;"> in other languages.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">In the second case we defined an arguments set. It's like defining a set, except without values. We require that the passed set contains the keys "</span><span style="font-family: Courier New, Courier, monospace;">a</span><span style="font-family: inherit;">" and "</span><span style="font-family: Courier New, Courier, monospace;">b</span><span style="font-family: inherit;">". Then we can use those "</span><span style="font-family: Courier New, Courier, monospace;">a</span><span style="font-family: inherit;">" and "</span><span style="font-family: Courier New, Courier, monospace;">b</span><span style="font-family: inherit;">" in the function body directly.</span></div>
<pre>nix-repl> mul { a = 3; b = 4; c = 6; }
error: anonymous function at (string):1:2 called with unexpected argument `c', at (string):1:1
nix-repl> mul { a = 3; }
error: anonymous function at (string):1:2 called without required argument `b', at (string):1:1
</pre>
<div>
Only a set with exactly the attributes required by the function is accepted, nothing more, nothing less.</div>
<div>
<br /></div>
<h3>
Default and variadic attributes</h3>
<div>
<br /></div>
<div>
It is possible to specify <b>default values</b> of attributes in the arguments set:</div>
<pre>nix-repl> mul = { a, b ? 2 }: a*b
nix-repl> mul { a = 3; }
6
nix-repl> mul { a = 3; b = 4; }
12
</pre>
<div>
Also you can allow passing more attributes (<b>variadic</b>) than the expected ones:</div>
<pre>nix-repl> mul = { a, b, ... }: a*b
nix-repl> mul { a = 3; b = 4; c = 2; }
</pre>
<div>
However, in the function body you cannot access the "<span style="font-family: Courier New, Courier, monospace;">c</span>" attribute. The solution is to give a name to the given set with the<span style="font-family: inherit;"> <b>@-pattern</b></span>:</div>
<pre>nix-repl> mul = s@{ a, b, ... }: a*b*s.c
nix-repl> mul { a = 3; b = 4; c = 2; }
24
</pre>
<div>
That's it, you give a name to the whole parameter with <span style="font-family: Courier New, Courier, monospace;">name@</span> before the set pattern.</div>
<div>
<br /></div>
<div>
Advantages of using argument sets:</div>
<div>
<ul>
<li>Named unordered arguments: you don't have to remember the order of the arguments.</li>
<li>You can pass sets, that adds a whole new layer of flexibility and convenience. </li>
</ul>
</div>
<div>
Disadvantages:</div>
<div>
<ul>
<li>Partial application does not work with argument sets. You have to specify the whole attribute set, not part of it.</li>
</ul>
<div>
You may find similarities with <a href="https://docs.python.org/2/faq/programming.html#how-can-i-pass-optional-or-keyword-parameters-from-one-function-to-another">Python **kwargs</a>.<br />
<br /></div>
</div>
<h3>
Imports</h3>
<div>
<br /></div>
<div>
The "<span style="font-family: Courier New, Courier, monospace;">import</span>" function is built-in and provides a way to parse a .nix file. The natural approach is to define each component in a .nix file, then compose by importing these files.</div>
<div>
<br /></div>
<div>
Let's start with the bare metal.</div>
<div>
<br /></div>
<div>
a.nix:</div>
<pre>3
</pre>
<div>
b.nix:</div>
<pre>4
</pre>
<div>
mul.nix:</div>
<pre>a: b: a*b
</pre>
<pre>nix-repl> a = import ./a.nix
nix-repl> b = import ./b.nix
nix-repl> mul = import ./mul.nix
nix-repl> mul a b
12
</pre>
<div>
Yes it's really that straight. You import a file, and it gets parsed as expression. Note that the scope of the imported file does not inherit the scope of the importer.</div>
<div>
<br /></div>
<div>
test.nix:</div>
<pre>x
</pre>
<pre>nix-repl> let x = 5; in import ./test.nix
error: undefined variable `x' at /home/lethal/test.nix:1:1
</pre>
<div>
So how do we pass information to the module? Use functions, like we did with mul.nix .<br />
A more complex example:<br />
<br />
test.nix:<br />
<pre>{ a, b ? 3, trueMsg ? "yes", falseMsg ? "no" }:
if a > b
then builtins.trace trueMsg true
else builtins.trace falseMsg false
</pre>
<pre>nix-repl> import ./test.nix { a = 5; trueMsg = "ok"; }
trace: ok
true
</pre>
<div>
Explaining:</div>
<div>
<ul>
<li>In test.nix we return a function. It accepts a set, with default attributes <span style="font-family: Courier New, Courier, monospace;">b</span>, <span style="font-family: Courier New, Courier, monospace;">trueMsg</span> and <span style="font-family: Courier New, Courier, monospace;">falseMsg</span>.</li>
<li><span style="font-family: Courier New, Courier, monospace;">builtins.trace</span> is a <a href="http://nixos.org/nix/manual/#ssec-builtins">built-in function</a> that takes two arguments. The first is the message to display, the second is the value to return. It's usually used for debugging purposes.</li>
<li>Then we import test.nix, and call the function with that set.</li>
</ul>
<div>
So when is the message shown? Only when it's in need to be evaluated.</div>
</div>
<div>
<br /></div>
<div>
<div>
<h3>
Next pill</h3>
</div>
<div>
<br /></div>
<div>
...we will finally write our first derivation.<br />
<br />
Nix pill 6 is available for <a href="http://lethalman.blogspot.it/2014/07/nix-pill-6-our-first-derivation.html">reading here</a>.<br />
<br />
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
<br /></div>
</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com13tag:blogger.com,1999:blog-32054652.post-43653168397191253692014-07-26T10:52:00.000-07:002014-07-28T06:37:01.685-07:00Nix pill 4: the basics of the languageWelcome to the fourth Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/07/nix-pill-3-enter-environment.html">third pill</a> we entered the Nix environment. We installed software as user, managed the profile, switched between generations, and queried the nix store. That's the very basics of nix administration somehow.<br />
<br />
The <a href="http://nixos.org/nix/manual/#chap-writing-nix-expressions">Nix language</a> is used to write derivations. The <a href="http://nixos.org/nix/manual/#sec-nix-build">nix-build</a> tool is used to build derivations. Even as a system administrator that wants to customize the installation, it's necessary to master Nix. Using Nix for your jobs means you get the features we saw in the previous pills for free.<br />
<br />
The syntax is very uncommon thus looking at existing examples may lead to thinking that there's a lot of magic behind. In reality, it's only about writing utility functions for making things convenient.<br />
On the other hand, this same syntax is great for describing packages.<br />
<br />
<u>Important</u>: in Nix, everything is an expression, there are no statements. This is common to many functional languages.<br />
<u>Important</u>: values in Nix are immutable.<br />
<br />
<h3>
Value types</h3>
<div>
<br /></div>
<div>
We've installed nix-repl in the previous pill. If you didn't, <span style="font-family: Courier New, Courier, monospace;">nix-env -i nix-repl</span>. The nix-repl syntax is slightly different than nix syntax when it comes to assigning variables, but no worries. I prefer playing with nix-repl with you before cluttering your mind with more complex expressions.</div>
<div>
<br /></div>
<div>
Launch nix-repl. <span style="font-family: inherit;">First of all, nix supports basic arithmetic operations: +, -, and *. The integer division can be done with builtins.div.</span></div>
<pre>nix-repl> 1+3
4
nix-repl> builtins.div 6 3
2
</pre>
<span style="font-family: inherit;">Really, why doesn't nix have basic operations such as division? Because it's not needed for creating packages. Nix is not a general purpose language, it's a domain-specific language for writing packages.</span>
<br />
<div>
<span style="font-family: inherit;">Just think that builtins.div is not being used in the whole of our <a href="https://github.com/NixOS/nixpkgs/">nixpkgs repository</a>: it's useless.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
Other operators are ||, && and ! for booleans, and relational operators such as !=, ==, <, >, <=, >=. In Nix, <, >, <= and >= are not much used. There are also <a href="http://nixos.org/nix/manual/#table-operators">other operators</a> we will see in the course of this series.</div>
<div>
<br /></div>
<div>
<span style="font-family: inherit;">Nix has integer (not floating point), string, path, boolean and null </span><a href="http://nixos.org/nix/manual/#ssec-values" style="font-family: inherit;">simple types</a><span style="font-family: inherit;">. Then there are lists, sets and functions. These types are enough to build an operating system.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
Nix is strongly typed, but it's not statically typed. That is, you cannot mix strings and integers, you must first do the conversion. </div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
Try to use <span style="font-family: Courier New, Courier, monospace;">/</span> between two numbers:</div>
<pre>nix-repl> 2/3
/home/nix/2/3
</pre>
<div>
Nix parsed 2/3 as a relative path to the current directory. Paths are parsed as long as there's a slash. Therefore to specify the current directory, use <span style="font-family: Courier New, Courier, monospace;">./</span><span style="font-family: inherit;">.</span></div>
<div>
<span style="font-family: inherit;">In addition, Nix also parses urls.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">Not all urls or paths can be parsed this way. If a syntax error occurs, it's still possible to fallback to plain strings. Parsing urls and paths are convenient for additional safety.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<h3>
<span style="font-family: inherit;">Identifiers</span></h3>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">Not much to say, except that dash (-) is allowed in identifiers. That's convenient since many packages use dash in its name. In fact:</span></div>
<pre>nix-repl> a-b
error: undefined variable `a-b' at (string):1:1
nix-repl> a - b
error: undefined variable `a' at (string):1:1
</pre>
<div>
As you can see, <span style="font-family: Courier New, Courier, monospace;">a-b</span> is parsed as identifier, not as operation between a and b.</div>
<div>
<br /></div>
<h3>
<span style="font-family: inherit;">Strings</span></h3>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
It's important to understand the syntax for strings. When reading Nix expressions at the beginning, you may find dollars ($) ambiguous in their usage.</div>
<div>
Strings are enclosed by double quotes ("), or two single quotes ('').</div>
<pre>nix-repl> "foo"
"foo"
nix-repl> ''foo''
"foo"
</pre>
<div>
In python you can use also single quotes for strings like 'foo', but not in Nix.</div>
<div>
<br /></div>
<div>
It's possible to <a href="http://nixos.org/nix/manual/#ssec-values">interpolate</a> whole Nix expressions inside strings with ${...} and only with ${...}, not $foo or {$foo} or anything else.</div>
<pre>nix-repl> foo = "strval"
nix-repl> "$foo"
"$foo"
nix-repl> "${foo}"
"strval"
nix-repl> "${2+3}"
error: cannot coerce an integer to a string, at (string):1:2
</pre>
<div>
Note: ignore the foo = "strval" assignment, it's nix-repl special syntax.</div>
<div>
<br /></div>
<div>
As said previously, you cannot mix integers and strings. You explicitly need conversion. We'll see this later: function calls are another story.</div>
<div>
<br /></div>
<div>
Using the syntax with two single quotes, it's useful for writing double quotes inside strings instead of escaping:</div>
<pre>nix-repl> ''test " test''
"test \" test"
nix-repl> ''${foo}''
"strval"
</pre>
<div>
Escaping ${...} within double quoted strings is done with the backslash. Within two single quotes, it's done with '':</div>
<pre>nix-repl> "\${foo}"
"${foo}"
nix-repl> ''test ''${foo} test''
"test ${foo} test"
</pre>
<div>
No other magic about strings for now.<br />
<br /></div>
<div>
</div>
<h3>
Lists</h3>
<div>
<br /></div>
<div>
Lists are a sequence of expressions delimited by space (not comma):</div>
<pre>nix-repl> [ 2 "foo" true (2+3) ]
[ 2 "foo" true 5 ]
</pre>
<div>
Lists, like anything else in Nix, are immutable. Adding or removing elements from a list is possible, but will return a new list.</div>
<div>
<br /></div>
<h3>
Sets</h3>
<div>
<br /></div>
<div>
Sets are an association between a string key and a Nix expression. Keys can only be strings. When writing sets you can also use identifiers as keys.</div>
<pre>nix-repl> s = { foo = "bar"; a-b = "baz"; "123" = "num"; }
nix-repl> s
{ 123 = "num"; a-b = "baz"; foo = "bar"; }
</pre>
<div>
Note: here the string representation from nix is wrong, you can't write { 123 = "num"; } because 123 is not an identifier.</div>
<div>
You need semicomma (;) after every key-value assignment.</div>
<div>
<br /></div>
<div>
For those reading Nix expressions from nixpkgs: do not confuse sets with argument sets used in functions.</div>
<div>
<br /></div>
<div>
To access elements in the set:</div>
<pre>nix-repl> s.a-b
"baz"
nix-repl> s."123"
"num"
</pre>
<div>
Yes, you can use strings for non-identifiers to address keys in the set.</div>
<div>
<br /></div>
<div>
You cannot refer inside a set to elements of the same set:</div>
<pre>nix-repl> { a = 3; b = a+4; }
error: undefined variable `a' at (string):1:10
</pre>
<div>
To do so, use <a href="http://nixos.org/nix/manual/#idm47361539166560">recursive sets</a>:</div>
<pre>nix-repl> rec { a= 3; b = a+4; }
{ a = 3; b = 7; }
</pre>
<div>
This will be very convenient when defining packages.</div>
<div>
<br /></div>
<h3>
If expression</h3>
<div>
<br /></div>
<div>
Expressions, not statements.</div>
<pre>nix-repl> a = 3
nix-repl> b = 4
nix-repl> if a > b then "yes" else "no"
"no"
</pre>
<div>
You can't have only the "then" branch, you must specify also the "else" branch, because an expression must have a value in all cases.</div>
<div>
<br /></div>
<div>
<br /></div>
<h3>
Let expression</h3>
<div>
<br /></div>
<div>
This kind of expression is used to define local variables to inner expressions.</div>
<pre>nix-repl> let a = "foo"; in a
"foo"
</pre>
<div>
The syntax is: first assign variables, then "in" expression. The overall result will be the final expression after "in".</div>
<pre>nix-repl> let a = 3; b = 4; in a + b
7
</pre>
<div>
Let's write two let expressions, one inside the other:</div>
<pre>nix-repl> let a = 3; in let b = 4; in a + b
7
</pre>
<div>
With let you cannot assign twice to the same variable. You can however shadow outer variables:</div>
<pre>nix-repl> let a = 3; a = 8; in a
error: attribute `a' at (string):1:12 already defined at (string):1:5
nix-repl> let a = 3; in let a = 8; in a
8
</pre>
<div>
You cannot refer to variables in a let expression outside of it:</div>
<pre>nix-repl> let a = (let b = 3; in b); in b
error: undefined variable `b' at (string):1:31
</pre>
<div>
You can refer to variables in the let expression when assigning variables like with recursive sets:</div>
<pre>nix-repl> let a = 4; b = a + 5; in b
9
</pre>
<div>
So beware when you want to refer to a variable from the outer scope, but it's being defined in the current let expression. Same applies to recursive sets.</div>
<div>
<br /></div>
<h3>
With expression</h3>
<div>
<br /></div>
<div>
This kind of expression is something you hardly see in other languages. Think of it like a more granular "using" of C++, or "from module import *" from Python. You decide per-expression when to include symbols into the scope.</div>
<pre>nix-repl> longName = { a = 3; b = 4; }
nix-repl> longName.a + longName.b
7
nix-repl> with longName; a + b
7
</pre>
<div>
That's it, it takes a set and includes symbols in the scope of the inner expression. Of course, only valid identifiers from the set keys will be included.</div>
<div>
If a symbol exists in the outer scope and also in the "with" scope, it will <b>not</b> be shadowed. You can however still refer to the set:</div>
<pre>nix-repl> let a = 10; in with longName; a + b
14
nix-repl> let a = 10; in with longName; longName.a + b
7
</pre>
<h3>
Laziness</h3>
<div>
<br /></div>
<div>
Nix evaluates expression only <a href="http://en.wikipedia.org/wiki/Lazy_evaluation">when needed</a>. This is a great feature when working with packages.</div>
<pre>nix-repl> let a = builtins.div 4 0; b = 6; in b
6
</pre>
<div>
Since "a" is not needed, there's no error about division by zero, because the expression is not in need to be evaluated.</div>
<div>
That's why we can have all the packages <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix">defined here</a>, yet access to specific packages very fast.</div>
<div>
<br /></div>
<div>
<div>
<h3>
Next pill</h3>
</div>
<div>
<br /></div>
<div>
...we will talk about functions and imports. In this pill I've tried to avoid function calls as much as possible, otherwise the post would have been too long.<br />
<br />
Nix pill 5 is available for <a href="http://lethalman.blogspot.it/2014/07/nix-pill-5-functions-and-imports.html">reading here</a>.<br />
<br />
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com212tag:blogger.com,1999:blog-32054652.post-46872653290982328912014-07-25T07:37:00.000-07:002014-07-26T10:54:15.115-07:00Nix pill 3: enter the environmentWelcome to the third Nix pill. In the previous <a href="http://lethalman.blogspot.it/2014/07/nix-pill-2-install-on-your-running.html">second pill</a> we have installed Nix on our running system. Now we can finally play with it a little, things also apply to NixOS users.<br />
<div>
<br /></div>
<div>
<h3>
Enter the environment</h3>
<br />
In the previous pill we created a nix user, so let's start by switching user with <span style="font-family: Courier New, Courier, monospace;">su - nix</span><span style="font-family: inherit;">. If your ~/.profile got evaluated, then you should now be able to run commands like nix-env and nix-store.</span><br />
<span style="font-family: inherit;">If that's not the case:</span><br />
<pre>$ source ~/.nix-profile/etc/profile.d/nix.sh</pre>
<pre></pre>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">I remind you, </span><span style="font-family: Courier New, Courier, monospace;">~/.nix-profile/etc</span><span style="font-family: inherit;"> points to the nix-1.7 derivation. At this point, we are in our nix user profile.</span>
<br />
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;">Install something</span></h3>
</div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">Finally something practical! Installation in the nix environment is an interesting process. Let's install <a href="https://github.com/edolstra/nix-repl">nix-repl</a>, a simple command line tool for playing with the Nix language. Yes, Nix is a <a href="http://nixos.org/nix/manual/#idm47361539226272">pure, lazy, functional language,</a> not only a set of tools to manage derivations.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">Back to the installation:</span></div>
<pre>$ nix-env -i nix-repl<span style="font-family: Times New Roman;"><span style="white-space: normal;">
</span></span>
installing `nix-repl-1.7-1734e8a'
these paths will be fetched (18.61 MiB download, 69.53 MiB unpacked):
[...]
building path(s) `/nix/store/f01lfzbw7n0yzhsjd33xfj77li9raljv-<b>user-environment</b>'
created 24 symlinks in user environment
</pre>
<div>
Now you can run nix-repl. Things to notice:</div>
<div>
<ul>
<li>We did install software as user, only for the nix user.</li>
<li>It created a new user environment. That's a new generation of our nix user profile.</li>
<li>The <a href="http://nixos.org/nix/manual/#sec-nix-env">nix-env</a> tool manages environments, profiles and their generations.</li>
<li>We installed nix-repl by derivation name minus the version. I repeat: we did specify the <b>derivation name</b> (minus the version) to install.</li>
</ul>
</div>
<div>
We can list generations without walking through the /nix hierarchy:</div>
<pre>$ nix-env --list-generations
1 2014-07-24 09:23:30
2 2014-07-25 08:45:01 (current)
</pre>
<div>
List installed derivations:</div>
<pre>$ nix-env -q
nix-1.7
nix-repl-1.7-1734e8a
</pre>
<div>
So, where did nix-repl really got installed? <span style="font-family: Courier New, Courier, monospace;">which nix-repl</span> is <span style="font-family: Courier New, Courier, monospace;">~/.nix-profile/bin/nix-repl</span> which points to the store.</div>
<div>
We can also list the derivation paths with nix-env -q --out-path . So that's how those derivation paths are called: the <b>output</b> of a build.</div>
<div>
<br /></div>
<h3>
Path merging</h3>
<div>
<br /></div>
<div>
At this point you sure have the necessity to run "man". Even if you already have man system-wide outside of the nix environment, you can install and use it within nix with <span style="font-family: Courier New, Courier, monospace;">nix-env -i man</span>. As usual, a new generation will be created, and ~/.nix-profile will point to it.</div>
<div>
Let's inspect the <a href="http://nixos.org/nix/manual/#sec-profiles">profile</a> a bit:</div>
<pre>$ ls -l ~/.nix-profile/
dr-xr-xr-x 2 nix nix 4096 Jan 1 1970 bin
lrwxrwxrwx 1 nix nix 55 Jan 1 1970 etc -> /nix/store/clnpynyac3hx3a6z5lsy893p7b4rwnyf-<b>nix-1.7</b>/etc
[...]
</pre>
<div>
Now that's interesting. When only nix-1.7 was installed, bin/ was a symlink to nix-1.7. Now it's a real directory, no symlink.</div>
<pre>$ ls -l ~/.nix-profile/bin/
[...]
man -> /nix/store/83cn9ing5sc6644h50dqzzfxcs07r2jn-<b>man-1.6g</b>/bin/man
[...]
nix-env -> /nix/store/clnpynyac3hx3a6z5lsy893p7b4rwnyf-<b>nix-1.7</b>/bin/nix-env
[...]
nix-repl -> /nix/store/0fcl92chxbbs8axb994rg12vxddg1ivs-<b>nix-repl-1.7-1734e8a</b>/bin/nix-repl
[...]
</pre>
<div>
All clear. nix-env merged the paths from the installed derivations. <span style="font-family: Courier New, Courier, monospace;">which man</span> points to the nix profile, rather than the system man, because <span style="font-family: Courier New, Courier, monospace;">~/.nix-profile/bin</span> is at the head of <span style="font-family: Courier New, Courier, monospace;">$PATH</span>.</div>
<div>
<br /></div>
<h3>
Rollback / switch generation</h3>
<div>
<br /></div>
<div>
The last command installed "man". We should be at generation #3, unless you changed something in the middle. Let's say we want to rollback to the old generation:</div>
<pre>$ nix-env --rollback
switching from generation 3 to 2
</pre>
<div>
Now <span style="font-family: Courier New, Courier, monospace;">nix-env -q</span> does not list "man" anymore. <span style="font-family: Courier New, Courier, monospace;">ls -l `which man`</span> should now be your system installed one.</div>
<div>
Enough with the joke, let's go back to the last generation:</div>
<pre>$ nix-env -G 3
switching from generation 2 to 3
</pre>
<div>
I invite you to read the manpage of nix-env. nix-env requires an operation to perform, then there are common options for all operations, and there are options specific to an operation.</div>
<div>
<br /></div>
<div>
You can of course also <a href="http://nixos.org/nix/manual/#idm47361539520832">delete and upgrade packages</a>.</div>
<div>
<br /></div>
<h3>
Querying the store</h3>
<div>
<br /></div>
<div>
So far we learned how to query and manipulate the environment. But all of the environment components point to the store.</div>
<div>
To query and manipulate the store, there's the <a href="http://nixos.org/nix/manual/#sec-nix-store">nix-store</a> command. We can do neat things, but we'll only see some queries for now.</div>
<div>
<br /></div>
<div>
Show direct runtime dependencies of nix-repl:</div>
<pre>$ nix-store -q --references `which nix-repl`
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-<b>glibc-2.19</b>
/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-<b>gcc-4.8.3</b>
/nix/store/25lg5iqy68k25hdv17yac72kcnwlh4lm-<b>boehm-gc-7.2d</b>
/nix/store/g21di262aql6xskx15z3qiw3zh3wmjlb-<b>nix-1.7</b>
/nix/store/jxs2k83npdin18ysnd104xm4sy29m8xp-<b>readline-6.2</b>
</pre>
<div>
The argument to nix-store can be anything as long as it points to the nix store. It will follow symlinks.</div>
<div>
It may not make sense for you right now, but let's print reverse dependencies of nix-repl:</div>
<pre>$ nix-store -q --referrers `which nix-repl`
/nix/store/8rj57vahlndqwg4q095x5qvfa1g4rmvr-<b>env-manifest.nix</b>
/nix/store/9c8ak2h7c6vbr9kqk8i2n96bwg55br7y-<b>env-manifest.nix</b>
/nix/store/dr1y2saask2k4y2wrmxw443302pp02z2-<b>user-environment</b>
/nix/store/f01lfzbw7n0yzhsjd33xfj77li9raljv-<b>user-environment</b>
</pre>
<div>
Did you expect it? Our environments depend upon nix-repl. Yes, the environments are in the store, and since there are symlinks to nix-repl, therefore the environment depends upon nix-repl.</div>
<div>
It lists two environments, generation 2 and generation 3.<br />
<br />
The manifest.nix file contains metadata about the environment, such as which derivations are installed. So that nix-env can list them, upgrade or remove them. Guess what, the current manifest.nix can be found in <span style="font-family: Courier New, Courier, monospace;">~/.nix-profile/manifest.nix</span>.</div>
<div>
<br /></div>
<h3>
Closures</h3>
<div>
<br /></div>
<div>
The closure of a derivation is the list of all dependencies, recursively, down to the bare minimum necessary to use that derivation.</div>
<pre>$ nix-store -qR `which man`
[...]
</pre>
<div>
<br />
Copying all those derivations to the nix store of another machine makes you able to run "man" out of the box on that other machine. That's the base of nix deployment, you can already foresee the potential when deploying software in the cloud (hint: <span style="font-family: Courier New, Courier, monospace;">nix-copy-closure</span> and <span style="font-family: Courier New, Courier, monospace;">nix-store --export</span>).</div>
<div>
<br /></div>
<div>
A nicer view of the closure:</div>
<pre>$ nix-store -qR --tree `which man`
[...]
</pre>
<div>
<br />
With the above command, you can know exactly why a <b>runtime</b> dependency, being it direct or indirect, has been picked for a given derivation.<br />
<br />
Same applies to environments of course. As an exercise run <span style="font-family: Courier New, Courier, monospace;">nix-store -qR --tree ~/.nix-profile</span> , see that the first children are direct dependencies of the user environment: the installed derivations, and the manifest.nix.</div>
<div>
<br /></div>
<h3>
Dependency resolution</h3>
<div>
<br /></div>
<div>
There isn't anything like apt which solves a SAT problem in order to satisfy dependencies with lower and upper bounds on versions. Because there's no need. A derivation X depends on derivation Y, always.</div>
<div>
<br /></div>
<h3>
Fancy disrupt</h3>
<div>
<br /></div>
<pre>$ nix-env -e '*'
uninstalling `man-1.6g'
uninstalling `nix-repl-1.7-1734e8a'
uninstalling `nix-1.7'
[...]
</pre>
<div>
<br />
Ops, that uninstalled all derivations from the environment, including nix. We are not able to run nix-env, what now?</div>
<div>
Environments are a convenience for the user, but Nix is still there, in the store!</div>
<div>
<br /></div>
<div>
First pick one nix-1.7 derivation: <span style="font-family: Courier New, Courier, monospace;">ls /nix/store/*nix-1.7</span>, say /nix/store/g21di262aql6xskx15z3qiw3zh3wmjlb-<b>nix-1.7</b>.</div>
<div>
<br /></div>
<div>
The first possibility is to rollback:</div>
<pre>$ /nix/store/g21di262aql6xskx15z3qiw3zh3wmjlb-<b>nix-1.7</b>/bin/nix-env --rollback
</pre>
<div>
<br />
The second possibility is to install nix, thus creating a new generation:</div>
<pre>$ /nix/store/g21di262aql6xskx15z3qiw3zh3wmjlb-<b>nix-1.7</b>/bin/nix-env -i /nix/store/g21di262aql6xskx15z3qiw3zh3wmjlb-<b>nix-1.7</b>
</pre>
<div>
<b><br /></b></div>
<h3>
<b>Channels</b></h3>
<div>
<br /></div>
<div>
So where are we getting packages from? We said something already in <a href="http://lethalman.blogspot.it/2014/07/nix-pill-2-install-on-your-running.html">pill 2</a>. There's a list of channels from which we get packages, usually we use a single channel. The tool to manage channels is <a href="http://nixos.org/nix/manual/#sec-nix-channel">nix-channel</a>.</div>
<pre>$ nix-channel --list
nixpkgs http://nixos.org/channels/nixpkgs-unstable
</pre>
<div>
<br />
That's basically the contents of <span style="font-family: Courier New, Courier, monospace;">~/.nix-channels</span>. Note: <span style="font-family: Courier New, Courier, monospace;">~/.nix-channels</span> is not a symlink to the nix store!</div>
<div>
<br /></div>
<div>
To update the channel run <span style="font-family: Courier New, Courier, monospace;">nix-channel --update</span> . It will download the new nix expressions (descriptions of the packages), create a new generation of the channels profile and unpack under <span style="font-family: Courier New, Courier, monospace;">~/.nix-defexpr/channels</span> .</div>
<div>
That's much similar to apt-get update.</div>
<div>
<br /></div>
<h3>
Conclusion</h3>
<div>
<br /></div>
<div>
We learned how to query the user environment and to manipulate it by installing and uninstalling software. Upgrading software is as straight as it gets by reading <a href="http://nixos.org/nix/manual/#idm47361539520832">the manual</a> (<span style="font-family: Courier New, Courier, monospace;">nix-env -u '*'</span> will upgrade all packages in the environment).</div>
<div>
Everytime we change the environment, a new generation gets created. Switching between generations is easy and immediate.</div>
<div>
<br /></div>
<div>
Then we queried the store. We inspected the dependencies and reverse dependencies of store paths.</div>
<div>
We still see symlinks to compose paths from the nix store, our lovely trick.</div>
<div>
<br /></div>
<div>
Quick analogy with programming languages. You have the heap with all the objects, that's the nix store. You have objects that point to other objects, those are the derivations. Will be this the right path?</div>
<div>
<br /></div>
<div>
<h3>
Next pill</h3>
</div>
<div>
<br /></div>
<div>
...we will learn the basics of the Nix language. The Nix language is used to describe how to build derivations, and it's the base for everything else including NixOS. Therefore it's very important to understand the syntax and the semantics.<br />
<br />
Nix pill 4 is available for <a href="http://lethalman.blogspot.it/2014/07/nix-pill-4-basics-of-language.html">reading here</a>.<br />
<br />
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com5tag:blogger.com,1999:blog-32054652.post-73386666391976985972014-07-24T08:38:00.000-07:002014-07-25T09:29:47.331-07:00Nix pill 2: install on your running systemWelcome to the second Nix pill. In <a href="http://lethalman.blogspot.it/2014/07/nix-pill-1-why-you-should-give-it-try.html">the first pill</a> we briefly described Nix.<br />
Now we'll install Nix on our running system and understand what changed in our system after the installation.<br />
<br />
<a href="http://nixos.org/nix/manual/#chap-installation">Nix installation</a> is as easy as installing any other package. It will not revolutionize our system, it will stay in its own place out of our way.<br />
<br />
<h3>
Download</h3>
<br />
You can grab the last stable tarball (nix 1.7 during this writing) or the package for your distro here: <a href="http://hydra.nixos.org/release/nix/nix-1.7">http://hydra.nixos.org/release/nix/nix-1.7</a> .<br />
At the time you are reading, there may be nix 1.8. You can see a list of releases here: <a href="http://hydra.nixos.org/project/nix#tabs-releases">http://hydra.nixos.org/project/nix#tabs-releases</a> .<br />
<br />
I prefer using the precompiled binaries, because it's a nice self-contained environment, just works and it's a little less invasive (in that it does not install anything under /etc or /usr). So in this series I will use the nix bootstrap distribution.<br />
In the download page above you can find binaries for linux i686, x86_64 and darwin.<br />
<br />
<u>Note</u>: if you are using a rolling distribution however, I suggest not to install the package (for example on Debian sid) because you will experience broken perl dependencies when running nix tools.<br />
<br />
<h3>
<b>Installation</b></h3>
<div>
<b><br /></b></div>
To ensure we don't mess with the system, you could create a custom user to let him own the Nix store, let's call it "nix".<br />
<br />
As root:<br />
<pre># adduser nix
# mkdir -m 0755 /nix && chown nix /nix
</pre>
From now on, all the operations we do on the shell are done from this nix user:<br />
<pre># su - nix
$ tar -xf nix-1.7-x86_64-linux.tar.bz2
$ cd nix-1.7-x86_64-linux
$ ./install</pre>
<br />
My pills are not a simple tutorial, there's are several <a href="http://nixos.org/nix/manual/#chap-package-management">articles</a> <a href="https://www.domenkozar.com/2014/01/02/getting-started-with-nix-package-manager/">out</a> <a href="https://ocharles.org.uk/blog/posts/2014-02-04-how-i-develop-with-nixos.html">there</a> to learn the basics of nix and unix. We'll instead walk through the nix system to understand the fundamentals.<br />
<br />
First thing to note: those stuff in nix store refer to software in nix store itself. It doesn't use libc from our system or whatelse. It's a self-contained nix bootstrap.<br />
<br />
Quick note: in a normal installation, the store is owned by root and multiple users can install and build software through a nix daemon.<br />
<br />
<h3>
<b>So small nix store</b></h3>
<div>
<b><br /></b></div>
Start inspecting the output of the install command:<br />
<pre>copying Nix to /nix/store..........................
</pre>
Effectively, that's right the /nix/store we were talking in the first pill. The contents is the strictly necessary software to bootstrap a Nix system. You can see bash, core utils, the toolchain, perl libraries, sqlite and nix itself with its own tools and libnix.<br />
You surely noticed that in /nix/store not only directories are allowed, but also files, always in the form <span style="font-family: Courier New, Courier, monospace;"><i>hash</i>-<i>name</i></span>.<br />
<br />
<h3>
<b>The holy database</b></h3>
<div>
<b><br /></b></div>
Right after copying the store, the installation process initializes the database with the current information:<br />
<pre>initialising Nix database...
</pre>
Oh Nix also has a database. It's under <span style="font-family: Courier New, Courier, monospace;">/nix/var/nix/db</span>. It is an sqlite database that keeps track of the dependencies between derivations.<br />
The schema is very simple: there's a table of valid paths, mapping from auto increment integer to store path.<br />
Then there's a dependency relation from one path to other paths.<br />
Inspect it running <span style="font-family: Courier New, Courier, monospace;">/nix/store/*sqlite*/bin/sqlite3 /nix/var/nix/db/db.sqlite</span><br />
<br />
<b>Important rule</b>: never change /nix/store manually because that wouldn't be in sync with the sqlite db, unless you know what you are doing.<br />
<br />
<h3>
<b>The first profile</b></h3>
<div>
<b><br /></b></div>
Then we discover the <a href="http://nixos.org/nix/manual/#sec-profiles">profile</a> concept during the installation:<br />
<pre>creating /home/nix/.nix-profile
installing `nix-1.7'
building path(s) `/nix/store/xxhdgml3rshn8mkaqxb86gp4r276sp9d-<b>user-environment</b>'
created 6 symlinks in user environment
</pre>
A profile in nix is a general and very convenient concept for realizing rollbacks. Profiles are used to compose more components that are spread among multiple paths, under a new unified path. Not only, profiles are made up of multiple generations: they are versioned. Whenever you change a profile, a new generation is created.<br />
Generations thus can be switched and rollback-ed atomatically.<br />
<br />
Let's take a closer look at our profile:<br />
<br />
<pre>$ ls -l ~/.nix-profile/
bin -> /nix/store/clnpynyac3hx3a6z5lsy893p7b4rwnyf-<b>nix-1.7</b>/bin
[...]
manifest.nix -> /nix/store/82dg18wz250vvcxjbclgyy5yg2kfz1gw-<b>env-manifest.nix</b>
[...]
share -> /nix/store/clnpynyac3hx3a6z5lsy893p7b4rwnyf-<b>nix-1.7</b>/share
</pre>
That nix-1.7 derivation in the nix store is nix itself, with binaries and libraries. The installation basically reproduced the hierarchy of the nix-1.7 derivation in the profile by means of symbolic links.<br />
The contents of this profile are special, because only one program has been installed in our profile, therefore e.g. the bin directory fully points to the only program being installed.<br />
<br />
But that's only the contents of the latest generation of our profile. In fact, <span style="font-family: Courier New, Courier, monospace;">~/.nix-profile</span> itself is a symbolic link to <span style="font-family: Courier New, Courier, monospace;">/nix/var/nix/profiles/default</span> .<br />
In turn, that's a symlink to <span style="font-family: Courier New, Courier, monospace;">default-1-link</span><span style="font-family: inherit;"> in the same directory. Yes, that means it's the generation #1 of the default profile.</span><br />
Finally that's a symlink to the nix store "user-environment" derivation that you saw printed during the installation process.<br />
<br />
The manifest.nix will be meat for the next pill.<br />
<br />
<h3>
<b>Meet nixpkgs expressions</b></h3>
<div>
<b><br /></b></div>
More wild output from the installer:<br />
<pre>downloading Nix expressions from `http://releases.nixos.org/nixpkgs/nixpkgs-14.10pre46060.a1a2851/nixexprs.tar.xz'...
unpacking channels...
created 2 symlinks in user environment
modifying /home/nix/.profile...
</pre>
<br />
<div>
<a href="http://nixos.org/nix/manual/#chap-writing-nix-expressions">Nix expressions</a> are used to describe packages and how to build them. <a href="http://nixos.org/nixpkgs/">Nixpkgs</a> is the repository containing all these expressions: <a href="https://github.com/NixOS/nixpkgs">https://github.com/NixOS/nixpkgs</a> .<br />
The installer downloaded the package descriptions from commit a1a2851.<br />
<br />
The second profile we meet is the channels profile. <span style="font-family: Courier New, Courier, monospace;">~/.nix-defexpr/channels</span> points to <span style="font-family: Courier New, Courier, monospace;">/nix/var/nix/profiles/per-user/nix/channels</span> which points to <span style="font-family: Courier New, Courier, monospace;">channels-1-link</span> which points to a nix store directory containing the downloaded nix expressions.<br />
<br />
Channels are a set of packages and expressions available for download. Similar to debian stable and unstable, there's a stable and unstable channel. In this installation, we're tracking <a href="http://releases.nixos.org/nixpkgs/nixpkgs-14.10pre46060.a1a2851/">nixpkgs unstable</a>.<br />
<br />
Don't bother yet with nix expressions.<br />
<br />
Finally, for your own convenience, it modified <span style="font-family: Courier New, Courier, monospace;">~/.profile</span> to automatically enter the nix environment. What <span style="font-family: Courier New, Courier, monospace;">~/.nix-profile/etc/profile.d/nix.sh</span> really does is simply adding <span style="font-family: Courier New, Courier, monospace;">~/.nix-profile/bin</span> to <span style="font-family: Courier New, Courier, monospace;">PATH</span> and <span style="font-family: Courier New, Courier, monospace;">~/.nix-defexpr/channels/nixpkgs</span> to <span style="font-family: Courier New, Courier, monospace;">NIX_PATH</span>. We'll discuss about NIX_PATH in another pill.<br />
Read nix.sh, it's short.<br />
<br />
<h3>
FAQ: Can I change /nix to something else?</h3>
<div>
<br /></div>
You can, but there's a strong reason to keep using /nix instead of a different directory. All the derivations depend on other derivations by absolute path. I remind you in <a href="http://lethalman.blogspot.it/2014/07/nix-pill-1-why-you-should-give-it-try.html">pill 1</a> that bash pointed to a glibc under /nix/store.<br />
You can see now by yourself, don't worry if you see multiple bash derivations:<br />
<pre>$ ldd /nix/store/*bash*/bin/bash
[...]
</pre>
Keeping the store in /nix means we can grab the binary cache from nixos.org (just like you grab packages from debian mirrors) otherwise:<br />
<br />
<ol>
<li>glibc would be installed under /foo/store</li>
<li>thus bash needs to point to glibc under /foo/store</li>
<li>the binary cache won't help, and we'd have to recompile all the stuff by ourselves</li>
</ol>
After all /nix is a cool place.<br />
<br />
<h3>
Conclusion</h3>
<div>
<br /></div>
<div>
We've installed nix on our system, fully isolated and owned by the "nix" user as we're still diffident with this new system.</div>
<div>
We learned some new concepts like profiles and channels. In particular, with profiles we're able to manage multiple generations of a composition of packages, while with channels we're able to download binaries from nixos.org .</div>
<div>
The installation put everything under /nix, and some symlinks in the nix user home. That's because every user is able to install and use software in her own environment.</div>
<div>
<br /></div>
<div>
I hope I left nothing uncovered in a way that you think there's some kind of magic behind. It's all about putting components in the store and symlinking these components together.<br />
<br />
Also I hope the commands in this pill were consistent with your fresh nix installation.</div>
<br />
<h3>
Next pill... </h3>
<br />
...we will enter the nix environment and learn how to interact with the store.<br />
Pill 3 is available for <a href="http://lethalman.blogspot.it/2014/07/nix-pill-3-enter-environment.html">reading here</a>.<br />
<br />
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.</div>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com8tag:blogger.com,1999:blog-32054652.post-53078940044936380532014-07-23T07:11:00.001-07:002014-07-24T09:29:26.534-07:00Nix pill 1: why you should give it a try<b>Introduction</b><br />
<br />
Welcome to the first post of the <a href="http://nixos.org/nix/">Nix</a> in pills series. Nix is a purely functional package manager and deployment system for POSIX. There's a lot of documentation that describes what Nix, <a href="http://nixos.org/">NixOS</a> and related projects are.<br />
The purpose of this post is to convince you to give Nix a try. Installing NixOS is not required, but sometimes I may refer to NixOS as a real world example of Nix usage for building a whole operating system.<br />
<br />
<b>Rationale of pills</b><br />
<br />
The <a href="http://nixos.org/nix/manual/">Nix manual</a>, <a href="http://nixos.org/docs/papers.html">related papers</a>, <a href="http://www.infoq.com/articles/configuration-management-with-nix">articles</a> and <a href="https://nixos.org/wiki/Main_Page">wiki</a> are excellent in explaining how Nix/NixOS works, how you can use it and the amount of cools thing being done with it, however sometimes you may feel some magic happening behind the scenes hard to grasp in the beginning.<br />
This series aims to complement the missing explanations from the more formal documents.<br />
<br />
Follows a description of Nix. Just as pills, I'll try to be as short as possible.<br />
<br />
<b>Not being pure functional</b><br />
<br />
Most, if not all, widely used package managers (<a href="https://wiki.debian.org/dpkg">dpkg</a>, <a href="http://www.rpm.org/">rpm</a>, ...) mutate the global state of the system. If a package foo-1.0 installs a program to <span style="font-family: Courier New, Courier, monospace;">/usr/bin/foo</span>, you cannot install foo-1.1 as well, unless you change the installation paths or the binary name.<br />
Changing the installation paths means breaking the assumptions about the file system hierarchy.<br />
Changing the binary names means breaking possible users of that binary.<br />
<br />
Debian for example partially solves the problem with the <a href="https://wiki.debian.org/DebianAlternatives">alternatives</a>.<br />
<br />
So in theory it's possible with the current systems to install multiple versions of the same package, in practice it's very painful.<br />
Let's say you need an nginx service and also an nginx-openresty service. You have to create a new package that changes all the paths with e.g. an -openresty suffix.<br />
Or you want to run two different instances of mysql, 5.2 and 5.5. Same thing applies plus you have to also make sure the two mysqlclient libraries do not collide.<br />
<br />
This is not impossible but very inconvenient. If you want to install two whole stack of software like GNOME 3.10 and GNOME 3.12, you can imagine the amount of work.<br />
<br />
From an administrator view point: containers to the rescue. The solution nowadays is to create a container per service, especially when different versions are needed. That somehow solves the problem, but at a different level and with other drawbacks. Needing orchestration tools, setting up a shared cache of packages, and new wild machines to monitor rather than simple services.<br />
<br />
From a developer view point: virtualenv for python or jhbuild for gnome or whatelse. But then, how do you mix the two stacks? How do you avoid recompiling the same thing when it could be instead be shared? Also you need to setup your development tools to point to the different directories where libraries are installed. Not only, there's the risk that some of the software incorrectly uses system libraries.<br />
<br />
And so on. Nix solves all this at the packaging level and solves it well. A single tool to rule them all.<br />
<br />
<b>Being pure functional</b><br />
<b><br /></b>
Nix does no assumption about the global state of the system. This has many advantages, but also some drawbacks of course. The core of a Nix based system is the Nix store, usually installed under /nix/store, and some tools to manipulate the store. In Nix there's the notion of derivation rather than package. The difference might be subtle at the beginning, so I will often mix both of the words, ignore that.<br />
<br />
Derivations/packages are stored in the nix store as follows: <span style="font-family: Courier New, Courier, monospace;">/nix/store/<i>hash</i>-<i>name</i></span>, where the hash uniquely identifies the derivation (not true, it's a little more complex than this), and name is the name of the derivation.<br />
<br />
Let's take the bash derivation as example: <span style="font-family: Courier New, Courier, monospace;">/nix/store/s4zia7hhqkin1di0f187b79sa2srhv6k-bash-4.2-p45/</span> . It is a directory that contains <span style="font-family: Courier New, Courier, monospace;">bin/bash</span>.<br />
You get it, there's no <span style="font-family: Courier New, Courier, monospace;">/bin/bash</span>, there's only that self-contained build output in the store. Same goes for coreutils and everything else. To make their usage convenient from the shell, nix will put those paths in <span style="font-family: Courier New, Courier, monospace;">PATH</span>.<br />
What we have is basically a store of all packages and different versions of them, and things in the nix store are immutable.<br />
<br />
There's no ldconfig cache either. So where does bash find libc?<br />
<pre>$ ldd `which bash`
libc.so.6 => /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/libc.so.6 (0x00007f0248cce000)
</pre>
<br />
Turns out that when bash was built, it used that specific version of glibc and at runtime it will require exactly that glibc version.<br />
Don't be doubtful about the version in the derivation name: it's only a name for us humans. You may end up having a different hash given the same derivation name.<br />
<br />
What does it all mean? You could run mysql 5.2 with glibc-2.18, and mysql 5.5 with glibc-2.19. You could use your python module with python 2.7 compiled with gcc 4.6 and the same python module with python 3 compiled with gcc 4.8, all in the same system.<br />
In other words, no dependency hell, not even a dependency resolution algorithm. Straight dependencies from derivations to other derivations.<br />
<br />
From an administrator view point: need an old PHP version from lenny and want to upgrade to wheezy, not painful.<br />
<br />
From a developer view point: want to develop webkit with llvm 3.4 and 3.3, not painful.<br />
<br />
<b>Mutable vs immutable</b><br />
<b><br /></b>
When upgrading a library, package managers replace it in-place. All new applications run afterwards with the new library without being recompiled. After all, they all refer dynamically to libc6.so.<br />
While being immutable with Nix, upgrading a library like glibc means recompiling all applications, because the glibc path to the nix store is being hardcoded.<br />
<br />
So how do we deal with security updates? In Nix we have some tricks (yet pure) to solve this problem, but that's another story.<br />
<br />
Another problem is that unless software has in mind a pure functional model, or it can be adapted to, it's hard to compose applications at runtime.<br />
Let's take for example firefox. You install flash, and you get it working because firefox looks in a global path for plugins.<br />
In Nix, there's no such global path for plugins. Firefox therefore must know explicitly about the path to flash. We wrap the firefox binary to setup the necessary environment to make it find flash in the nix store. That will produce a new firefox derivation: be aware it takes a few seconds, but it made composition harder at runtime.<br />
<br />
No upgrade/downgrade scripts for your data. It doesn't make sense with this approach, because there's no real derivation to be upgraded. With nix you switch to using another software with its own stack of dependencies, but there's no formal notion of upgrade or downgrade when doing so.<br />
Migrating to the new data format is your own responsibility.<br />
<br />
<b>Conclusion</b><br />
<b><br /></b>
Nix lets you compose software at build time with maximum flexibility and with builds being as reproducible as possible. Not only, due to its nature deploying systems in the cloud is so easy, consistent and reliable that in the Nix world all existing self-containment and orchestration tools are deprecated by <a href="http://nixos.org/nixops/">NixOps</a>. Talking about this, no comparison with other tools is fair, nix rocks.<br />
It however <u>currently</u> falls off when speaking about dynamic composition at runtime or replacing low level libraries due to massive rebuilds.<br />
<br />
That may sound scary, however after running NixOS on both a server and a laptop desktop, I'm very satisfied so far. Some of the architectural problems just need some man power, other design problems are to be solved as a community yet.<br />
<br />
Considering <a href="https://nixos.org/nixpkgs/">Nixpkgs</a> (<a href="https://github.com/NixOS/nixpkgs">github link</a>) is a completely new repository of all the existing software, with a completely fresh concept, and with few core developers but overall year-over-year increasing contributions, the current state is way more than acceptable and beyond the experimental stage. In other words, it's worth your investment.<br />
<br />
<b>Next pill...</b><br />
<b><br /></b>
...we will install Nix on top of your current system (I assume GNU/Linux, but we also have OSX users) and start inspecting the installed stuff.<br />
Pill 2 is available for <a href="http://lethalman.blogspot.com/2014/07/nix-pill-2-install-on-your-running.html">reading here</a>.<br />
<br />
To be notified about the new pill, stay tuned on <a href="https://twitter.com/search?src=typd&q=%23NixPills">#NixPills</a>, follow <a href="https://twitter.com/lethalman">@lethalman</a> or subscribe to the <a href="http://lethalman.blogspot.com/feeds/posts/default/-/nixpills">nixpills rss</a>.Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com14tag:blogger.com,1999:blog-32054652.post-19420954927465830532014-07-09T14:05:00.000-07:002014-07-09T14:05:33.308-07:00Debugging and fixing a wmctrl bug in NixOSWithout wmctrl I'm lost. I use it together with xbindkeys so that pressing a combination of keys will raise a window of the desktop (editor, shell, browser, chat, ...).<br />
<br />
It turns out however that in NixOS wmctrl didn't work well. I had 3 windows opened, and wmctrl -l only showed one. Since gnome3 is still young in nixos, the first thought was that mutter did something wrong when returning the list of clients, due to some possible packaging mistake.<br />
<br />
However, xprop -root correctly listed 3 window ids for the _NET_CLIENT_LIST atom, so mutter did its job.<br />
<br />
Therefore it might be a problem of wmctrl. I'm always sure in these cases Debian has a patch, however I wanted to find out by myself.<br />
<br />
Let's prepare an environment for hacking wmctrl:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">$ nix-shell ~/nixpkgs -A wmctrl</span><br />
<span style="font-family: Courier New, Courier, monospace;">$ unpackPhase</span><br />
<span style="font-family: Courier New, Courier, monospace;">$ cd $sourceRoot</span><br />
<span style="font-family: Courier New, Courier, monospace;">$ configurePhase</span><br />
<span style="font-family: Courier New, Courier, monospace;">$ buildPhase</span><br />
<span style="font-family: Courier New, Courier, monospace;">$ ./wmctrl -l</span><br />
<br />
Ok it runs. Now let's dive into the code.<br />
<ol>
<li>Open main.c</li>
<li>Search for <span style="font-family: Courier New, Courier, monospace;">_NET_CLIENT_LIST</span></li>
<li>Note the for loop in list_windows function right below, it uses <span style="font-family: Courier New, Courier, monospace;">client_list_size / sizeof(Window)</span><span style="font-family: inherit;"> in the condition. </span>That may be the culprit. </li>
<li>We make use of the most advanced debugging technique: before the for loop add a <span style="font-family: Courier New, Courier, monospace;">printf("%lu\n%lu\n", client_list_size, sizeof(Window));</span></li>
</ol>
Now let's rebuild and run wmctrl:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">$ buildPhase</span><br />
<span style="font-family: Courier New, Courier, monospace;">$ ./wmctrl -l</span><br />
<br />
Bingo! <span style="font-family: Courier New, Courier, monospace;">client_list_size</span> is 12 (3 windows x 4 byte) while <span style="font-family: Courier New, Courier, monospace;">sizeof(Window)</span> is 8 instead of 4. Definitely a bug for 64-bit systems.<br />
Of course Debian has the patch: <a href="http://ftp.de.debian.org/debian/pool/main/w/wmctrl/wmctrl_1.07-7.debian.tar.gz">http://ftp.de.debian.org/debian/pool/main/w/wmctrl/wmctrl_1.07-7.debian.tar.gz</a><br />
<br />
<ol>
<li>Exit the nix-shell and delete the wmctrl source directory.</li>
<li>Copy <span style="font-family: Courier New, Courier, monospace;">debian/patches/01_64-bit-data.patch</span> under <span style="font-family: Courier New, Courier, monospace;">~/nixpkgs/pkgs/tools/X11/wmctrl/64-bit-data.patch</span>.</li>
<li>Edit <span style="font-family: Courier New, Courier, monospace;">~/nixpkgs/pkgs/tools/X11/wmctrl/default</span> and add <span style="font-family: Courier New, Courier, monospace;">patches = [ ./64-bit-data.patch ];</span><span style="font-family: inherit;"> in the derivation.</span></li>
<li>Finally install the patched wmctrl with <span style="font-family: Courier New, Courier, monospace;">nix-env ~/nixpkgs -iA wmctrl</span> and see that wmctrl -l now works.</li>
<li>Commit, push to nixpkgs master, profit.</li>
</ol>
Luca Brunohttp://www.blogger.com/profile/06329046616630750783noreply@blogger.com8