Friday, April 15, 2016

Cheap Docker images with Nix

Let's talk about Docker and Nix today. Before explaining what Nix is, if you don't know yet, and before going into the details, I will show you a snippet similar to a Dockerfile for creating a Redis image equivalent to the one in docker hub.

The final image will be around 42mb (or 25mb) in size, compared to 177mb.

EDIT: as mentioned on HN, alpine-based images can even go around 15mb in size.

If you want to try this, the first step is to install Nix.

Here's the redis.nix snippet:


Build it with: nix-build redis.nix
Load it with: docker load < result

Once loaded, you can see with docker images that it takes about 42mb of space.

Fundamental differences with classic docker builds

  • We do not use any base image, like it's done for most docker images including redis from the hub. It starts from scratch. In fact, we set up some basic shadow-related files with the shadowSetup utility, enough to add the redis user and make gosu work.
  • The Redis package is not being compiled inside Docker. It's being done by Nix, just like any other package.
  • The built image has only one layer, compared to dozens usually spitted by a readable Dockerfile. In our case, having multiple layers is useless because caching is handled by Nix, and not by Docker.

A smaller image

We can cut the size down to 25mb by avoid using id from coreutils. As an example we'll always launch redis without the entrypoint:


You might ask: but coreutils is still needed for the chown, mkdir and other commands like that!

The secret is that those commands are only used at build time and are not required at runtime in the container. Nix is able to detect that automatically for us.

It means we don't need to manually remove packages after the container is built, like with other package managers! See this line in Redis Dockerfile for example.

Using a different redis version

Let's say we want to build a Docker image with Redis 2.8.23. First we want to write a package (or derivation in Nix land) for it, and then use that inside the image:


Note we also added the tag 2.8.23 to the resulting image. And that's it. The beauty is that we reuse the same redis expression from nixpkgs, but we override only the version to build.

A generic build

There's more you can do with Nix. Being a language, it's possible to create a generic function for building Redis images given a specific package:


We created a "redisImage" function that takes a "redis" parameter as input, and returns a Docker image as output.

Build it with:
  • nix-build redis-generic.nix -A redisDocker_3_0_7 
  • nix-build redis-generic.nix -A redisDocker_2_8_23

Building off a base image

One of the selling points of Docker is reusing an existing image to add more stuff on top of it.

Nix comes with a completely different set of packages compared to other distros, with its own toolchain and glibc version. This doesn't mean it's not possible to base a new image off an existing Debian image for instance.

By using dockerTools.pullImage it's also possible to pull images from the Docker hub.


Build it with: nix-build redis-generic.nix -A redisOnDebian.

Note that we added a couple of things. We pass the base image (debianImage), to our generic redisImage function, and that we only initialize shadow-utils if the base image is null.

The result is a Docker image based off latest Debian but running Redis compiled with nixpkgs toolchain and using nixpkgs glibc. It's about 150mb. It has all the layers from the base image, plus the new single layer for Redis.

That said, it's as well possible to use one of the previously defined Redis images as base image. The result of `pullImage` and `buildImage` is a .tar.gz docker image in both cases.

You realize it's possible to build something quite similar to docker-library using only Nix expressions. It might be an interesting project.

Be aware that things like PAM configurations, or other stuff, created to be suitable for Debian may not work with Nix programs that use a different glibc.

Other random details

The code above has been made possible by using nixpkgs commit 3ae4d2afe (2016-04-14) onwards, commit at which I've finally packaged gosu and since the size of the derivations have been notably reduced.

Building the image is done without using any of the Docker commands. The way it works is as follows:
  1. Create a layer directory with all the produced contents inside. This includes the filesystem as well as the json metadata. This process will use certain build dependencies (like coreutils, shadow-utils, bash, redis, gosu, ...).
  2. Ask Nix what are the runtime dependencies of the layer directory (like redis, gosu). Such dependencies will be always a subset of the build dependencies.
  3. Add such runtime dependencies to the layer directory.
  4. Pack the layer in a .tar.gz by following the Docker specification.
I'd like to state that Nix has a safer and easier caching of operations while building the image.
As for Docker, great care has to be taken in order to use the layer cache correctly, because such caching is solely based on the RUN command string. This blog post explains it well.
This is not the case for Nix, because every output depends on a set of exact inputs. If any of the inputs change, the output will be rebuilt.

So what is Nix?

Nix is a language and deployment tool, often used as package manager or configuration builder and system provisioning. The operating system NixOS is based on it.

The code shown above is Nix. We have used the nixpkgs repository which provides several reusable Nix expressions like redis and dockerTools.

The Nix concept is simple: write a Nix expression, build it. This is how the building process works at a high-level:
  1. Read a Nix expression
  2. Evaluate it and determine the thing (called derivation) to be built.
  3. By evaluating the code, Nix is able to determine exactly the build inputs needed for such derivation.
  4. Build (or fetch from cache) all the needed inputs.
  5. Build (or fetch from the cache) the final derivation.
Nix stores all such derivations in a common nix store (usually /nix/store), identified by an hash. Each derivation may have dependencies to other paths in the same store. Each derivation is stored in a separate directory from other derivations.

Won't go deeper as there's plenty of documentation about how Nix works and how its storage works.

Hope you enjoyed the reading, and that you may give Nix a shot.

57 comments:

Anonymous said...

It was really annoying to put the definition/overview of Nix at the bottom, I guess you went for some kind writing trick, but it really makes it horrible for someone to learn what you're trying to showcase.

Luca Bruno said...

Thanks for the critique. My idea was to just show some code first, to compare it at a high-level with a Dockerfile at first sight.

Personally I often skip introductions in technical blog posts, and go straight in the middle of the article to see some code first. So I thought about moving this logic directly in a post.

Colonel Panic said...

I agree with Anonymous.

Anonymous said...

MB, not mb.....

Anonymous said...

because we all thought he was talking about millibits

Anonymous said...

I'm struggling to reproduce the steps.

I installed Nix by using the install instructions that are linked to at the top of the blog. Then I took the first code snippet and put it into redis.nix after which I ran nix-build redis.nix

At this point I get the following error:

error: attribute ‘gosu’ missing, at /home//redis.nix:11:14
(use ‘--show-trace’ to show detailed location information)

Any idea why it doesn't work?

Anonymous said...

To last poster: Lethalman mentioned that his `redis.nix` only works started on a certain, very recent, commit. You must be on an older commit of nixpkgs.

Anonymous said...

Great post. I'm really excited about Nix. I do have a question though - isn't using Nix with Docker kind of an odd combination? For instance was does building a a Docker image using Nix give you over creating a regular Dockerfile to build a Docker image? Maybe this was just for demonstration purposes?

Charles Strahan said...

Q: "What does building a a Docker image using Nix give you over creating a regular Dockerfile to build a Docker image?"

A:

* Better abstraction (e.g. the example of a function that produces docker images)
* The Hydra build/CI server obviates the need for paying for (or administering a self hosted) docker registry, and avoids the imperative push and pull model. Because a docker image is just another Nix package, you get distributed building, caching and signing for free.
* Because Nix caches intermediate packages builds, building a Docker image via Nix will likely be faster than letting Docker do it.
* Determinism. With Docker, you're not guaranteed that you'll build the same image across two machines (imagine the state of package repositories changing -- it's trivial to find different versions of packages across two builds of the same Dockerfile). With Nix, you're guaranteed that you have the same determinism that any other Nix package has (e.g. everything builds in a chroot without network access (unless you provide a hash of the result, for e.g. tarball downloads))

Unknown said...

@CharlesStrahan: You had me at 'determinism'.


Also, the idea of leaving Docker registry behind in favor of Hydra sounds appealing. Will start tinkering with all this bits.

@lethalman: Thanks for the great post.

Pádraig Brady said...

RE coreutils size, if you do want coreutils in the final image, building in "shared binary" mode can get all 100 utils for about 1MB. The only change should be to `./configure --enable-single-binary` in the build. See the coreutils-single package in Fedora for a concrete example

Yann Hodique said...

That's great. It got me thinking... what about going even further, and removing the glibc from the picture?
I mean, it's like 27MB by itself before compression

I extended your recipe with the following:
- su-exec instead of gosu
- removing various useless binaries from the redis image (only redis-server is really needed)
- statically built everything with musl

the result is there: https://gist.github.com/sigma/9887c299da60955734f0fff6e2faeee0

and the corresponding docker image is 1.2 MB:

$ docker images redis
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
redis 3.0.7-mini 5986d6c71430 46 years ago 1.187 MB

Luca Bruno said...

Awesome Yann! Yes, it's perfectly possible to go down to that size. I wanted in first place to emulate as much as possible the original redis image from the hub, in terms of operations.

I didn't think of musl, it could be very interesting to create a framework in nixpkgs around it for creating such docker images.

Gabriel Gonzalez said...

Is there a workaround to build this on OS X? At least one dependency of the nix expression is not supported on Darwin.

PierreR said...

I have had a got with:
http://lpaste.net/4138752819979091968

But I ended up with a docker image > 2G (full ghc copied). What's wrong ? I am using the stable 16.03 channel (maybe this needs something more recent) ?

Luca Bruno said...

Stable 16.03 does not have the multi outputs stuff that is in the unstable channel. Also don't know about much about the state of ghc in nix, and its multi outputs support.
Surely using the unstable channel will be much better.

Unknown said...

NixOS + Docker looks like splendid combination. Thank you.

Unknown said...

To have the first example working I had to replace 'chown' by '${coreutils}/bin/chown' at two places, otherwise it does not find the command (thus I'm not sure it's possible to exclude coreutils from the image).

Luca Bruno said...

If the entrypoint.sh is using chown, well it's not possible to exclude coreutils. But if you only run straight redis it is possible.

Neználek said...

We now have a PR for coreutils --enable-single-binary: https://github.com/NixOS/nixpkgs/pull/16406

Unknown said...

Spot on! exactly what am looking for.

wiki

Alexey Shmalko said...

You can further decrease image size if you use docker's built-in ability to change users:

config.User = "redis";

Unknown said...

This is great! Any change you would consider adding support for creating a rkt container?

Anonymous said...

> "Nix is able to detect that automatically for us."

How? How can Nix know which files are being read? Without more fine-grained explicit permissions, how is it possible to know what a given program might need upon runtime?

Unknown said...

Great Article About Docker Here you can find some frequently asked
Docker Interview Questions and Answers with details

Unknown said...

Your Collection of Interview Questions are very useful. Here You can find some Frequently Asked DevOps Interview Questions and Answers with explanation

Chef Interview Questions and Answers

Docker Interview Questions and Answers

GIT Interview Questions and Answers

Jenkins Interview Questions and Answers

Maven Interview Questions and Answers

Nagios Interview Questions and Answers

Puppet Interview Questions and Answers

Unknown said...

The most effective and influential blog related to customer support section.
visit here;


AOL Customer Care
AOL Customer Care Number
AOL Customer Care Phone Number

Unknown said...

I would like to orchestrate containers, provisioned by nix, how can I do this?

Chris League said...

Heh... I like how the "docker images" output says "created 48 years ago". :)

Guest Posting World said...

Nice and informative Blog post.

Unknown said...

For anyone trying to reproduce the 1.2MB file above:

${musl} should be replaced with ${musl.dev}

shadowSetup seems to set bash's shell to ${stdenv.shell} in the two files /etc/passwd and /etc/passwd- (not sure why that exists). I used sed to remove them, which removed bash and glibc from the closure.

institute of marketing said...

Nice blog..! I really loved reading through this article. Thanks for sharing such a amazing post with us and keep blogging..
Digital marketing courses in Bangalore with placements

Priyanka said...

Attend The Digital Marketing Courses in Bangalore From ExcelR. Practical Digital Marketing Courses in Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Digital Marketing Courses in Bangalore.
ExcelR Digital Marketing Courses in Bangalore

Rohini said...

Nice information, valuable and excellent design, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here...digital marketing course bangalore

CloudLearn ERP said...

I'm not one of those readers that comments on articles often, but yours really compelled me. There's a lot of interesting content in this article that is interesting and bold.
Best Data Science training in Mumbai

Data Science training in Mumbai

Mr Rahman said...

Really Great Post & Thanks for sharing.

Oflox Is The Best Website Design Company In Dehradun

Pravin Patel said...

Find all latest government jobs at Maharojgar

Get All Government Jobs / Sarkari Naukri recruitment Notifications Here for Freshers and Experienced. Qualification 10th, 12th, ANY DEGREE, PG, BTech, MBA. Government Jobs in India with state and local governments including city, county, and state public agencies. Apply now.

imexpert said...

This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.
data analytics course in mumbai

priyash said...

wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries. keep it up.
data analytics course in Bangalore

ravali said...

Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.

data science course

Rohini said...

Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.....Data Analytics courses

priyash said...

wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
Data Science Course

Anirban Ghosh said...

Thank you for your attention to detail and great writing style. Your professionalism shows in your article. I like your interesting views and appreciate your unique ideas. This is quality.
SAP training in Kolkata
SAP training Kolkata
Best SAP training in Kolkata
SAP course in Kolkata
SAP training institute Kolkata

Rajeev Yasiru Mathew said...

Rajeev Yasiru Mathew - Social Media Strategist - Official Website
Rajeev Yasiru Mathew Biography - Official Website
Portfolio - Rajeev Yasiru Mathew - Official Website
Newspaper Articles - The Official Website of Rajeev Yasiru
Digital Media Consulting Service - Rajeev Yasiru Mathew
COVID-19 The Digital Virus Threat - Rajeev Yasiru Mathew
Right To Privacy in Sri Lanka - Rajeev Yasiru Mathew
Lotus Tower: TRC goes silent on President’s claims
Presidential Poll: A digital wake up call - Rajeev Yasiru Mathew
Contact - The Official Website of Rajeev Yasiru Mathew

Rohini said...

I was just browsing through the internet looking for some information and came across your blog. I am impressed by the information that you have on this blog. It shows how well you understand this subject. Bookmarked this page, will come back for more....Data Analyst Course

Rohini said...

I was just browsing through the internet looking for some information and came across your blog. I am impressed by the information that you have on this blog. It shows how well you understand this subject. Bookmarked this page, will come back for more....Data Analyst Course

priyash said...

Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
Correlation vs Covariance
Simple linear regression

shane lee said...

he human imprinted memory can be changed by learning a new way of doing something, which is ironic to the brain, as it is different from what was committed to memory, thus allowing a memory overwrite. In artificial intelligence the imprinted data sets artificial intelligence courses in hyderabad

Priyanka said...

Attend The Data Analyst Course From ExcelR. Practical Data Analyst Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analyst Course.
Data Analyst Course

priyanka said...

Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
Correlation vs Covariance
Simple linear regression
data science interview questions

CloudLearn ERP said...

I am speechless as to how great this article is. The content is simple to understand and very engaging. This is by far some of the best content I've seen.
Data Science training in Mumbai
Data Science course in Mumbai
SAP training in Mumbai

Anirban Ghosh said...

It has been a long time since I've read anything so informative and compelling. I'm waiting for the next article from the writer. Thank you.
SAP training in Kolkata
SAP training Kolkata
Best SAP training in Kolkata
SAP course in Kolkata

Home Garden Blogs said...

Damien Grant
Damien Grant
Damien Grant
Damien Grant
Damien Grant
Damien Grant
Damien Grant
Damien Grant

dexterous said...

Thanks For Sharing Such Great Information, It's Really Nice And Informative...

GMAT online preparation
GMAT training in hyderabad
CAT Coaching in hyderabad

data scientist said...

There is no dearth of Data Science course syllabus or resources. Learn the advanced data science course concepts and get your skills upgraded from the pioneers in Data Science.
data science course bangalore
data science course syllabus

Roku Help Number said...

Roku device is the best service in the USA, Do you have an issue with your Roku Device? Get instant solution and support by Roku Help Number. We are dealing with a wide range of Roku streaming players and provide a complete solution for Roku Devices. Call Roku Customer Service Number.
https://roku-helpnumber.com/
https://roku-helpnumber.com/change-roku-stick-factory-settings-without-remote/
https://roku-helpnumber.com/how-to-fix-common-roku-error-codes/
https://roku-helpnumber.com/services/
https://roku-helpnumber.com/roku-customer-service-phone-number/
Roku Help Phone Number
Change Roku Stick Factory Setting Without Remote
How to Fix Common Roku Error Codes
Roku Customer Service Phone Number
Contact Roku Customer Service Number

priyanka said...

Amazing Article ! I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
Simple Linear Regression
Correlation vs covariance
data science interview questions
KNN Algorithm
Logistic Regression explained