Monday, January 12, 2015

Nix pill 18: nix store paths

Welcome to the 18th Nix pill. In the previous 17th pill we have scratched the surface of the nixpkgs repository structure. It is a set of packages, and it's possible to override such packages so that all other packages will use the overrides.

Before reading existing derivations, I'd like to talk about store paths and how they are computed. In particular we are interested in fixed store paths that depend on an integrity hash (e.g. a sha256), which is usually applied to source tarballs.

The way store paths are computed is a little contrived, mostly due to historical reasons. Our reference will be the Nix source code.

Source paths


Let's start simple. You know nix allows relative paths to be used, such that the file or directory is stored in the nix store, that is ./myfile gets stored into /nix/store/....... We want to understand how is the store path generated for such a file:
$ echo mycontent > myfile
I remind you, the simplest derivation you can write has a name, a builder and the system:
$ nix-repl
nix-repl> derivation { system = "x86_64-linux"; builder = ./myfile; name = "foo"; }
«derivation /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv»
Now inspect the .drv to see where is ./myfile being stored:
$ pp-aterm -i /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv
Derive(
  [("out", "/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo", "", "")]
, []
, ["/nix/store/xv2iccirbrvklck36f1g7vldn5v58vck-myfile"]
, "x86_64-linux"
...
Great, how did nix decide to use xv2iccirbrvklck36f1g7vldn5v58vck ? Keep looking at the nix comments.

Note: doing nix-store --add myfile will store the file in the same store path.

Step 1, compute the hash of the file


The comments tell us to first compute the sha256 of the NAR serialization of the file. Can be done in two ways:
$ nix-hash --type sha256 myfile
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3
Or:
$ nix-store --dump myfile|sha256sum
2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3  -
In general, Nix understands two contents: flat for regular files, or recursive for NAR serializations which can be anything.

Step 2, build the string description


Then nix uses a special string which includes the hash, the path type and the file name. We store this in another file:
$ echo -n "source:sha256:2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3:/nix/store:myfile" > myfile.str

Step 3, compute the final hash


Finally the comments tell us to compute the base-32 representation of the first 160 bits (truncation) of a sha256 of the above string:
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
xv2iccirbrvklck36f1g7vldn5v58vck

Output paths


Output paths are usually generated for derivations. We use the above example because it's simple. Even if we didn't build the derivation, nix knows the out path hs0yi5n5nw6micqhy8l1igkbhqdkzqa1. This is because the out path only depends on inputs.

It's computed in a similar way to source paths, except that the .drv is hashed and the type of derivation is output:out. In case of multiple outputs, we may have different output:<id>.

At the time nix computes the out path, the .drv contains an empty string for each out path. So what we do is getting our .drv and replacing the out path with an empty string:
$ cp -f /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv myout.drv
$ sed -i 's,/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo,,g' myout.drv
The myout.drv is the .drv state in which nix is when computing the out path for our derivation:
$ sha256sum myout.drv
1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5  myout.drv
$ echo -n "output:out:sha256:1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5:/nix/store:foo" > myout.str
$ nix-hash --type sha256 --truncate --base32 --flat myout.str
hs0yi5n5nw6micqhy8l1igkbhqdkzqa1
Then nix puts that out path in the .drv, and that's it.

In case the .drv has input derivations, that is it references other .drv, then such .drv paths are replaced by this same algorithm which returns an hash.

In other words, you get a final .drv where every other .drv path is replaced by its hash.

Fixed-output paths


Finally, the other most used kind of path is when we know beforehand an integrity hash of a file. This is usual for tarballs.

A derivation can take three special attributes: outputHashMode, outputHash and outputHashAlgo which are well documented in the nix manual.

The builder must create the out path and make sure its hash is the same as the one declared with outputHash.

Let's say our builder should create a file whose contents is mycontent:
$ echo mycontent > myfile
$ sha256sum myfile
f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb  myfile
nix-repl> derivation { name = "bar"; system = "x86_64-linux"; builder = "none"; outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb"; }
«derivation /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv»
Inspect the .drv and see that it also stored the fact that it's a fixed-output derivation with sha256 algorithm, compared to the previous examples:
$ pp-aterm -i /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv
Derive(
  [("out", "/nix/store/a00d5f71k0vp5a6klkls0mvr1f7sx6ch-bar", "sha256", "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb")]
...
It doesn't matter which input derivations are being used, the final out path must only depend on the declared hash.
What nix does is to create an intermediate string representation of the fixed-output content:
$ echo -n "fixed:out:sha256:f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb:" > mycontent.str
$ sha256sum mycontent.str 
423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639  myfile.str
Then proceed as it was a normal derivation output path:
$ echo -n "output:out:sha256:423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639:/nix/store:bar" > myfile.str
$ nix-hash --type sha256 --truncate --base32 --flat myfile.str
a00d5f71k0vp5a6klkls0mvr1f7sx6ch
Hence, the store path only depends on the declared fixed-output hash.

Conclusion


There are other types of store paths, but you get the idea. Nix first hashes the contents, then creates a string description, and the final store path is the hash of this string.

Also we've introduced some fundamentals, in particular the fact that Nix knows beforehand the out path of a derivation since it only depends on the inputs. We've also introduced fixed-output derivations which are especially used by the nixpkgs repository for downloading and verifying source tarballs.

Next pill


...we will introduce stdenv. In the previous pills we rolled our own mkDerivation convenience function for wrapping the builtin derivation, but the nixpkgs repository also has its own convenience functions for dealing with autotools projects and other build systems.

Pill 19 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

20 comments:

Anonymous said...

Do you by chance have a "Nix pill" that discusses running a private nix store (via http or ssh).
In my application I cannot pull cached files from the web, I need the store to be built/cached locally.

I see I can override the store cache path in the nix.conf.
Anyway if you have written a pill about this I would be interested in reading it.
regards,
Brad

htop said...

thanks for sharing this information
aws training center in chennai
aws training in chennai
aws training in omr
aws training in sholinganallur
aws training institute in chennai
best aws training in sholinganallur

technical raja said...


51 sad status




52 sad sttaus in hindi





53 love status video whatsapp




54 tmil whatsapp status video

technical raja said...


status for wahstapp

Anirban Ghosh said...

This article reflects many of my own thoughts on this subject. You are truly unique.
SAP training in Mumbai
Best SAP training in Mumbai
SAP training institute Mumbai

BestTrainingMumbai said...

This article reflects my very own large number musings regarding this matter. You are genuinely one of a kind.
SAP training in Kolkata
Best SAP training in Kolkata
SAP training institute in Kolkata

CloudLearn ERP said...

Your composing style says a great deal regarding what your identity is and as I would like to think I'd need to state you're savvy.
SAP training in Mumbai
Data Science training in Mumbai
Best data science training in Mumbai

Roy said...

movie download

Shiva said...

You post is very informative. I like your post Please update more

Digital Brolly said...

Learn Digital Marketing course from Digital Brolly

Tech News said...

I'm impressed, I must say. Very rarely do I come across a blog that's both informative and entertaining, and let me tell you,
you've hit the nail on the head. Your blog is important; the issue is something that not enough people are talking intelligently

Attitude Status

Love shayari
Status Shop
Sad Status
Attitude Girl Status in hindi
Ansh Pandit Shayari
New Love Shayari
Trust Shayari
Good Morning in Hindi
New Attitude Status in Hindi

Sunil said...

Nice Article,
Digital Marketing Training in KPHB
at Digital Brolly

technical raja said...

Love Shayari

Romantic Shayari

Sad Shayari

Shayari Sad In Hindi

success status

shayari love

Anonymous said...

I am highly satisfied to read this flawless piece of writing. It has really inspired me to read more on this topic. I must say it is a real grand work done by you. Web Hosting plays a very important role in the business world. And it is important to have the best hosting services. Buy the best vps hosting germany service for your website.

Ravi Varma said...

I’m Ravi Varma, a Serial Entrepreneur who enjoys exploring new ideas and business opportunities. I am a fan of technology, design, and entrepreneurship.

salar said...

We make you launch a real-time live campaign on Facebook so you get work experience to get a job easily on Social Media Marketing.

Unknown said...

Digital Marketing Institute in KPHB, we address all major and minor aspects required for any student’s advancement in digital marketing. Clutch USA named our Digital Marketing Institute the best SEO firm.

The future of digital marketing is promising and full of possibilities.
As a result, skilled digital marketers who can keep up with the rising demand are in high order.

In the Emblix Academy Digital marketing institute in KPHB, you will learn about all the major and minor modules of digital marketing, from Search engine marketing to Social Media Marketing and almost all Tools used for Digital Marketing.


Website: Digital Marketing Institute in KPHB

Sudharshan Bhat guruji said...

Nice blog with good content,thanks for sharing.
For Astrological service contact Shri Durga astro center,They gives
Best Astrologer in Dharwad

Hindi Film said...

I wish I could be with you wherever you are and wherever you go every day. Just so I wouldn’t have to miss you so.

fake love quotes

Mom Dad Quotes

quotes about black love

brother from another mother quotes

One Sided Love Quotes

Positive Attitude Quotes

Best Sad Love Quotes

Romantic Short Love Quotes

Digital Marketing Course in Rajouri Garden said...

Digital marketing course in Delhi are an ideal way to further your career. You can get advice from experienced marketing professionals and establish your professional network as soon as you can. In fact, Digital Marketing Institute Digital Marketing course recently introduced a unique certification course for business leaders. It is designed to help improve your knowledge of marketing and boost the confidence to achieve your goals. Whether you are a senior marketer or rising star, this program is ideal for you. The program includes 12 classes and online modules, the program also features interactive Q&As along with support from a community of experts. It will provide all the information you need to make it. Presented by Raja Rajamannar, a senior market leader this program is open to senior marketing professionals, in addition to to professionals across all functions.