Deploying NPM packages with the Nix package manager
Post on 14-Feb-2017
86 Views
Preview:
Transcript
Deploying NPM packages with the Nix packagemanager
Sander van der Burg
Feb 4, 2017
Sander van der Burg Deploying NPM packages with the Nix package manager
The Nix project
Family of declarative software deployment tools:
Nix. A purely functional package manager
NixOS. Nix based GNU/Linux distribution
Hydra. Nix based continuous build and integration server
Disnix. Nix based distributed service deployment
NixOps. NixOS-based multi-cloud deployment tool
Sander van der Burg Deploying NPM packages with the Nix package manager
The Nix project
Non-functional properties:
Generic. Can be used with many programming languages,component technologies, and operating systems.Reproducible. (Almost) no impurities – if inputs are the same,result should be the same.Reliable. Dependency completeness, (almost) atomicupgrades and rollbacks.Efficient. Only the required deployment activities areexecuted.
Sander van der Burg Deploying NPM packages with the Nix package manager
NixOS configuration
/etc/nixos/configuration.nix
{pkgs, ...}:
{
boot.loader.grub.device = "/dev/sda";
fileSystems = [ { mountPoint = "/"; device = "/dev/sda2"; } ];
swapDevices = [ { device = "/dev/sda1"; } ];
services = {
openssh.enable = true;
xserver = {
enable = true;
desktopManager.kde4.enable = true;
};
};
environment.systemPackages = [ pkgs.mc pkgs.firefox ];
}
Sander van der Burg Deploying NPM packages with the Nix package manager
NixOS configuration
nixos-rebuild switch
Nix package manager builds a complete system configuration
Includes all packages and generates all configuration files, e.g.OpenSSH configuration
Upgrades are (almost) atomic
Components are stored safely next to each other, due to hashesNo files are automatically removed or overwritten
Users can switch to older generations of system configurationsnot garbage collected yet
Sander van der Burg Deploying NPM packages with the Nix package manager
NixOS bootloader
Sander van der Burg Deploying NPM packages with the Nix package manager
The Nix project
Basis of all tools: The Nix package manager
Sander van der Burg Deploying NPM packages with the Nix package manager
Nix store
Main idea: store all packagesin isolation from each other:
/nix/store/rpdqxnilb0cg...
-firefox-3.5.4
Paths contain a 160-bitcryptographic hash of allinputs used to build thepackage:
Sources
Libraries
Compilers
Build scripts
. . .
/nix/store
l9w6773m1msy...-openssh-4.6p1
bin
ssh
sbin
sshd
smkabrbibqv7...-openssl-0.9.8e
lib
libssl.so.0.9.8
c6jbqm2mc0a7...-zlib-1.2.3
lib
libz.so.1.2.3
im276akmsrhv...-glibc-2.5
lib
libc.so.6
Sander van der Burg Deploying NPM packages with the Nix package manager
Nix expressions
openssh.nix
{ stdenv, fetchurl, openssl, zlib }:
stdenv.mkDerivation {
name = "openssh-4.6p1";
src = fetchurl {
url = http://.../openssh-4.6p1.tar.gz;
sha256 = "0fpjlr3bfind0y94bk442x2p...";
};
buildCommand = ’’
tar xjf $src
./configure --prefix=$out --with-openssl=${openssl}
make; make install
’’;
}
Sander van der Burg Deploying NPM packages with the Nix package manager
Nix expressions
all-packages.nix
openssh = import ../tools/networking/openssh {
inherit fetchurl stdenv openssl zlib;
};
openssl = import ../development/libraries/openssl {
inherit fetchurl stdenv perl;
};
stdenv = ...;
openssl = ...;
zlib = ...;
perl = ...;
nix-env -f all-packages.nix -iA openssh
Produces a /nix/store/l9w6773m1msy...-openssh-4.6p1
package in the Nix store.
Sander van der Burg Deploying NPM packages with the Nix package manager
User environments
I Users can havedifferent sets ofinstalled applications.
I nix-env operationscreate new userenvironments in thestore.
I We can atomicallyswitch between them.
I These are roots of thegarbage collector.
PATH
/nix/.../profiles
current
42
/nix/store
pp56i0a01si5...-user-envbin
firefoxssh
l9w6773m1msy...-openssh-4.6p1bin
sshrpdqxnilb0cg...-firefox-3.5.4
binfirefox
Sander van der Burg Deploying NPM packages with the Nix package manager
User environments
I Users can havedifferent sets ofinstalled applications.
I nix-env operationscreate new userenvironments in thestore.
I We can atomicallyswitch between them.
I These are roots of thegarbage collector.
PATH
/nix/.../profiles
current
42
/nix/store
pp56i0a01si5...-user-envbin
firefoxssh
l9w6773m1msy...-openssh-4.6p1bin
sshrpdqxnilb0cg...-firefox-3.5.4
binfirefox
aqn3wygq9jzk...-openssh-5.2p1bin
ssh
(nix-env -u openssh)
Sander van der Burg Deploying NPM packages with the Nix package manager
User environments
I Users can havedifferent sets ofinstalled applications.
I nix-env operationscreate new userenvironments in thestore.
I We can atomicallyswitch between them.
I These are roots of thegarbage collector.
PATH
/nix/.../profiles
current
42
/nix/store
pp56i0a01si5...-user-envbin
firefoxssh
l9w6773m1msy...-openssh-4.6p1bin
sshrpdqxnilb0cg...-firefox-3.5.4
binfirefox
aqn3wygq9jzk...-openssh-5.2p1bin
sshi3d9vh6d8ip1...-user-env
binsshfirefox
(nix-env -u openssh)
Sander van der Burg Deploying NPM packages with the Nix package manager
User environments
I Users can havedifferent sets ofinstalled applications.
I nix-env operationscreate new userenvironments in thestore.
I We can atomicallyswitch between them.
I These are roots of thegarbage collector.
PATH
/nix/.../profiles
current
42
43
/nix/store
pp56i0a01si5...-user-envbin
firefoxssh
l9w6773m1msy...-openssh-4.6p1bin
sshrpdqxnilb0cg...-firefox-3.5.4
binfirefox
aqn3wygq9jzk...-openssh-5.2p1bin
sshi3d9vh6d8ip1...-user-env
binsshfirefox
(nix-env -u openssh)
Sander van der Burg Deploying NPM packages with the Nix package manager
User environments
I Users can havedifferent sets ofinstalled applications.
I nix-env operationscreate new userenvironments in thestore.
I We can atomicallyswitch between them.
I These are roots of thegarbage collector.
PATH
/nix/.../profiles
current
42
43
/nix/store
pp56i0a01si5...-user-envbin
firefoxssh
l9w6773m1msy...-openssh-4.6p1bin
sshrpdqxnilb0cg...-firefox-3.5.4
binfirefox
aqn3wygq9jzk...-openssh-5.2p1bin
sshi3d9vh6d8ip1...-user-env
binsshfirefox
(nix-env -u openssh)
Sander van der Burg Deploying NPM packages with the Nix package manager
User environments
I Users can havedifferent sets ofinstalled applications.
I nix-env operationscreate new userenvironments in thestore.
I We can atomicallyswitch between them.
I These are roots of thegarbage collector.
PATH
/nix/.../profiles
current
43
/nix/store
pp56i0a01si5...-user-envbin
firefoxssh
l9w6773m1msy...-openssh-4.6p1bin
sshrpdqxnilb0cg...-firefox-3.5.4
binfirefox
aqn3wygq9jzk...-openssh-5.2p1bin
sshi3d9vh6d8ip1...-user-env
binsshfirefox
(nix-env --remove-generations old)
Sander van der Burg Deploying NPM packages with the Nix package manager
User environments
I Users can havedifferent sets ofinstalled applications.
I nix-env operationscreate new userenvironments in thestore.
I We can atomicallyswitch between them.
I These are roots of thegarbage collector.
PATH
/nix/.../profiles
current
43
/nix/store
rpdqxnilb0cg...-firefox-3.5.4bin
firefoxaqn3wygq9jzk...-openssh-5.2p1
binssh
i3d9vh6d8ip1...-user-envbin
sshfirefox
(nix-collect-garbage)
Sander van der Burg Deploying NPM packages with the Nix package manager
Nix expressions
openssh.nix
{ stdenv, fetchurl, openssl, zlib }:
stdenv.mkDerivation {
name = "openssh-4.6p1";
src = fetchurl {
url = http://.../openssh-4.6p1.tar.gz;
sha256 = "0fpjlr3bfind0y94bk442x2p...";
};
buildCommand = ’’
tar xjf $src
./configure --prefix=$out --with-openssl=${openssl}
make; make install
’’;
}
Nix complements existing build tools with a pure environment
You can also run other build tools in an environment, e.g.cmake, ant, scons
Sander van der Burg Deploying NPM packages with the Nix package manager
Pure environments
Precautions in service of making build results bit-identical(regardless on what machine the package has been built):
Clearing (most) environment variables or setting them todummy values
Modifying search environment variables to contain Nix storepaths to only specified dependencies (e.g. PATH, PYTHONPATH,CLASSPATH, PERL5LIB)
Using designated temp folders and output folders
Making packages immutable by making their files read-only
Resetting timestamps to 1
Chroot environments/namespaces
Restricting network access
Sander van der Burg Deploying NPM packages with the Nix package manager
What about Node.js/NPM projects?
NPM is Node.js’ ubiquitous deployment tool.
NPM also does dependency management in addition to buildmanagement
NPM’s dependency management conflicts with Nix’sdependency management
Sander van der Burg Deploying NPM packages with the Nix package manager
package.json configuration
{"name" :"nijs",
"version" :"0.0.23",
"description" :"An internal DSL for the Nix package manager in JavaScript",
"repository" :{"type" :"git",
"url" :"https://github.com/svanderburg/nijs.git"
},"author" :"Sander van der Burg",
"license" :"MIT",
"main" :"./lib/nijs",
"dependencies" :{"optparse" :">= 1.0.3",
"slasp": "0.0.4"
}}
$ npm install$ ls node modules/optparse slasp
Sander van der Burg Deploying NPM packages with the Nix package manager
Dealing with a conflicting dependency manager
Solution: substitute the conflicting dependency manager byproviding the dependencies with Nix first!
Sander van der Burg Deploying NPM packages with the Nix package manager
Solution: substitute NPM’s dependency management
{ stdenv, fetchgit, nodejs, optparse, slasp }:
let
nodeSources = ...
in
stdenv.mkDerivation {
name = "nijs-0.0.23";
src = fetchgit {
src = https://github.com/svanderburg/nijs.git;
rev = "...";
sha256 = "...";
};
buildInputs = [ nodejs ];
buildCommand = ’’
# Compose node_modules/ folder from the dependencies
...
# Perform the build by running npm install
# Since a node_modules/ folder with the dependencies already exists,
# NPM will not obtain them
npm --registry http://www.example.com --nodedir=${nodeSources} install
’’;
}
Sander van der Burg Deploying NPM packages with the Nix package manager
Solution: substitute NPM’s dependency management
{ stdenv, fetchgit, nodejs, optparse, slasp }:
let
nodeSources = ...
in
stdenv.mkDerivation {
name = "nijs-0.0.23";
src = fetchgit {
src = https://github.com/svanderburg/nijs.git;
rev = "...";
sha256 = "...";
};
buildInputs = [ nodejs ];
buildCommand = ’’
# Compose node_modules/ folder from the dependencies
...
# Perform the build by running npm install
# Since a node_modules/ folder with the dependencies already exists,
# NPM will not obtain them
npm --registry http://www.example.com --nodedir=${nodeSources} install
’’;
}
Sander van der Burg Deploying NPM packages with the Nix package manager
Substituting dependencies
We may be able to automatically generate a Nix expressionfrom a package.json file since they capture similar proper-ties!
Solution: substitute NPM’s dependency management
{ stdenv, fetchgit, nodejs, optparse, slasp }:
let
nodeSources = ...
in
stdenv.mkDerivation {
name = "nijs-0.0.23";
src = fetchgit {
src = https://github.com/svanderburg/nijs.git;
rev = "...";
sha256 = "...";
};;
buildInputs = [ nodejs ];
buildCommand = ’’
# Compose node_modules/ folder from the dependencies
...
# Perform the build by running npm install
# Since a node_modules/ folder with the dependencies already exists,
# NPM will not obtain them
npm --registry http://www.example.com --nodedir=${nodeSources} install
’’;
}
Sander van der Burg Deploying NPM packages with the Nix package manager
Substituting dependencies
Appears to be straight forward! Is it really that straightforward?
Generating Nix expressions
It is actually much more complicated!
Sander van der Burg Deploying NPM packages with the Nix package manager
NPM dependency classes
Dependencies:
Run-time dependencies of a package
Development dependencies:
Build-time dependencies, e.g. transpilersShould only be installed if they have been requested
Peer dependencies:
Check whether a shared dependency does not conflict.In old versions of NPM, missing peer dependencies were alsoinstalled.
Bundled dependencies:
Statically bundled with a package. No need to install.
Optional dependencies:
Dependencies that are allowed to break.Typically used to bundle platform-specific code.Permitting errors (especially non-deterministic ones)incompatible with Nix deployment model
Sander van der Burg Deploying NPM packages with the Nix package manager
NPM dependency classes
Dependencies:
Run-time dependencies of a package
Development dependencies:
Build-time dependencies, e.g. transpilersShould only be installed if they have been requested
Peer dependencies:
Check whether a shared dependency does not conflict.In old versions of NPM, missing peer dependencies were alsoinstalled.
Bundled dependencies:
Statically bundled with a package. No need to install.
Optional dependencies:
Dependencies that are allowed to break.Typically used to bundle platform-specific code.Permitting errors (especially non-deterministic ones)incompatible with Nix deployment model
Sander van der Burg Deploying NPM packages with the Nix package manager
Interpreting dependency classes
We only need to consider dependencies and development de-pendencies (if they are requested) and substract the bundleddependencies
Version specifiers: NPM registry
NPM version specifiers are nominal and semantically versioned:
Precise version numbers: e.g. 1.0.0, 2.1.1
Version ranges and wildcards: e.g. 1.0.x, *, >= 1.0.1
Tags: latest, beta
The above version specifiers refer to packages obtained from theNPM registry.
Sander van der Burg Deploying NPM packages with the Nix package manager
Version specifiers: external packages
Local filesystem paths, e.g. /home/sander/nijs
External URLs, e.g.http://example.com/packages/nijs-0.0.23.tgz
Git URLs, e.g.https://github.com/svanderburg/nijs.git
GitHub, BitBucket, GitLab, e.g. github:svanderburg/nijs
Above specifiers obtain packages from other sources than theregistry. They indirectly resolve to a package with a name andversion number.
Sander van der Burg Deploying NPM packages with the Nix package manager
Version specifiers: translation to Nix
Nix version specifiers are exact (any point of variation is reflectedin the hash prefix):
/nix/store/rpdqxnilb0cg...-firefox-3.5.4
To make a translation from NPM version specifiers to exactversion specifiers:
We must snapshot a version range and pinpoint the resolvedversion (version ranges are unsupported in Nix)
We must supply the output hash to make it deterministic:
src = fetchurl {
url = "https://registry.npmjs.org/nijs/-/nijs-0.0.23.tgz";
sha1 = "dbf8f4a0acafbe3b8d9b71c24cbd1d851de6c31a";
};
Sander van der Burg Deploying NPM packages with the Nix package manager
Version specifiers: translation to Nix
Nix version specifiers are exact (any point of variation is reflectedin the hash prefix):
/nix/store/rpdqxnilb0cg...-firefox-3.5.4
To make a translation from NPM version specifiers to exactversion specifiers:
We must snapshot a version range and pinpoint the resolvedversion (version ranges are unsupported in Nix)
We must supply the output hash to make it deterministic:
src = fetchurl {
url = "https://registry.npmjs.org/nijs/-/nijs-0.0.23.tgz";
sha1 = "dbf8f4a0acafbe3b8d9b71c24cbd1d851de6c31a";
};
Sander van der Burg Deploying NPM packages with the Nix package manager
Private, shared and cyclic dependencies
When including a package as a CommonJS module:
./node2nix/node modules/nijs/lib/execute/index.js
var slasp = require(’slasp’);
The module loader searches recursively upwards in node modules/
sub folders for the corresponding package:
./node2nix/nijs/node_modules/slasp
./node2nix/node_modules/slasp
./node_modules/slasp
Sander van der Burg Deploying NPM packages with the Nix package manager
Private, shared and cyclic dependencies
Private dependency: a package residing in a package’snode modules/ sub folder.
Shared dependency: a package residing in a parent directory’snode modules/ sub folder.
Sander van der Burg Deploying NPM packages with the Nix package manager
Private, shared and cyclic dependencies
When installing dependencies with NPM, NPM will not install adependency privately if a conforming shared dependency exists.Some undesired implications:
A version range does not always yield the latest version thatfits in a version range.
Cyclic dependencies are permitted. Bad practice, as packagesare supposed to be units of reuse.
Sander van der Burg Deploying NPM packages with the Nix package manager
Flat-module installations
npm < 3.x installs dependencies privately by default (unless aconforming dependency exists in any parent node modules/
folder):
webapp/
...
package.json
node_modules/
express/
...
package.json
node_modules/
accepts/
array-flatten/
content-disposition/
...
ejs/
...
package.json
Sander van der Burg Deploying NPM packages with the Nix package manager
Flat-module installations
npm ≥ 3.x implements flat-module installations (moving packagesas high as possible in the node modules/ hierarchy):
webapp/
...
package.json
node_modules/
accepts/
array-flatten/
content-disposition/
express/
...
package.json
ejs/
...
package.json
Because no packages conflict, they all appear in the top levelnode modules/ folder. Implications:
Shorter path names, some degree of deduplicationThe order in which packages are installed matters
Sander van der Burg Deploying NPM packages with the Nix package manager
Simulating flat-module installations
We cannot build each NPM dependency as a Nix package, with aseparate store path:
CommonJS module resolves its own symlink location. Somerelative paths may not work:
var slasp = require(’../slasp’);
Flattening the module hierarchy is imperative.
Nix packages are made immutable after they have been built.
Solution: compose the entire dependency tree statically ahead oftime in one Nix package
Sander van der Burg Deploying NPM packages with the Nix package manager
Replacing impure version specifiers
Some version specifiers (e.g. tags and Git URLs), trigger NPM toalways check the remote locations for changes in each npm
install run:
Replace these version specifiers by: ’*’. Downside: itsometimes confuses NPM, in particular flat moduleinstallations.
Better solution: create fake processes (e.g git) giving adeterministic result. This is still an open issue.
Sander van der Burg Deploying NPM packages with the Nix package manager
The solution1: node2nix
1Sort of, but not entirely :-)Sander van der Burg Deploying NPM packages with the Nix package manager
Example: using node2nix on project-level
Generating Nix expressions from a package.json file:
$ node2nix
Building the project as a Nix package:
$ nix-build -A package
./result/bin/node2nix --help
Opening a development shell:
$ nix-shell -A shell
$ node bin/node2nix.js --help
Sander van der Burg Deploying NPM packages with the Nix package manager
Example: using node2nix on a package set
custom-packages.json
[
"node2nix"
,"bower"
,{"nijs": "https://github.com/svanderburg/nijs.git" },{"grunt-cli": "1.2.0" }]
Generating Nix expressions:
$ node2nix -i custom-packages.json
Installing an NPM package with Nix:
$ nix-env -f default.nix -iA node2nix
$ node2nix --help
Sander van der Burg Deploying NPM packages with the Nix package manager
Example: simulating global dependencies
Global packages do not exist in Nix build environments. This isproblematic for certain kinds of projects, such as Grunt projects,requiring the grunt-cli to be installed globally:
package.json
{"name": "grunt-test",
"version": "0.0.1",
"private": "true",
"devDependencies": {"grunt": "*",
"grunt-contrib-jshint": "*",
"grunt-contrib-watch": "*"
}}
$ npm install
$ grunt build
Sander van der Burg Deploying NPM packages with the Nix package manager
Example: simulating global dependencies
We can supply global packages as extra packages:
supplement.json
[
"grunt-cli"
]
And generate the expressions as follows:
$ node2nix -d -i package.json \
--supplement-input supplement.json
Sander van der Burg Deploying NPM packages with the Nix package manager
Example: simulating global dependencies
override.nix
{ pkgs ? import <nixpkgs> {}
, system ? builtins.currentSystem
}:
let
nodePackages = import ./default.nix {
inherit pkgs system;
};
in
nodePackages // {
package = nodePackages.package.override {
postInstall = "grunt";
};
}
$ nix-build override.nix
Sander van der Burg Deploying NPM packages with the Nix package manager
Conclusion
I have explained Nix and NPM’s deployment concepts
I have described node2nix generating Nix expressions frompackage.json configurations, making it possible to deployNPM packages with the Nix package manager.
Besides the package manager, it also becomes possible todeploy entire systems using Node.js with NixOps and Disnix.
Sander van der Burg Deploying NPM packages with the Nix package manager
Lessons
Naming packages: a name and version number typically doesnot suffice!
Nix uses hash codes derived from all build inputs.NET global assembly cache: strong names
Organization: isolate dependencies
Cyclic dependencies: disallow them – packages are supposedto be units of reuse
Sander van der Burg Deploying NPM packages with the Nix package manager
Related work: other generators
npm2nix: Original generation attempt done by Shea Levy.Composes Nix store paths for each dependency and symlinksthem.
Faster, but less accurate when dealing when shareddependenciesDoes not support flat module installations or cyclicdependencies
nixfromnpm: Implementation of npm2nix’s Nix concepts inHaskell. Can partially regenerate expressions.
Sander van der Burg Deploying NPM packages with the Nix package manager
Related work: NPM alternatives
yarn: parallelization for speed improvements, version locking
ied: parallelization, content-addressable store, atomicity,maximal sharing of dependencies (some of its concepts areheavily inspired by Nix according to its author :-) )
Sander van der Burg Deploying NPM packages with the Nix package manager
References
Nix project: http://nixos.org.
Nix package manager: http://nixos.org/nix.
node2nix: https://github.com/svanderburg/node2nix.
Sander van der Burg Deploying NPM packages with the Nix package manager
Questions
Sander van der Burg Deploying NPM packages with the Nix package manager
top related