nixos Entry: First time install on X201 Date: Sun Sep 10 15:18:59 EDT 2017 https://nixos.org/nixos/download.html Download minimal 64bit .iso from website dd it to USB flash drive and boot Alt-F8 is manual Alt-F1 is console # fdisk # mkfs.ext4 -L nixos /dev/sda1 # mount /dev/disk/by-label/nixos /mnt # nixos-generate-config --root /mnt # nano /mnt/etc/nixos/configuration.nix edit: boot.loader.grub.device = "/dev/sda"; # nixos-install Provide root password at end of install # reboot OK on 320G 2.5 internal sata drive. OK on 16G SD card. Likely will have to patch the grub device after boot (if it boots). # nix-env -i emacs git darcs openssh After editing /etc/nixos/configuration.nix # nixos-rebuild switch # wpa_passphrase /etc/wpa_supplicant.co Entry: Problem with wpa_supplicant Date: Sun Sep 10 16:48:55 EDT 2017 ... building path(s) ‘/nix/store/j8n4qvh0xw4i50bn4zxvvd0yr6dfcmbs-nixos-system-nixos-17.03.18 updating GRUB 2 menu... stopping the following units: systemd-sysctl.service, systemd-udevd-control.socket, systemd-udevd-kernel.socket, systemd-udevd.service activating the configuration... setting up /etc... setting up tmpfiles reloading the following units: dbus.service restarting the following units: sshd.service starting the following units: systemd-sysctl.service, systemd-udevd-control.socket, systemd-udevd-kernel.socket the following new units were started: systemd-vconsole-setup.service warning: the following units failed: wpa_supplicant.service ● wpa_supplicant.service - WPA Supplicant Loaded: loaded (/nix/store/nqr89hh8ak67il72m8hs0bypngxgz2xy-unit-wpa_supplicant.service/wpa_supplicant.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2017-09-10 16:45:19 EDT; 18ms ago Process: 8233 ExecStart=/nix/store/mcjsz71bxc294fayyjwg3bnbpmpvapgi-unit-script/bin/wpa_supplicant-start (code=exited, status=255) Main PID: 8233 (code=exited, status=255) Sep 10 16:45:19 nixos systemd[1]: Started WPA Supplicant. Sep 10 16:45:19 nixos systemd[1]: wpa_supplicant.service: Main process exited, code=exited, status=255/n/a Sep 10 16:45:19 nixos systemd[1]: wpa_supplicant.service: Unit entered failed state. Sep 10 16:45:19 nixos systemd[1]: wpa_supplicant.service: Failed with result 'exit-code'. warning: error(s) occurred while switching to the new configuration Entry: Next Date: Sun Sep 10 17:00:54 EDT 2017 TODO: - declarative vs. ad-hoc - wpa_supplicant - xmonad OK: - emacs Entry: ad-hoc vs declarative Date: Sun Sep 10 18:02:03 EDT 2017 Use "nix-env -iA" instead of "nix-env -i" Entry: intro Date: Sun Sep 10 18:30:11 EDT 2017 https://ebzzry.io/en/nix/ Entry: system vs. user Date: Sun Sep 10 20:04:49 EDT 2017 Probably good to separate system vs. user programs, since nix also works on other systems. Assuming that later it is possible to move things around and factor them out, let's not do that for now. Entry: Add packages Date: Sun Sep 10 20:53:46 EDT 2017 Might be good to also figure out as soon as possible how to add nix packages. E.g look at what is installed in emacs through the package manager: flycheck, rust-mode, ... That's currently still a manual step. Entry: xmonad Date: Mon Sep 11 14:14:40 EDT 2017 xmonad.enableContribAndExtras = true this also pulls in ghc Entry: haskell development. Date: Sun Nov 5 11:56:03 EST 2017 That should not really be a problem right? Stack seems to be the way to go these days, so let's do that. How to build haskell project in NixOS with stack https://gist.github.com/kuznero/19675f3c11363e0c5b424005938bc302 Entry: Ad-hoc vs. declarative Date: Sun Nov 5 12:10:09 EST 2017 How do I choose between declarative and ad-hoc installation? Or else: how to move packages from ad-hoc to declarative? Entry: next: openvpn Date: Sun Nov 5 12:25:58 EST 2017 So here's a decision to make: some config will be per host, like openvpn. Reserve a directory for things like that so they are easy to update when cloning a host from a global config file + some per host config files. Is it ok to put this in /etc ? I'm using /etc/host as the place where nix config looks for files. For openvpn, config path will be /etc/host/openvpn/zoo.conf [root@nixos:/etc]# ls -al |grep -v static Entry: conclusion for now? Date: Thu Nov 9 12:11:28 EST 2017 It pretty much works, but there are some problems: - Declarative config is annoying. Is there a way to keep using ad-hoc, and convert it to declarative later on? - SD card gives errors on wakeup which seems to cause filesystem inconsistencies which causes a readonly mount of root filesystem or even boot problems -- had to take out sd card and run fsck on other machine. Overall, I will need some time to transition. I've switched back to Debian on zora to be able to get work done. EDIT: 11/0, giving it another try. This system is designed very well and I think it's worth it living with the limitations. Entry: Nix is C/C++ and Perl? Date: Fri Nov 10 09:05:01 EST 2017 There I was, somehow thinking this was written in Haskell. And more revealing, being disappointed and wanting to run away from it! EDIT: https://nixos.wiki/wiki/FAQ#Why_is_Nix_written_in_C.2B.2B_rather_than_a_functional_language_like_Haskell.3F Entry: Nixos vs nix Date: Fri Nov 10 11:07:33 EST 2017 It looks like it is best to separate system from user install. For now I do not know how to make the ad-hoc installs declarative, so let's just use a single bootstrap command using nix-env. Install nix on core? No. Let's use a barebones debian host. Maybe let the nixos thing go for a while and assume basic semantics to be a base debian install that can be cloned manually. Do I have one? Entry: Installing nix on zoe Date: Fri Nov 10 11:38:26 EST 2017 Actually I can just use a real host, assuming this will be easy to uninstall. Make sure /nix is readable by user. It cannot be a symlink. $ curl https://nixos.org/nix/install | sh Is it possible to install multi-user nix on Debian? https://gist.github.com/joepie91/043a51a7b70be5f50f1d Though it feels as if I should stick to some base Debian install for anything that is a server, and use nix only for setting up a build environment. Entry: Split install into system and user Date: Fri Nov 10 14:10:34 EST 2017 System should have basics to be able to run as terminal and have darcs,git etc to be able to manage some archives. Rest should be in user profile. How to list what is in user profile? Entry: Creating a nix package Date: Fri Nov 10 15:26:05 EST 2017 One reason to use nix is to have a stable build environment with clear updates. This is where all the "output" of /etc/net should go. Entry: Upgrades Date: Fri Nov 10 16:24:34 EST 2017 From: https://github.com/NixOS/nix/issues/614 It is mentioned that a channel subscription is needed. Then your user profile can be upgraded: nix-env --leq -u '*' So it seems that configuration specification does not include versions. How to fix a particular version? Entry: Profiles Date: Fri Nov 10 16:37:48 EST 2017 nix-env -iA nixos.racket Seems to create a new profile: tom@zora:/nix/var/nix/profiles/per-user/tom$ ls -al total 16 drwxr-xr-x 2 tom users 4096 Nov 10 16:35 . drwxrwxrwt 4 root root 4096 Sep 10 16:01 .. lrwxrwxrwx 1 tom users 14 Nov 10 16:35 profile -> profile-2-link lrwxrwxrwx 1 tom users 60 Nov 10 16:15 profile-1-link -> /nix/store/pp5s9ngnaby5aypcq7dfn4nzd8rbaar2-user-environment lrwxrwxrwx 1 tom users 60 Nov 10 16:35 profile-2-link -> /nix/store/dkz38aibd5046r68zmp6rlvkq5qwl5i7-user-environment tom@zora:/nix/var/nix/profiles/per-user/tom$ nix-env -iA nixos.rustc Which then created a new profile in the GC roots. Ok that's neat. Ok, so how to copy a profile from one host to another? https://stackoverflow.com/questions/28159181/how-to-copy-a-nix-profile nix-env -q tom@zora:~$ nix-env -q ghc-8.0.2 racket-6.8 rustc-1.15.1 nix-copy-closure --help Which is very very cool! For now this is enough. But there are other ways to do it: https://nixos.wiki/wiki/FAQ#How_can_I_manage_software_with_nix-env_like_with_configuration.nix.3F Entry: nixos in lxc container Date: Fri Nov 10 17:41:52 EST 2017 scroll down: https://github.com/NixOS/nixpkgs/issues/9735 On a nixos machine: nix-build -E '(import {}).containerTarball.x86_64-linux' Seems to work after some minor edits: root@zoo:/lxc/nixos# cat config lxc.network.type = veth lxc.network.link = br0 lxc.network.flags = up lxc.network.hwaddr = 00:16:3e:f6:ff:de lxc.rootfs = /var/lib/lxc/nixos/rootfs lxc.rootfs.backend = dir # Added for NixOS. lxc.include = /usr/share/lxc/config/common.conf lxc.init_cmd = /init Entry: Emacs with packages Date: Fri Nov 10 21:47:31 EST 2017 https://nixos.org/nixos/manual/ Manual has some section in it about installing emacs with selected packages. Let's give that a try. build errors: output path ‘/nix/store/1ii1lnwpgpgrb3isc87f0b5f814z3wi6-frame-cmds.el’ has sha256 hash ‘143szv36bvcgrrima8ss1nn6v9s3lrlqbjf240lyx4mmv24lxds0’ when ‘05mdds242vcav9gy15phvlk9q8iy29wnc8bp0c7i6z6bblairfzj’ was expected Mentioned here https://github.com/NixOS/nixpkgs/issues/18934 that this might have problems with packages getting out-of-date. Also mentioned to use 'use-package' in emacs, which should be able to declaratively specify the packages except for the versions. Might be good enough. https://github.com/jwiegley/use-package So how to install that? Also: https://github.com/travisbhartwell/nix-emacs Entry: Next Date: Sat Nov 11 08:53:25 EST 2017 - continue setting up nixos VM: emacs, fix up pool/bootstrap maybe? seems ok for now, openvpn not needed a ssh access works - allow ssh root logins with keys: works out of the box once authorized keys are installed. - try nix-copy-closure between two instances - make /etc/net into a package so it can pull in the dependencies it needs (sqlite, erlang, and gcc as build dep, ...) - start building geno->pheno build with stubs for rust, haskell, C and erlang projects. - debug the weird emacs ssh / routing problem Entry: nix-copy-closure Date: Sat Nov 11 09:57:00 EST 2017 https://blog.joel.mx/posts/how-to-use-nix-copy-closure-step-by-step tom@zora:~$ nix-env -q --no-name --out-path tcpdump /nix/store/yqlyyybdkjmg9x22zy9acjxkfnl1gbxy-tcpdump-4.9.2 HOST=nixos P=$(nix-env -q --no-name --out-path tcpdump) nix-copy-closure --sign --include-outputs $HOST $P ssh $HOST nix-env -i $P See pool/bin/nix-copy.sh But it looks like it is best not to do this and just install packages by name+version. Entry: agda Date: Sat Nov 11 10:58:33 EST 2017 tom@nixos:~/bin$ nix-env -f "" -iA haskellPackages.Agda Entry: nixpkgs Date: Sat Nov 11 11:11:40 EST 2017 Agda needs nixpkgs. What is that? EDIT: collection of packages outside of the main release. Entry: /nix on NFS Date: Sat Nov 11 13:41:23 EST 2017 What about exporting /nix readonly to the local net? This would be a simple way to have all local machines run the same software without need for separate management, e.g. for clustering purposes. Local machines then can have simple links to /nix as if it was local. Management will happen on the central machine. https://github.com/nixos-users/wiki/wiki/NFS Anyway autofs is not needed for that is this would only be for Debian machines. EDIT: Tried on lroom, readonly. Seems to work ok. root@lroom:~# cat /etc/fstab |grep /nix 10.1.3.2:/var/lib/lxc/ph/rootfs/nix /nix nfs ro 0 0 tom@lroom:~$ cat .bashrc |grep nix [ -f /etc/debian_version ] && PATH=$PATH:~/.nix-profile/bin Entry: Mouse acceleration Date: Sat Nov 11 13:53:12 EST 2017 Is set different than my Debian setups. How to change? Entry: autofs Date: Sat Nov 11 14:42:38 EST 2017 Can't find a standard way to replicate Debian's /net There are some .nix files around though. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/misc/autofs.nix Entry: Building a package Date: Sat Nov 11 15:59:10 EST 2017 See FAQ nix-shell -p pkgconfig zlib pkg-config --cflags zlib What roots are the -dev packages stored in? Seems they are not rooted by default; nix-shell needs "--add-root". Not sure how this works since docs is missing. An active nix-shell does act as a root, so this is consistent. In practice: don't do unnecessary gc. Next: try this out on a trivial hello world C program. Inputs: - github repo with hello.c source code - gcc, make, ... Then look at what comes out: the drv file and the closure. Entry: Cloning a nix system? Date: Sat Nov 11 18:19:29 EST 2017 Proper way is probably to perform a standard install, then copy the configuration and ad-hoc installs over. Not going to do that for 16G->32G upgrade; just copying the image. Doing it while system is live which will likely give some errors but doing it anyway. EDIT: Seems to work. Entry: Closures Date: Sun Nov 12 08:27:32 EST 2017 I still don't understand how a closure is stored. It appears to be a simplification of the .drv file, where all build dependencies are removed. Entry: Installing a Haskell package Date: Sun Nov 12 08:38:02 EST 2017 nix-env -f "" -i -A haskellPackages.pretty-show Entry: create package Date: Sun Nov 12 17:21:01 EST 2017 https://nixos.wiki/wiki/Create_and_debug_nix_packages https://nixos.org/nixpkgs/manual/ https://nixos.org/releases/nix/nix-1.7/manual/ http://lethalman.blogspot.it/2014/08/nix-pill-8-generic-builders.html So what to do here? Difference between nixpkg and stand-alone? Does it make sense to start a local package repo? Entry: example: start irc when network goes up Date: Sun Nov 12 17:54:43 EST 2017 https://nixos.wiki/wiki/NixOS:extend_NixOS Entry: Run time dependencies Date: Mon Nov 13 22:31:06 EST 2017 http://lethalman.blogspot.it/2014/08/nix-pill-9-automatic-runtime.html 1 Dump the derivation as NAR, a serialization of the derivation output. Works fine whether it's a single file or a directory. 2 For each build dependency .drv and its relative out path, search the contents of the NAR for this out path. 3 If found, then it's a runtime dependency. Entry: local cache Date: Mon Nov 13 22:53:58 EST 2017 https://nixos.org/nix/manual/ $ nix-env -i firefox --option ssh-substituter-hosts alice@avalon Entry: building something Date: Sat Nov 18 10:13:20 EST 2017 [nix-shell:~/git/rai]$ make make -C src make[1]: Entering directory '/home/tom/pub/git/rai/src' gcc -g -Wall -Wno-unused-variable -I. -I..//src -I../copy -fPIC -std=gnu99 -ffast-math -O3 -I/usr/local/pd/src -lm -o sp_host.elf sp_host.c proc.o proc_sp.o /home/tom/.nix-profile/bin/ld: cannot find crt1.o: No such file or directory /home/tom/.nix-profile/bin/ld: cannot find crti.o: No such file or directory /home/tom/.nix-profile/bin/ld: cannot find -lm /home/tom/.nix-profile/bin/ld: cannot find -lgcc_s /home/tom/.nix-profile/bin/ld: cannot find -lc /home/tom/.nix-profile/bin/ld: cannot find -lgcc_s /home/tom/.nix-profile/bin/ld: cannot find crtn.o: No such file or directory collect2: error: ld returned 1 exit status make[1]: *** [rules.mk:115: sp_host.elf] Error 1 make[1]: Leaving directory '/home/tom/pub/git/rai/src' make: *** [Makefile:7: all] Error 2 Also tried without "nix-shell -p", same result. I had installed "gcc" and "binuitils" manually. Maybe what I misunderstand here, is that it is not possible to install a build environment in the normal environment? Or at least, not desirable. At the very least, make a nix configuration file that declares the env, then use nix-shell. This seems to work with "nix-shell" with import {}; stdenv.mkDerivation { name = "rai-current"; } Next: to build rai, racket is needed. How to bring that into the environment? Also: use "nix-shell --pure", because I did have it installed in the user profile. buildInputs = [racket]; Next: needs racket packages. This is a problem in general when deps come from the internet using some external package manager. Same for emacs e.g. Maybe rai is not the example to start with. I got it to build eventually, but only by: - installing local ~/.racket packages - making the "link" How to install those somewhere else? [nix-shell:~/pub/git/rai]$ raco pkg install --scope installation rsound open-output-file: cannot open output file path: /nix/store/wwsjz5r70k433s6yl3knz2z50fdw8k0p-racket-6.8/share/racket/pkgs/.LOCKpkgs.rktd What is needed is a racket with packages. Entry: missing man pages Date: Sat Nov 18 10:50:42 EST 2017 "nix-env -i nix" did install it in the store: tom@phoo:/nix/store$ find -name nix-env.1.gz ./30yi6znfz3vn6fqpaz9l5xxxj1xcjf2v-nix-1.11.14-man/share/man/man1/nix-env.1.gz but it is not accessible from the profile I have no idea how to make sure this ends up in the profile. After running "nix-collect-garbage" the file is gone. It seems this has something to do with the original LXC setup. On zora, the man page is installed in: /nix/var/nix/profiles/system/sw/share/man/man1/nix-env.1.gz Check this out: https://github.com/NixOS/nixpkgs/issues/24717 .config/nixpkgs/overlays/default.nix: self: super: { # add man to the outputs to install for nix nix = super.nix.overrideAttrs (oldAttrs: { meta = oldAttrs.meta // { outputsToInstall = [ "out" "man" ]; }; }); } tom@ph:~/.config$ ln -s ../dot/config/nixpkgs . Ok that worked. So what is an overlay? https://nixos.org/nix-dev/2016-December/022386.html https://nixos.org/nixpkgs/manual/ (grep for "overlay") Entry: Building things in nixos Date: Sat Nov 18 12:02:22 EST 2017 - wrap everything (all dependencies) as a nix pkg or - set up a dedicated environment Maybe it is best to do it two-step always? I've noticed some issues for external package manager that insist on writing dot directories with packages. Basically, if declarative is not possible, make it at least idempotent. So let's start with defining the idea of a default build context. Once that works, it might be possible to tweak things so they can build independently. Entry: Changing "HOME" ? Date: Sat Nov 18 12:12:55 EST 2017 Trying that out for now.. Yep works. So let's do that. Entry: A nix development machine Date: Sat Nov 18 12:30:53 EST 2017 1. A development machine is a collection of compilers (curren build environments) that allow to compile source into some form of non-standard output. 2. Outputs are not necessarily packages used as part of the build system. If this is needed, build a wrapper. 3. All build environments should be isolated from each other, to be able to work on multiple projects. Entry: Requesting a specific version? Date: Sat Nov 18 13:42:37 EST 2017 On core nix I have racket 6.11, on phoo there is 6.8 Entry: Next Date: Sat Nov 18 14:31:00 EST 2017 - find out how to pin versions in default.nix for nix-shell - create some packages, cache them locally Entry: Pinning versions Date: Sat Nov 18 15:49:32 EST 2017 This doesn't seem to be possible if nixpkgs doesn't actually have the package version. Only resort is to create a local package hierarchy, and name the packages explicitly. So let's start doing this with racket. Start here: tom@core:~/git$ git clone https://github.com/NixOS/nixpkgs Having a look at how racket is built. in pkgs/top-level/all-packages.nix : racket = callPackage ../development/interpreters/racket { # racket 6.11 doesn't build with gcc6 + recent glibc: # https://github.com/racket/racket/pull/1886 # https://github.com/NixOS/nixpkgs/pull/31017#issuecomment-343574769 stdenv = overrideCC stdenv gcc7; }; Now, what I don't understand is how nixpkgs evolves over time. E.g. I have one fixed at racket-6.8 and one at racket-6.11 6.8 https://github.com/NixOS/nixpkgs/blob/17.03/pkgs/development/interpreters/racket/default.nix 6.10.1 https://github.com/NixOS/nixpkgs/blob/17.09/pkgs/development/interpreters/racket/default.nix So it seems that nixos nixpkgs seem to be maintained as git tags. Entry: How to build a package? Date: Sat Nov 18 16:16:44 EST 2017 tom@phoo:~/nixpkgs/pkgs/development/interpreters/racket$ nix-shell error: cannot auto-call a function that has an argument without a default value (‘stdenv’) nix-build -E '(import {}).callPackage ./default.nix {}' --option build-cores 0 # Also in default.nix, set enableParallelBuilding = true; Not sure why that was off? Maybe nixos does parallelism per package? Entry: Closures vs. build instructions Date: Sat Nov 18 17:07:58 EST 2017 So I would want the collection of .nix files that lead up to a particluar closure. In practice, it is likely sufficient to: - start with a well-defined root (e.g. 17.03 or 17.09) - keep a copy of that root as nixpkgs - add modified packages on top of that Entry: So what is a channel? And can I "patch" against one? Date: Sat Nov 18 17:31:31 EST 2017 Entry: buildroot Date: Sat Nov 18 19:10:49 EST 2017 Buildroot build fails: ~/exo/br/target/i586/build/host-gcc-initial-5.4.0/libcpp/expr.c ./../libcpp -I. -I../../libcpp/../include -I../../libcpp/include -c -o macro.o -MT macro.o -MMD -MP -MF .deps/macro.Tpo ../../libcpp/macro.c ../../../libcpp/expr.c: In function ‘unsigned int cpp_classify_number(cpp_reader*, const cpp_token*, const char**, source_location)’: ../../../libcpp/expr.c:681:18: error: format not a string literal and no format arguments [-Werror=format-security] 0, message); ^ ../../../libcpp/expr.c:684:39: error: format not a string literal and no format arguments [-Werror=format-security] virtual_location, 0, message); ^ [nix-shell:/home/tom_pub/exo]$ gcc --version gcc (GCC) 6.4.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Probably fixable by upgrading buildroot. Current is 54242226f034c1245d379df5f98cefcc13553437 br/buildroot (2017.02-563-g54242226f) 46a082afec58d8081a3a896a7f64b9cbfb71a19c rebar3 (3.3.6-119-g46a082af) tom@core:~/exo/br/buildroot$ git pull https://github.com/buildroot/buildroot.git Updated 2017.02.x branch (bugfix branch?) For now this doesn't really seem to work. Stick to the buildroot vm. Entry: Packages and deps Date: Sat Nov 18 22:08:17 EST 2017 So, adding package on top of nixpkg is simple. But how to do 2 packages, with one depending on the other? They are just derivations. The easiest way is probably to have them refer each other by path, and have a top file that contains and builds the leaf derivations. It is the build of the deriviations that gets cached. Entry: side chain repo Date: Mon Nov 20 22:17:43 EST 2017 Building a set of related packages looks very much like a channel, so how to build a channel? https://github.com/NixOS/nixpkgs/issues/2793 https://nixos.org/nix/manual/#sec-nix-push Basically, just nix-push, some server, and nix-build / nix-env from .nix package. The only point of that is to make binary caches. Just start with an ad-hoc set of packages as described in the previous post, then start composing them. What might be useful is hydra, to build all variants of a project from version control. This gives channels for free. Entry: building libopencm3 with arm-gcc-embedded Date: Tue Nov 21 15:29:38 EST 2017 Seemed to work. Now I need a custom build script to properly perform the install phase. installing install flags: install SHELL=/nix/store/4sp0sc2wxas7213x59smrwhliqmhc5yk-bash-4.4-p12/bin/bash make: *** No rule to make target 'install'. Stop. builder for ‘/nix/store/qrxmykdg7ddwl1sarr0rdmr3zq4sl7g2-libopencm3.drv’ failed with exit code 2 error: build of ‘/nix/store/qrxmykdg7ddwl1sarr0rdmr3zq4sl7g2-libopencm3.drv’ failed Makefile:3: recipe for target 'all' failed make: *** [all] Error 100 Entry: default build script / builder.sh Date: Tue Nov 21 15:40:54 EST 2017 what does it do? source $stdenv/setup genericBuild Not too hard to find it ls -l /nix/store/*-stdenv/setup From inspection, it seems possible to configure the phases, and likely not run install that way. # Just a hack to make it continue installTargets = "--help"; Problem is that it really doesn't install anything, so the output directory is empty. The simplest is probably to keep the standard build script as it sets up the environment, but to add a wrapper makefile. makefile = "wrap.mk"; all: make all install: cd libopencm3 ; cp -a include lib $(out)/ To get there, I've added "pwd ; ls -al ; set" in there to get an idea of what it is doing. Entry: output vs "inside" of a build Date: Tue Nov 21 17:32:35 EST 2017 so I'm installing a c compiler and a library, but these are not final products. how to have them as final? e.g. I want to have that arm compiler and C library in my environment. how to use that .nix file to set up a nix shell? So basically, nix-build, nix-shell and nix-env can be used with the .nix file. nix-build and nix-env expose only the end results while nix-shell has all the dependencies available. to remove a profile from nix-env: $ nix-env --list-generations Entry: buildroot Date: Tue Nov 21 18:29:10 EST 2017 https://botbot.me/freenode/nixos/2016-08-22/?page=10 "mbrgm: if you build this in a chroot, you can just provide your own implementation of /bin/true, I'd say." But following the discussion there, I am discouraged. The simplest thing is likely to use a minimal Debian chroot to perform the build. Entry: nixos and debian chroot build? Date: Tue Nov 21 18:53:05 EST 2017 So this is quite out there as far as yak shaving goes, but what I need is a way to create buildroot as part of a nix build. If a Debian chroot is part of that, then so be it. But first, check out how to do chrooted builds in nixos. That might actually be the hard part. If it is just about installing some links in fixed locations it can't be that hard. Entry: patch shebangs Date: Tue Nov 21 20:58:47 EST 2017 https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/setup-hooks/patch-shebangs.sh Entry: buildroot Date: Tue Nov 21 21:57:54 EST 2017 giving up pure build. this is madness. use a debian VM or something. Entry: alternative to buildroot? Date: Tue Nov 21 22:43:01 EST 2017 I'm tempted to write a nix version of this. But really I don't need it. I need a way to predictably reproduce a build vm. Anything else is distraction. To capture all build problems, make sure it compiles in a clean environment. This is a "function", so doesn't need services: it is a root filesystem. I have scripts for this. debootstrap.sh kvm-debootstrap.sh apt-get squid-deb-proxy Entry: package a debian chroot Date: Wed Nov 22 00:43:27 EST 2017 And use this as an external dependency in nixos? https://wiki.debian.org/fakechroot fakechroot fakeroot debootstrap --foreign --variant=fakechroot stretch . Didn't work. Neither on nixos nor debian. Entry: mini root fs Date: Thu Nov 23 23:59:46 EST 2017 Maybe should just give it a try. snarf build flags from buildroot. test rfs using qemu. - busybox rfs - dropbear - kernel - bootloader Can be done incrementally. Kernel and bl are an "addition" to the mostly sharable root fs. Mentioned here: https://github.com/NixOS/nixpkgs/issues/4963 - minimal cross-compile - build natively in vm Also, this is built with nix: https://github.com/ambrop72/aprinter Entry: rec defs Date: Fri Nov 24 00:40:28 EST 2017 https://nixos.org/nix/manual/ Example 14.3. Composing GNU Hello (all-packages.nix) rec { ... }; packages can refer to each other this way. structure still needs to not have cycles of course.. Entry: embedded root fs Date: Sat Nov 25 00:49:02 EST 2017 Goal: Build busybox with an arm cross, and execute "pwd" with qemu. See if it is possible to keep nix paths, i.e. have it coexist with normal install. EDIT: This needs a C library also. Entry: standard package layout Date: Sat Nov 25 19:12:56 EST 2017 Make it so that it's possible to install a package from the .nix file $ nix-env -f my.nix -i somepackage Top level expression in my.nix needs to be a set containing derivations. $ nix-build my.nix then builds all the derivations in the set. Entry: C compiler with ulibc Date: Sun Nov 26 09:18:40 EST 2017 I'll need a compiler that is bound to ulibc. Maybe buildroot should still be used to bootstrap gcc+libc, bootloader and kernel. https://www.uclibc.org/toolchains.html How to bootstrap the whole thing? I've documented this somewhere.. Can't find it. Probably deeply burried in some project .txt file somewhere.. So plan, nixify: - cross-binutils - cross-gcc - cross-uclibc Use it to build bootloader + linux. For inspiration: https://github.com/NixOS/nixpkgs/blob/master/doc/old/cross.txt But that's old Here's the new docs: https://nixos.org/nixpkgs/manual/#chap-cross So documentation is incomplete, and it refers to wiki pages that no longer exist. https://nixos.org/wiki/CrossCompiling https://nixos.wiki/wiki/CrossCompiling Too much to learn at once. Try to just build the tools in isolation. EDIT: found a config in all-packages that invokes pkgsCross.buildPackages.gccCrossStageStatic; with crossSystem attribute. This is right at the heart of a lot of hurt! Just compile the Ben Nanote example compiler. My mods caused this error: tom@panda:~/nix-custom$ make nix-build ./standalone.nix --option build-cores 0 error: Unknown ABI: uclibc (use ‘--show-trace’ to show detailed location information) Makefile:4: recipe for target 'all' failed make: *** [all] Error 1 Actually the Ben Nanote one also fails: /nix/store/4sp0sc2wxas7213x59smrwhliqmhc5yk-bash-4.4-p12/bin/bash ./libtool --tag=CXX --mode=link g++ -W -Wall -fvisibility=hidden -g -O2 -module -export-symbols ../../gcc-6.4.0/libcc1/libcc1plugin.sym -Xcompiler '-static-libstdc++' -Xcompiler '-static-libgcc' -o libcc1plugin.la -rpath /nix/store/hw3qn3icxfpck86xawj5ysxdp3ab5v2k-gcc-6.4.0-mips64el-unknown-linux-gnu-stage-static/lib/gcc/mips64el-unknown-linux-gnu/6.4.0/plugin plugin.lo callbacks.lo connection.lo marshall.lo -Wc,../libiberty/pic/libiberty.a libtool: link: g++ -fPIC -DPIC -shared -nostdlib /nix/store/03v97inn212jf1lychvsz2if6f1n4wzb-glibc-2.25-49/lib/crti.o /nix/store/x3wnjni3my5r8d8vpja3qsygvg4bcbgl-gcc-6.4.0/lib/gcc/x86_64-unknown-linux-gnu/6.4.0/crtbeginS.o .libs/plugin.o .libs/callbacks.o .libs/connection.o .libs/marshall.o -Wl,-rpath -Wl,/nix/store/i8sna1nj7cmpyxyzppck0py4dlx6kyj0-gcc-6.4.0-lib/lib/../lib64 -Wl,-rpath -Wl,/nix/store/i8sna1nj7cmpyxyzppck0py4dlx6kyj0-gcc-6.4.0-lib/lib/../lib64 -L/nix/store/abjz8q174c6kq7fr2wa5j2c8r90a5hrj-texinfo-6.5/lib -L/nix/store/b1j623s73qapb2vwrllwwadg9nhk6z8z-gettext-0.19.8/lib -L/nix/store/yz8pw4d7gg2r24m7q2ss3g8mv55vf76b-perl-5.24.3/lib -L/nix/store/zn5pbm4zc0y2vibgsgf8y3nzr51sf6k4-gmp-6.1.2/lib -L/nix/store/vk8vynyabkff4g369b8h356r901d00qb-mpfr-3.1.3/lib -L/nix/store/isd29hsk5c83c3d52lmpbkalm9dln79n-libmpc-1.0.3/lib -L/nix/store/gvfcwvn9myj8kzdcc56dm70hv8zdigrv-libelf-0.8.13/lib -L/nix/store/nzsx07bw4araqqnyylwa636935k9ar67-isl-0.14.1/lib -L/nix/store/16yqxrddgvyycag35kmafhx0y535k65d-zlib-1.2.11-dev/lib -L/nix/store/p0pi16040hgjkwb5931nw8iclvlixcg4-zlib-1.2.11/lib -L/nix/store/03v97inn212jf1lychvsz2if6f1n4wzb-glibc-2.25-49/lib -L/nix/store/i8sna1nj7cmpyxyzppck0py4dlx6kyj0-gcc-6.4.0-lib/lib -L/nix/store/046pkfmbvjnj0z69m6zhhykbnnjhbm9d-gcc-wrapper-6.4.0/bin -L/nix/store/x3wnjni3my5r8d8vpja3qsygvg4bcbgl-gcc-6.4.0/lib/gcc/x86_64-unknown-linux-gnu/6.4.0 -L/nix/store/x3wnjni3my5r8d8vpja3qsygvg4bcbgl-gcc-6.4.0/lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../lib64 -L/nix/store/x3wnjni3my5r8d8vpja3qsygvg4bcbgl-gcc-6.4.0/lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../.. /nix/store/i8sna1nj7cmpyxyzppck0py4dlx6kyj0-gcc-6.4.0-lib/lib/../lib64/libstdc++.so -lm -lc -lgcc_s /nix/store/x3wnjni3my5r8d8vpja3qsygvg4bcbgl-gcc-6.4.0/lib/gcc/x86_64-unknown-linux-gnu/6.4.0/crtendS.o /nix/store/03v97inn212jf1lychvsz2if6f1n4wzb-glibc-2.25-49/lib/crtn.o -static-libstdc++ -static-libgcc ../libiberty/pic/libiberty.a -Wl,-soname -Wl,libcc1plugin.so.0 -Wl,-retain-symbols-file -Wl,../../gcc-6.4.0/libcc1/libcc1plugin.sym -o .libs/libcc1plugin.so.0.0.0 libtool: link: (cd ".libs" && rm -f "libcc1plugin.so.0" && ln -s "libcc1plugin.so.0.0.0" "libcc1plugin.so.0") libtool: link: (cd ".libs" && rm -f "libcc1plugin.so" && ln -s "libcc1plugin.so.0.0.0" "libcc1plugin.so") libtool: link: ar rc .libs/libcc1plugin.a plugin.o callbacks.o connection.o marshall.o libtool: link: ranlib .libs/libcc1plugin.a libtool: link: ( cd ".libs" && rm -f "libcc1plugin.la" && ln -s "../libcc1plugin.la" "libcc1plugin.la" ) make[3]: Leaving directory '/tmp/nix-build-gcc-6.4.0-mips64el-unknown-linux-gnu-stage-static.drv-0/build/libcc1' make[2]: Leaving directory '/tmp/nix-build-gcc-6.4.0-mips64el-unknown-linux-gnu-stage-static.drv-0/build/libcc1' make[1]: Leaving directory '/tmp/nix-build-gcc-6.4.0-mips64el-unknown-linux-gnu-stage-static.drv-0/build' make: *** [Makefile:860: all] Error 2 builder for ‘/nix/store/yzyal2n14323i0wj2pn0kf36ilk1r9ps-gcc-6.4.0-mips64el-unknown-linux-gnu-stage-static.drv’ failed with exit code 2 cannot build derivation ‘/nix/store/d1z833wllg1j9f1hlwvmyj8x6zyc78d1-mips64el-unknown-linux-gnu-gcc-cross-wrapper-6.4.0-mips64el-unknown-linux-gnu-stage-static.drv’: 1 dependencies couldn't be built error: build of ‘/nix/store/d1z833wllg1j9f1hlwvmyj8x6zyc78d1-mips64el-unknown-linux-gnu-gcc-cross-wrapper-6.4.0-mips64el-unknown-linux-gnu-stage-static.drv’ failed Makefile:4: recipe for target 'all' failed make: *** [all] Error 100 tom@panda:~/nix-custom$ Ok, this is too complicated for now. I need a tutorial, or some deep source reading. Grepped for crossSystem, ended up via nixpkgs/doc/cross-compilation.xml at: nixpkgs/lib/systems/examples.nix So the idea is to re-instantiate nixpkgs with crossSystem defined. Entry: Again, cross compilation Date: Sun Nov 26 11:25:47 EST 2017 So the idea is to re-instantiate nixpkgs with crossSystem attribute defined. https://nixos.org/nixpkgs/manual/#chap-cross Trying the raspberry pi example: pi_gcc = (nixpkgsFun { crossSystem = lib.systems.examples.raspberryPi; }).buildPackages.gccCrossStageStatic; tom@panda:~/nix-custom$ ./result-5/bin/armv6l-unknown-linux-gnueabihf-gcc test.c test.c: In function 'main': test.c:2:5: warning: implicit declaration of function 'puts' [-Wimplicit-function-declaration] puts(argv[0]); ^~~~ /nix/store/lis00bns5j1sl6lk84l49bh1clp1f3p7-armv6l-unknown-linux-gnueabihf-binutils-2.28.1/bin/armv6l-unknown-linux-gnueabihf-ld: cannot find crt1.o: No such file or directory /nix/store/lis00bns5j1sl6lk84l49bh1clp1f3p7-armv6l-unknown-linux-gnueabihf-binutils-2.28.1/bin/armv6l-unknown-linux-gnueabihf-ld: cannot find crti.o: No such file or directory /nix/store/lis00bns5j1sl6lk84l49bh1clp1f3p7-armv6l-unknown-linux-gnueabihf-binutils-2.28.1/bin/armv6l-unknown-linux-gnueabihf-ld: cannot find -lssp_nonshared /nix/store/lis00bns5j1sl6lk84l49bh1clp1f3p7-armv6l-unknown-linux-gnueabihf-binutils-2.28.1/bin/armv6l-unknown-linux-gnueabihf-ld: cannot find -lssp /nix/store/lis00bns5j1sl6lk84l49bh1clp1f3p7-armv6l-unknown-linux-gnueabihf-binutils-2.28.1/bin/armv6l-unknown-linux-gnueabihf-ld: cannot find -lc /nix/store/lis00bns5j1sl6lk84l49bh1clp1f3p7-armv6l-unknown-linux-gnueabihf-binutils-2.28.1/bin/armv6l-unknown-linux-gnueabihf-ld: cannot find crtn.o: No such file or directory collect2: error: ld returned 1 exit status It likely needs parameters to point it to the C library. tom@panda:/nix/store$ ls -d *gnueabi* 319y3d9f8s7cmfqhxf20qhk7kqnxkxpb-gcc-6.4.0-armv6l-unknown-linux-gnueabihf-stage-static.drv 68b47ii8r2si70wdfa500qx55xglsyav-gcc-6.4.0-armv6l-unknown-linux-gnueabihf-stage-static-lib gjmkpp20xhjhr2gfcc8ivpx7z2ikq968-armv6l-unknown-linux-gnueabihf-gcc-cross-wrapper-6.4.0-armv6l-unknown-linux-gnueabihf-stage-static l1nwbf0zbci65zx2lxclj64cvfw9wf5v-armv6l-unknown-linux-gnueabihf-binutils-2.28.1.drv lis00bns5j1sl6lk84l49bh1clp1f3p7-armv6l-unknown-linux-gnueabihf-binutils-2.28.1 p4c2x3xa36isv3sdq6bbr1wbyizhx4r4-gcc-6.4.0-armv6l-unknown-linux-gnueabihf-stage-static xi0wgfsxgh20522z4qgd1j04zf19al57-armv6l-unknown-linux-gnueabihf-gcc-cross-wrapper-6.4.0-armv6l-unknown-linux-gnueabihf-stage-static.drv tom@panda:/nix/store$ But the compiler isn't needed directly if the package is defined: # Surpringly, this just works and is in the cache. # Look at what the closure looks like pi_busybox = (nixpkgsFun { crossSystem = lib.systems.examples.raspberryPi; }).busybox; Anything else I try wont work, e.g. pi_erlang = (nixpkgsFun { crossSystem = lib.systems.examples.raspberryPi; }).screen; Fails with: configure: error: in `/tmp/nix-build-flex-2.6.4-armv6l-unknown-linux-gnueabihf.drv-0/flex-2.6.4': configure: error: no acceptable C compiler found in $PATH See `config.log' for more details builder for ‘/nix/store/sdmhrdkgzl05fan8hy2zfpq9llhqdr9r-flex-2.6.4-armv6l-unknown-linux-gnueabihf.drv’ failed with exit code 1 cannot build derivation ‘/nix/store/m9xhh6qvfb0sq15yvmr12mcj2sdb0kbr-linux-pam-1.2.1-armv6l-unknown-linux-gnueabihf.drv’: 1 dependencies couldn't be built building path(s) ‘/nix/store/6y5b6sa5x6jfxclgycshx7qjzgzyn2n5-ncurses-6.0-20170902-armv6l-unknown-linux-gnueabihf’, ‘/nix/store/fx8cwn3dp4lck89i73p3srdyswr7vc7n-ncurses-6.0-20170902-armv6l-unknown-linux-gnueabihf-man’, ‘/nix/store/i8f8bnm4696z4n037m0jiz9j9fn899fa-ncurses-6.0-20170902-armv6l-unknown-linux-gnueabihf-dev’ killing process 10594 cannot build derivation ‘/nix/store/gx4hj6npgvlyx9mkzb8fw7blsxz2q174-screen-4.6.2-armv6l-unknown-linux-gnueabihf.drv’: 1 dependencies couldn't be built error: build of ‘/nix/store/gx4hj6npgvlyx9mkzb8fw7blsxz2q174-screen-4.6.2-armv6l-unknown-linux-gnueabihf.drv’ failed So it seems there is a lot of trouble getting this to compile the standard system. It might be easier to compile it on-target using an emulator, then to strip a closure. In fact, let's try that idea in isolation: find the closure of a single program and run it in a chroot. I have to give this up. Entry: Copying closures manually without nix on the target Date: Sun Nov 26 12:28:58 EST 2017 $ cp -auxv `nix-store -qR /nix/store/4sp0sc2wxas7213x59smrwhliqmhc5yk-bash-4.4-p12` . after setting up a proper link to /bin/bash, this works in a chroot $ nix-env -q dstat-0.7.3 --out-path or just $ nix-build So it's possible to deploy to e.g. lxc once it builds on the dev machine. Still, probably best to stick with standard nix installs. E.g: tom@panda:~/nix-custom$ nix-store -qR /nix/store/b953ln3rfgc822dvqbp76iq13irzawxq-busybox-1.27.2-armv6l-unknown-linux-gnueabihf /nix/store/03v97inn212jf1lychvsz2if6f1n4wzb-glibc-2.25-49 /nix/store/18s27waif2l883szk6hal81wfbj6nf6a-linux-headers-4.4.10-armv6l-unknown-linux-gnueabihf /nix/store/232zf2gjh4dlcaih5chb3xnggijabx1n-linux-headers-4.4.10 /nix/store/i8sna1nj7cmpyxyzppck0py4dlx6kyj0-gcc-6.4.0-lib /nix/store/68b47ii8r2si70wdfa500qx55xglsyav-gcc-6.4.0-armv6l-unknown-linux-gnueabihf-stage-static-lib /nix/store/zn5pbm4zc0y2vibgsgf8y3nzr51sf6k4-gmp-6.1.2 /nix/store/vk8vynyabkff4g369b8h356r901d00qb-mpfr-3.1.3 /nix/store/isd29hsk5c83c3d52lmpbkalm9dln79n-libmpc-1.0.3 /nix/store/3h4w8xjjr5dp20gvvmr0mbyhwyd5sa53-glibc-2.25-49-bin /nix/store/j5cp2zw7i1m4f9526js8734i4nl1h478-glibc-2.25-49-dev /nix/store/p0pi16040hgjkwb5931nw8iclvlixcg4-zlib-1.2.11 /nix/store/x3wnjni3my5r8d8vpja3qsygvg4bcbgl-gcc-6.4.0 /nix/store/lis00bns5j1sl6lk84l49bh1clp1f3p7-armv6l-unknown-linux-gnueabihf-binutils-2.28.1 /nix/store/nzsx07bw4araqqnyylwa636935k9ar67-isl-0.14.1 /nix/store/p4c2x3xa36isv3sdq6bbr1wbyizhx4r4-gcc-6.4.0-armv6l-unknown-linux-gnueabihf-stage-static /nix/store/28hjm5wz53f50nfl1wcl03x40k49az0l-glibc-2.25-49-armv6l-unknown-linux-gnueabihf-armv6l-unknown-linux-gnueabihf /nix/store/b953ln3rfgc822dvqbp76iq13irzawxq-busybox-1.27.2-armv6l-unknown-linux-gnueabihf Now this has a bunch of stuff that isn't necessary. Conclusion for now is that it is not ready yet. To use the cross compiler it is probably best to start over, and not use nixpkgs. Also, I need to think about what I want to do. Once the buildroot was running on the pi, I didn't use it. Not sure why. Maybe it was just to get it running? Similarly, I really like the idea of nixos, and would like to combine it with something that can generate a simple closure for an embedded linux. Entry: cross-compiling, conclusion Date: Sun Nov 26 16:27:55 EST 2017 1. The infrastructure is there, but it seems that most packages are not meant to be cross-compiled. 2. Cross-compiled packages have large closures which include the build tools. This looks like something that just hasn't been fixed yet. 3. Performing a non-nix cross-compile might still be doable. However, compiling buildroot on nix is problematic. In general, unless I really need it, it is probably best to stay away from it. Entry: nix-custum as extension of nixpkg Date: Sun Nov 26 16:49:24 EST 2017 Or a channel? Entry: Use case: erlang 18.3 Date: Tue Nov 28 04:20:38 EST 2017 Is no longer in Debian, so create a .nix file to build it. What I want here is multiple environments which contain the dev packages specific to a particular project. EDIT: All I find is nix-shell. This one says to use buildInputs: https://ariya.io/2016/06/isolated-development-environment-using-nix with import {}; stdenv.mkDerivation rec { name = "env"; env = buildEnv { name = name; paths = buildInputs; }; buildInputs = [ python python27Packages.virtualenv python27Packages.pip go_1_4 lua5_3 ]; } "nix-env --switch-profile" sets ~/nix-profile So this is a global operation. For local profiles, stick to nix shell? So I guess using nix-shell is fine. What I want though, is a way to save it so it doesn't get collected. https://www.reddit.com/r/NixOS/comments/34ofjx/how_to_protect_nixshell_packages_from/ mentions nix-instantiate, but i can't get that to work. it's probably easiest to collect them in a separate profile using nix-install. Entry: Separate profile without changing ~/nix-profile Date: Tue Nov 28 21:14:40 EST 2017 On panda, single user install. $ cd ~ $ rm .nix-profile $ ln -s /nix/var/nix/profiles/default .nix-profile On nixos: tom@phoo:~$ ls -al .nix-profile lrwxrwxrwx 1 tom users 42 Nov 10 20:30 .nix-profile -> /nix/var/nix/profiles/per-user/tom/profile Looks like there is a bug in "nix-env --switch-profile" -- they do not get created automatically. https://github.com/NixOS/nix/issues/1396 For panda, I will just copy one. That doesn't work: they are in the nix store! Do they get modified there? Ok on single user they are stored here: tom@panda:/nix/var/nix/profiles$ ls -al lrwxrwxrwx 1 tom tom 15 Nov 28 21:28 default -> default-14-link lrwxrwxrwx 1 tom tom 60 Nov 28 05:04 default-13-link -> /nix/store/8xxvdmc13fvyxvy65k79fj6851aczrk7-user-environment lrwxrwxrwx 1 tom tom 60 Nov 28 21:28 default-14-link -> /nix/store/sd920gfzskbp1kr9xvikakyw55dsagyf-user-environment on multi-user the are in: tom@phoo:/nix/var/nix/profiles/per-user/tom$ ls -al So for now, stay away from multiple profiles. Just use nix-shell. It's cleaner anyway. Entry: some tips Date: Tue Nov 28 21:46:04 EST 2017 http://blog.timsears.com/posts/morenix.html Entry: keeping nix-shell open Date: Wed Nov 29 00:23:33 EST 2017 So it seems it's enough to just keep a shell open for each project, maybe in a screen or emacs session. This allows the garbage collector to be run. Once reinstalls or recompiles become expensive, it is probably best to set up a local substitution cache. Practically though, using a bit more disk space is unlikely to be a problem. Entry: dependency management Date: Wed Nov 29 12:47:47 EST 2017 Here's what I really need: - default setup: git project with some libraries as submodules - custom setup: inject new versions of libraries, without doing a git commit Basically, "override dependency" to some locally installed library, instead of what is in git submodule, or managed through rebar. The real problem is that there are a couple of different package management mechanisms at play, and I really just want one. Entry: buildroot anyway Date: Fri Dec 1 15:20:11 EST 2017 So pretty much I will need to wrap an RPC call into a Debian container. lxc-create can instantiate a VM in about 3 minutes if debs are cached and the container is created on a ram disk. lxc-create from cache takes only a couple of seconds. to wrap this in a nix expression, a tar of the VM should probably be stored in the nix store as produced by another platform. Transferring these archives isn't even much of a big deal if they run in ramdisk on the target host. So basically: 1. Nix machine calls a "debian factory" to create a VM image, and archives it in its store 2. Nix machine pushes this image to run it under a unique name on an lxc host, and gets a root SSH key 3. Nix machine calls into the VM to create a build product 4. Target VM is removed All of this doesn't even need nix. It can be done using just make. It's the idea. Entry: running as root? Date: Fri Dec 1 16:46:17 EST 2017 How to run a build script as root? Entry: nixos module system vs nixpkgs derivations? Date: Sat Dec 2 11:48:38 EST 2017 Renzo‏ @k0001 -- Nixos module system, on the other hand.... That's hard to compose with. I wish Nixos modules were more like Nixpkgs derivations. Entry: list nixpkgs attributes Date: Sat Dec 2 20:49:43 EST 2017 nix-env -qaP emacs installing: nix-env -f "" -i -A haskellPackages.pretty-show Entry: nix store + btrfs snapshots Date: Sun Dec 3 00:58:21 EST 2017 Combine: - lxc debian container creation - btrfs snapshots - nix build system Use nix store to store references to btrfs snapshots stored on a separate host. After running nix-gc, delete all snapshots that are no longer referenced. The main point of insight is that it is only necessary to have one central garbage-collected store. On top of this it is straightforward to add external objects. Entry: debian repo Date: Sun Dec 3 01:08:19 EST 2017 To make installs predictable, a snapshottable debian repository is necessary. https://wiki.debian.org/DebianRepository Entry: Include Debian sysroot in build Date: Sun Dec 10 14:36:17 EST 2017 Last week I ran into a post about a project that did this. Entry: Moving more things to nix Date: Mon Aug 27 14:51:57 EDT 2018 But first: how do I know what version of nixpkgs I'm running? Maybe a better way to ask this is: how to maintain a nixpkgs fork? Entry: How to duplicate nix installs? Date: Mon Aug 27 17:33:14 EDT 2018 I need a reproducible way to "root" all the packages. I've found a 'nix-custom' git repository in my ~/git dir. Maybe that contains the necessary information? Entry: ~/git/nix-fixed Date: Mon Aug 27 17:39:29 EDT 2018 Starting a new project: nix-fixed. Goal is to create a nix-shell that has a more recent ghc. https://github.com/NixOS/nixpkgs-channels For stability and maximum binary package support, it is recommended to maintain custom changes on top of one of the channels, e.g. nixos-18.03 git remote add channels https://github.com/NixOS/nixpkgs-channels.git I'm missing context. This is supposed to be run inside a git repository, but what repository would that be? I'm seriously missing the point. Let's ask some questions. - How do I know which package set I'm running (channels, nixpkgs). - what does the nixpkgs-channels.git look like? what is the difference between that and nixpkgs.git? Entry: nixpkgs-channels vs nixpkgs Date: Mon Aug 27 17:53:46 EDT 2018 These are two versions of the same thing. I'm going to remove nixpkgs and stick to nixpkgs-channels tom@panda:~/git/nixpkgs-channels$ git checkout remotes/origin/nixos-18.03 tom@panda:~/git$ ls nixpkgs-channels/ ls nixpkgs-channels/ COPYING default.nix doc lib maintainers nixos pkgs README.md tom@panda:~/git$ ls nixpkgs ls nixpkgs COPYING default.nix doc lib maintainers nixos pkgs README.md tom@panda:~/git/nixpkgs-channels$ git branch -a * nixos-unstable remotes/origin/HEAD -> origin/nixos-unstable remotes/origin/nixos-13.10 remotes/origin/nixos-14.04 remotes/origin/nixos-14.04-small remotes/origin/nixos-14.12 remotes/origin/nixos-14.12-small remotes/origin/nixos-15.09 remotes/origin/nixos-15.09-small remotes/origin/nixos-16.03 remotes/origin/nixos-16.03-small remotes/origin/nixos-16.03-testing remotes/origin/nixos-16.09 remotes/origin/nixos-16.09-small remotes/origin/nixos-17.03 remotes/origin/nixos-17.03-small remotes/origin/nixos-17.09 remotes/origin/nixos-17.09-small remotes/origin/nixos-18.03 remotes/origin/nixos-18.03-small remotes/origin/nixos-unstable remotes/origin/nixos-unstable-small remotes/origin/nixpkgs-17.09-darwin remotes/origin/nixpkgs-18.03-darwin remotes/origin/nixpkgs-unstable tom@panda:~/git/nixpkgs$ git branch -a * master remotes/origin/0.5-stable remotes/origin/HEAD -> origin/master remotes/origin/add-notice-file remotes/origin/codeowners remotes/origin/cpan-update remotes/origin/darwin-callpackage remotes/origin/darwin-full-purity remotes/origin/darwin-sandbox remotes/origin/fake-xcode remotes/origin/fetchurl-user-agent remotes/origin/freebsd-losser remotes/origin/gcc8 remotes/origin/gnu-kfreebsd remotes/origin/grsec-stdenv remotes/origin/haskell-outputs-bin-etc-lib remotes/origin/haskell-updates remotes/origin/hydra remotes/origin/hydraBump remotes/origin/iso-cloud-init remotes/origin/kmod-lib-modules remotes/origin/kube-update remotes/origin/lib+paths remotes/origin/linkchecker-use-default-requests remotes/origin/logistics remotes/origin/master remotes/origin/meteor remotes/origin/mkMap remotes/origin/nix-upgrade remotes/origin/nixpart remotes/origin/openssh-drop-dsa remotes/origin/openssl-1.1 remotes/origin/pam-ssh-security remotes/origin/python-unstable remotes/origin/release-13.10 remotes/origin/release-14.04 remotes/origin/release-14.12 remotes/origin/release-15.09 remotes/origin/release-16.03 remotes/origin/release-16.09 remotes/origin/release-17.03 remotes/origin/release-17.09 remotes/origin/release-18.03 remotes/origin/reproducibility remotes/origin/revert-44398-blueman-python3 remotes/origin/revert-45041-fix/urbit-update remotes/origin/ryantrinkle/ledgerblue-0.1.17 remotes/origin/ryantrinkle/remove-empty-file remotes/origin/security-updates remotes/origin/staging remotes/origin/staging-18.03 remotes/origin/staging-next remotes/origin/staging.patchShebangs remotes/origin/syscall-tracing remotes/origin/unstable-aarch64 remotes/origin/whois remotes/origin/x86_64-darwin remotes/origin/yegortimoshenko-patch-1 remotes/origin/yegortimoshenko-patch-2 remotes/origin/yegortimoshenko-patch-3 remotes/origin/yegortimoshenko-patch-5 remotes/origin/yegortimoshenko-patch-6 remotes/origin/yegortimoshenko-patch-7 Entry: different nixpkgs Date: Mon Aug 27 18:04:16 EDT 2018 so I have a channel checked out somewhere. How to create a default.nix file that points to that? with import {}; stdenv.mkDerivation { name = "test-ghc"; buildInputs = [ ghc ]; } Nothing more than this, with nixpkgs-custom linking to the proper nixpkgs-channel. with import ./nixpkgs-custom {}; stdenv.mkDerivation { name = "test-ghc"; buildInputs = [ ghc ]; } Entry: nixos multi-user Date: Mon Aug 27 19:32:45 EDT 2018 https://nixos.wiki/wiki/Install_Nix_in_multi-user_mode_on_non-NixOS Entry: nix-shell creates /tmp/env-vars Date: Mon Aug 27 20:11:02 EDT 2018 make[2]: Entering directory '/home/bob/top_src/gw_src/meta' rm -rf build ; mkdir build # to avoid surprises nix-shell --attr env release.nix --run "cabal test --log=/dev/stdout" /nix/store/3ch9lc8by88b2xx16cnyx5lrdl9lfxc2-stdenv/setup: line 756: /tmp/env-vars: Permission denied Why on earth does this use a "global variable"? It makes no sense! https://github.com/NixOS/nix/issues/262 Entry: panda setup Date: Mon Aug 27 21:42:06 EDT 2018 # Get a list of channels. $ nix-channel --list nixpkgs https://nixos.org/channels/nixpkgs-unstable # Add and remove channels (doesn't update) $ nix-channel --remove nixpkgs $ nix-channel --add https://nixos.org/channels/nixos-18.03 nixpkgs # Download current version of channel. $ nix-channel --update # Update all packages. $ nix-env -u '*' So it should now be the same as gwtest? No. Doing the same there. So, to pin everything, create a custom channel? I still don't understand the difference between a direct link to nixpkgs tree in a default.nix file, and a channel. https://www.reddit.com/r/NixOS/comments/4btjnf/fully_setting_up_a_custom_private_nix_repository/ It appears that the is the entry name in the nix-channel list. Entry: Overlays Date: Wed Sep 5 14:46:50 EDT 2018 Goal: capture a canonical build system such that at any time, I can make an incremental change to any of my own experimental devices. I.e. my "exo". Start by understanding nix overlays, and do this for rust first, since I'm going to need rust for sure. https://beyermatthias.de/blog/2017/04/08/use-the-overlay-luke/ Entry: Cross compilers Date: Fri Sep 7 23:29:02 EDT 2018 What is actually necessary to set up a cross environment? https://nixos.wiki/wiki/Cross_Compiling Entry: pkgsCross Date: Sat Sep 15 10:52:52 EDT 2018 The plan is to have this ready for 18.09, so for now use unstable. I'm going to try this in isolation on phoo 10.1.1.161 https://matthewbauer.us/blog/ Let's just give it a try! With tool chains already present, let's see what is needed to get a basic bootable system up. tom@phoo:~$ nix build -f channel:nixos-unstable pkgsCross.raspberryPi.busybox [25 copied (758.7 MiB), 134.8 MiB DL] Does that means it worked? result/bin/busybox was indeed there Now trying erlang. That didn't compile. But it's a start. Entry: Switch more "tools" to nixos. Date: Sun Sep 16 16:29:18 EDT 2018 E.g. use the panda profile on all the other amd64 hosts. Entry: solving some issues Date: Tue Mar 12 06:59:37 EDT 2019 I want to get this into incremental changes mode. Some questions: - how does cabal2nix actually work? - how can I override some packages? - how to include it into exo/deps - how to build offline There seem to be two ways to extend or modify nixpkgs. - Put things in your local .nix file - clone nixpkgs and modify I'm starting from the one I found in hatd: fde2012519999145c541547bce310f327080e997 Merge #44931: firefox: disable VLC plugin Date: Fri Aug 24 09:29:06 2018 +0200 So, how to direct the nix shell to this env? I've set up exo with: deps/nixpkgs clone of hatd nixpkgs nix release.nix bound to ../deps/nixpkgs How do I know which version a haskell package is at? This didn't return anything: tom@panda:~/exo/nix/nixpkgs$ grep -re '*llvm-hs-pretty*' Let's start with exploring https://github.com/NixOS/cabal2nix It gives this example: tom@panda:~/exo/nix$ cabal2nix cabal://mtl { mkDerivation, base, stdenv, transformers }: mkDerivation { pname = "mtl"; version = "2.2.2"; sha256 = "8803f48a8ed33296c3a3272f448198737a287ec31baa901af09e2118c829bef6"; libraryHaskellDepends = [ base transformers ]; homepage = "http://github.com/haskell/mtl"; description = "Monad classes, using functional dependencies"; license = stdenv.lib.licenses.bsd3; } So I should be able to get the llvm-hs-pretty package this way: tom@panda:~/exo/nix$ cabal2nix cabal://llvm-hs-pretty { mkDerivation, array, base, bytestring, directory, filepath , llvm-hs, llvm-hs-pure, mtl, prettyprinter, stdenv, tasty , tasty-golden, tasty-hspec, tasty-hunit, text, transformers }: mkDerivation { pname = "llvm-hs-pretty"; version = "0.5.0.0"; sha256 = "0dca50bf44df9128fe6f4ad0ed09281c1fc4e615ceac390b2197d2f7f8e9259c"; libraryHaskellDepends = [ array base bytestring llvm-hs-pure prettyprinter text ]; testHaskellDepends = [ base directory filepath llvm-hs llvm-hs-pure mtl tasty tasty-golden tasty-hspec tasty-hunit text transformers ]; homepage = "https://github.com/llvm-hs/llvm-hs-pretty"; description = "A pretty printer for LLVM IR"; license = stdenv.lib.licenses.mit; } Can I just build that derivation directly? I'm not sure this is the right question to ask. Why is the above different from just including llvm-hs-pretty in a cabal file? How does it resolve, basically? EDIT: Now that nixpkgs points at the repo mentioned above, the versions do seem to be higher: * llvm-hs-pretty Synopsis: A pretty printer for LLVM IR. Default available version: 0.5.0.0 Installed versions: [ Not installed ] Homepage: https://github.com/llvm-hs/llvm-hs-pretty License: MIT I rememver the other one being at 0.2 I don't understand why. I've added llvm-hs-pretty to exo-hs.cabal again. This is the error: test: tests/input/: getDirectoryContents:openDirStream: does not exist (No such file or directory) I think it is the same as before. It's still at: Building library for llvm-hs-pretty-0.2.0.0.. Is there a way to override running the tests during package build? So question remains: where does this version come from? EDIT: I'm going to have to trace this down. Setting a >= constraint on the version in the .cabal file doesn't have any effect. Where does it get its versions? cabal2nix page links to: https://nixos.org/nixpkgs/manual/#users-guide-to-the-haskell-infrastructure "The Haskell package set is not registered in the top-level namespace because it is *huge*. If all Haskell packages were visible to these commands, then name-based search/install operations would be much slower than they are now. We avoided that by keeping all Haskell-related packages in a separate attribute set called haskellPackages, which the following command will list: " tom@panda:~/exo/hs$ nix-env -f ./nixpkgs -qaP -A haskellPackages | grep llvm-hs haskellPackages.llvm-hs llvm-hs-5.1.3 haskellPackages.llvm-hs_6_0_0 llvm-hs-6.0.0 haskellPackages.llvm-hs-pretty llvm-hs-pretty-0.2.0.0 haskellPackages.llvm-hs-pure llvm-hs-pure-5.1.2 haskellPackages.llvm-hs-pure_6_0_0 llvm-hs-pure-6.0.0 So it appears that it will always take the version from there. Reproducing just the error: tom@panda:~/exo/hs$ nix-env -f ./nixpkgs -iA haskellPackages.llvm-hs-pretty ... running tests Running 1 test suites... Test suite test: RUNNING... test: tests/input/: getDirectoryContents:openDirStream: does not exist (No such file or directory) Test suite test: FAIL Test suite logged to: dist/test/llvm-hs-pretty-0.2.0.0-test.log 0 of 1 test suites (0 of 1 test cases) passed. builder for '/nix/store/mcprcpib8l31qkyjkw4qk84395332yzq-llvm-hs-pretty-0.2.0.0.drv' failed with exit code 1 error: build of '/nix/store/mcprcpib8l31qkyjkw4qk84395332yzq-llvm-hs-pretty-0.2.0.0.drv' failed So I should be able to go into nixpkgs and find that package. Yep, here it is: ~/exo/deps/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix "llvm-hs-pretty" = callPackage ({ mkDerivation, array, base, bytestring, directory, filepath , llvm-hs, llvm-hs-pure, mtl, pretty-show, tasty, tasty-golden , tasty-hspec, tasty-hunit, text, transformers, wl-pprint-text }: mkDerivation { pname = "llvm-hs-pretty"; version = "0.2.0.0"; sha256 = "133kyksbp88q0wavp3wdjg69h9fpwi7nq626nvikdy46cf7lgklh"; libraryHaskellDepends = [ array base bytestring llvm-hs-pure text wl-pprint-text ]; testHaskellDepends = [ base directory filepath llvm-hs llvm-hs-pure mtl pretty-show tasty tasty-golden tasty-hspec tasty-hunit text transformers ]; homepage = "https://github.com/llvm-hs/llvm-hs-pretty"; description = "Pretty printer for LLVM IR"; license = stdenv.lib.licenses.mit; hydraPlatforms = stdenv.lib.platforms.none; }) {}; So a couple of paths here: - find a way to fix the build error - try a different nixpkgs tree to see if it has a different version - override the package locally It is updated in the current listing. version = "0.6.1.0"; So can I just merge nixpkgs? Let's try it. It might brake some things though... Let's at least pick a release version Branch 19-03 also has the same version. Set up git properly - in ~/git/nixpkgs, restore origin to https://github.com/NixOS/nixpkgs.git - fetch the branch into main ~/git tom@panda:~/git/nixpkgs.git$ git fetch origin release-19.03:release-19.03 - then fetch and merge it into exo tom@panda:~/exo/deps/nixpkgs$ git fetch origin release-19.03 tom@panda:~/exo/deps/nixpkgs$ git merge origin/release-19.03 That merge failed big time. So just do: tom@panda:~/exo/deps/nixpkgs$ git reset --hard origin/release-19.03 It fails building: Setup: The program 'llvm-config' version ==7.0.* is required but the version found at /nix/store/xznp9rd6r6lim185s35a5ni11ra3p0ja-llvm-6.0.1/bin/llvm-config is version 6.0.1 builder for '/nix/store/rmyrhg19gifvcwcsg0dhgxqbg93d4xnx-llvm-hs-7.0.1.drv' failed with exit code 1 cannot build derivation '/nix/store/cisvd2zk7dk52pf65bmrgjqdqx3m3963-ghc-8.6.3-with-packages.drv': 1 dependencies couldn't be built error: build of '/nix/store/cisvd2zk7dk52pf65bmrgjqdqx3m3963-ghc-8.6.3-with-packages.drv' failed I'm going to have to learn how to deal with this. It seems that there is no guarantee the versions match properly for all packages. Now, do i fix this in nixpkgs, or in the source package? It makes more sense to keep the environment consistent to my needs, so fix it in nixpkgs. I can either bump llvm to 7, or wind the haskell bindings back to 6. Messing with llvm seems more dangerous. Let's try the haskell side first. Changing in hackage-packages.nix : llvm-hs to "6.3.0" And again a fucking build error. This time it seems due to GHC version 8.6 Trying some more changes and more constraints pop up. This is not the right avenue. Going to revert back, saved it in: tom@panda:~/exo/deps/nixpkgs$ git-log |head 17640cb67fa338ada104c5f48200ef3cd4a60bb8 attempt_llvm_hs_fixes The other route is to upgrade llvm. There are build scripts for version 7. How to make sure it finds the right version? override "llvm-config" with version 7. First switch main env to 19.03 tom@panda:~/exo/hs$ nix-channel --add https://nixos.org/channels/nixos-19.03 nixpkgs Entry: cabal2nix, remarks Date: Tue Mar 12 09:08:34 EDT 2019 - it seems to work well in general, but expect some inconsistencies in nixpkgs for obscure things. (e.g. llvm-hs didn't have correc tllvm version) it seems best to just standardize on a global nixpkgs, and fix haskell module version conflicts in nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix - set up some kind of cache for local nixpkgs builds. Entry: how to point llvm-hs to llvm 7.0.1? Date: Tue Mar 12 09:33:48 EDT 2019 tom@panda:~$ nix-env -i llvm installing 'llvm-6.0.1' But 7.0.1 is not there. tom@panda:~/exo/deps/nixpkgs/pkgs$ nix-channel --update nixpkgs unpacking channels... created 5 symlinks in user environment Now this works: tom@panda:~$ nix-env -i llvm-7.0.1 In the derivation for llvm-hs, it just mentions llvm-config. Can I make this llvm-config-7.0.1? Basically, where is "llvm" or "llvm-config" defined? BTW did I say this is really neat? Requires some time investment though.. Global names are here: nixpkgs/pkgs/top-level/all-packages.nix but I find only "llvm" Grepping.. tom@panda:~/exo/hs/nixpkgs$ grep -re ' llvm-config =' * pkgs/development/haskell-modules/configuration-nix.nix: let base = super.llvm-general.override { llvm-config = pkgs.llvm_35; }; pkgs/development/haskell-modules/configuration-nix.nix: llvm-config = pkgs.llvm_6; Looks like this is just a bug? This is set to 6: nixpkgs/pkgs/development/haskell-modules/configuration-nix.nix llvm-hs = let dontCheckDarwin = if pkgs.stdenv.isDarwin then dontCheck else pkgs.lib.id; in dontCheckDarwin (super.llvm-hs.override { llvm-config = pkgs.llvm_6; }); setting it to 7 ok that did it Entry: get to know nix-shell better Date: Sat Mar 16 16:20:14 EDT 2019 Basically, I want an environment that has 'fakeroot' available. How to do that with nix-shell? It looks like with 19.03 nix-shell instances are independent. At least I see different directories in /tmp https://nixos.org/nixos/nix-pills/developing-with-nix-shell.html I don't get it from a quick browse. Maybe set up to build a package first? Entry: broke phoo Date: Sat Mar 16 16:54:28 EDT 2019 after upgrading 18.03 -> 19.03 tom@phoo:~$ nix-env --upgrade error: Nix database directory ‘/nix/var/nix/db’ is not writable: Permission denied it seems like this is not going to be a quick thing... There are still too many rough edges and churn. EDIT: This fixes the error: export NIX_REMOTE=daemon Entry: nixos lxc Date: Sat Mar 16 18:31:00 EDT 2019 https://us.images.linuxcontainers.org/ current 19.03 is broken. revert from backup. The problem existed before. Reverted. Trying 18.09 now. On 18.03 nix-daemon was running. Trying reboot into 18.09 Not running... maybe boot is not complete? Try to start it with console attached. It seems fine. Says 18.03 on login though.. Some more issues https://github.com/NixOS/nix/issues/2649 I'm going to revert it again. I wonder if I really need to update the system. Just updating nixpkgs per user is fine it appears. What I really need is a clean place to test dependencies. Entry: nix-daemon problmes after update Date: Mon Mar 18 10:21:52 EDT 2019 tom@panda:~$ nix-env --upgrade error: cloning builder process: Operation not permitted error: unable to start build process tom@panda:~$ export NIX_REMOTE=daemon tom@panda:~$ nix-env --upgrade warning: the group 'nixbld' specified in 'build-users-group' does not exist error: the group 'nixbld' specified in 'build-users-group' does not exist tom@panda:~$ export NIX_REMOTE=daemon tom@panda:~$ nix-env --upgrade warning: the group 'nixbld' specified in 'build-users-group' does not exist error: the group 'nixbld' specified in 'build-users-group' does not exist On panda, store is single user (tom). So daemon is not needed. Why do I get the Operation not permitted? Entry: Problems with nix 2.2 on ubuntu / debian Date: Mon Mar 18 10:27:16 EDT 2019 tom@panda:~$ nix --version nix (Nix) 2.2 2.1.3 is last working version. TO FIX: tom@panda:~$ cat /etc/nix/nix.conf sandbox = false Entry: Next? Date: Tue Mar 19 06:58:12 EDT 2019 I really want to push this further, but keep stumbling on the one thing I care about: building a decent cross compiling setup for a target architecture that is not necessarily nix. Basically I want buildroot + rust, with tools managed in nix. I'll settle for just build tools, and a way to compile bootloader, kernel, C library, busybox and erlang. Entry: lorri Date: Thu Mar 28 21:12:36 EDT 2019 https://www.tweag.io/posts/2019-03-28-introducing-lorri.html Entry: Need Nix > 2.0 Date: Wed Oct 16 20:04:28 EDT 2019 On gwtest, and I currently have: [master] build@gwtest-kz:~/cache/top_src/gw_src/meta$ nix-env --version nix-env (Nix) 1.11.16 And this is a multi-user setup. Entry: How to install nix multi-user? Date: Thu Oct 17 06:33:24 EDT 2019 I completely forgot, and it doesn't look I have notes. Nix in multi-user mode on a non-NixOS (eg. Debian) system https://gist.github.com/joepie91/043a51a7b70be5f50f1d ./configure --enable-gc --prefix=/usr/local make make install I had to "apt-get instal g++" to allow for -std=c++17 compared to the previous install. I hope this doesn't mess things up. I went back to 2.0.4 Entry: debnix Date: Thu Oct 17 07:08:43 EDT 2019 A minimal debian lxc host that can host most recent nix in multi user mode. Basically, I don't want to use a nixos build vm. Or maybe I should? Entry: debugging Date: Thu Oct 17 08:33:00 EDT 2019 I guess the only way to figure out how this works is to start tracing down error messages... [master] build@gwtest:~/cache/top_src/gw_src/deps/asm_tools/asm-tools$ make nix-build release.nix these derivations will be built: /nix/store/0aw4zvy9174w04wcwpcnnxhhlvv84ad1-asm-tools-1.0.0.drv building '/nix/store/0aw4zvy9174w04wcwpcnnxhhlvv84ad1-asm-tools-1.0.0.drv'... /nix/store/63karsgdg7fm3q0if4zfd7apbd8ac1ci-stdenv-linux/setup: line 804: /build/env-vars: No such file or directory setupCompilerEnvironmentPhase /nix/store/63karsgdg7fm3q0if4zfd7apbd8ac1ci-stdenv-linux/setup: line 804: /build/env-vars: No such file or directory Build with /nix/store/68pd1fg0xhjv642klsz9xpbsckvj5dxj-ghc-8.6.3. mkdir: cannot create directory ‘/build’: Permission denied builder for '/nix/store/0aw4zvy9174w04wcwpcnnxhhlvv84ad1-asm-tools-1.0.0.drv' failed with exit code 1 error: build of '/nix/store/0aw4zvy9174w04wcwpcnnxhhlvv84ad1-asm-tools-1.0.0.drv' failed Makefile:14: recipe for target 'release' failed make: *** [release] Error 100 That line contains: export > "$NIX_BUILD_TOP/env-vars" || true Where is this variable set? https://discourse.nixos.org/t/custom-installation-of-nix-unset-nix-build-top/4285 So it looks like there is an issue with the install. It defaults to /build and can be set in nix.conf Simple workaround seems to be to create /build with proper perms https://discourse.nixos.org/t/custom-installation-of-nix-unset-nix-build-top/4285 ok it moves now Entry: buildroot and nix Date: Tue Oct 22 17:00:02 EDT 2019 I really want to make this work. The main issue seems to be that buildroot assumes LSB, and nix violates that. https://nixos.wiki/wiki/Nix_vs._Linux_Standard_Base Is there a way to introduce a compatibility layer? I think apenwarr made an effort for redo: https://github.com/NixOS/nix/issues/1904 asking on irc: 17:09 < doelie> hi. anyone aware of current efforts to make buildroot work on nix/nixos? 17:13 < mdash> doelie: i haven't heard of any, clever did this though https://github.com/cleverca22/not-os 17:14 -!- Denommus [~user@unaffiliated/denommus] has quit [Remote host closed the connection] 17:14 < doelie> mdash: neat. thanks for the link! 17:14 < mdash> looks like this buildroot-using thing calls buildFHSUserEnv https://github.com/swift-nav/piksi_buildroot/blob/v2.3.0-release/default.nix 17:18 < doelie> mdash: buildFHSUserEnv is probably the missing element i wasn't aware of. thanks again. https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard https://nixos.org/nixpkgs/manual/#sec-fhs-environments https://github.com/swift-nav/piksi_buildroot/blob/v2.3.0-release/default.nix https://github.com/cleverca22/not-os EDIT: Added exo/deps/nixbr containing that default.nix Entry: Initial Date: Tue Oct 22 17:29:56 EDT 2019 From mdash at #nixos https://nixos.org/nixpkgs/manual/#sec-fhs-environments https://github.com/swift-nav/piksi_buildroot/blob/v2.3.0-release/default.nix To make this work I probably need a clean nixos machine. Do I have one that still works? tom@phoo:~$ nix --version nix (Nix) 2.0.1 copied default.nix from above tom@phoo:~/git/nixbr$ nix-shell ... ** Message: Requires Linux version >= 3.19 built with CONFIG_USER_NS ** Message: Run: sudo sysctl -w kernel.unprivileged_userns_clone=1 ** (process:3605): ERROR **: main: unshare: Operation not permitted Trace/breakpoint trap tom@phoo:~/git/nixbr$ https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923747 > > This is resolved by adding CONFIG_USER_NS=y to the kernel configuration. That option is enabled, but the feature is restricted by a sysctl (kernel.unprivileged_userns_clone) that is off by default. (The reason for this is discussed at .) Ben. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=898446 root@zoo:~# sysctl kernel.unprivileged_userns_clone kernel.unprivileged_userns_clone = 0 root@zoo:~# sysctl kernel.unprivileged_userns_clone=1 kernel.unprivileged_userns_clone = 1 tom@phoo:~/git/nixbr$ nix-shell (piksi-nix)tom@phoo:~/git/nixbr$ Entry: Host it in exo first Date: Sat Oct 26 09:12:07 EDT 2019 Use the exo defconfig first. EDIT: Looks like this doesn't work properly: tom@phoo:~/exo/br$ nix-shell mkdir: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference stat: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference id: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference mkdir: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference stat: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference id: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference mkdir: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory Both phoo and panda fail here: config.status: executing default commands make[1]: Leaving directory '/home/tom/exo/br/target/i586/build/host-gcc-initial-6.4.0/build' make: *** [package/pkg-generic.mk:250: /home/tom/exo/br/target/i586/build/host-gcc-initial-6.4.0/.stamp_built] Error 2 make: Leaving directory '/home/tom/exo/deps/buildroot' The lesson I take from this is that taking nixpkg snapshots is not a simple task. Things break often, and with a complex dev setup there are just a whole lot of constraints to satisfy. Maybe simplify first? Don't build a compiler. That is a big step in the build process anyway. No the reason is probably that the compiler is not set up correctly. Solve that first. Entry: Something very basic is wrong Date: Sat Oct 26 11:34:48 EDT 2019 tom@phoo:~/exo/br$ nix-shell mkdir: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference stat: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference id: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference mkdir: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference stat: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference id: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference mkdir: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory Maybe it needs a pure shell? Because: (piksi-nix)tom@phoo:~/exo/br$ git git: relocation error: /lib/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference (piksi-nix)tom@phoo:~/exo/br$ which git /home/tom/.nix-profile/bin/git I think this is because my .bashrc puts the path there. EDOT: Still fails on the same error. mkdir -p -- .deps make[1]: Leaving directory '/home/tom/exo/br/target/i586/build/host-gcc-initial-6.4.0/build' make: *** [package/pkg-generic.mk:250: /home/tom/exo/br/target/i586/build/host-gcc-initial-6.4.0/.stamp_built] Error 2 make: Leaving directory '/home/tom/exo/deps/buildroot' piksi-env-chrootenv:tom@phoo:~/exo/br$ I don't really get a good idea what the problem is, but the guess is that I'm trying to use clang to compile gcc? No piksi-env-chrootenv:tom@phoo:~/exo/br$ gcc --version gcc (GCC) 7.4.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Can I just switch to gcc 6.4.0 ? gcc -> gcc6 Then I get a bunch of these: collision between `/nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib64/locale/locale-archive' and `/nix/store/xvj2l5llpmcgx99cl37v5xqpazs38195-glibc-multi-2.27/lib64/locale/locale-archive' Entry: Empty .bashrc in case of [ ! -z "$IN_NIX_SHELL" ] Date: Sat Oct 26 11:46:14 EDT 2019 I think this might have messed up a whole lot of things in the past. Entry: So how to manage this? Date: Sat Oct 26 12:36:34 EDT 2019 Rebuilds are going to be often, so use the following approach: - During the day, work in the nix-shell environment Entry: What is the actual error? Date: Sat Oct 26 12:39:07 EDT 2019 ok I see. this is doing a parallel build. that's why the errors are messed up. Th actual error is this: /usr/bin/g++ -I../../libcpp -I. -I../../libcpp/../include -I../../libcpp/include -O2 -I/home/tom/exo/br/target/i586/host/include -W -Wall -Wno-narrowing -Wwrite-strings -Wmissing-format-attribute -pedantic -Wno-long-long -fno-exceptions -fno-rtti -I../../libcpp -I. -I../../libcpp/../include -I../../libcpp/include -c -o mkdeps.o -MT mkdeps.o -MMD -MP -MF .deps/mkdeps.Tpo ../../libcpp/mkdeps.c ../../libcpp/macro.c: In function 'bool create_iso_definition(cpp_reader*, cpp_macro*)': ../../libcpp/macro.c:3093:58: error: format not a string literal and no format arguments [-Werror=format-security] cpp_error (pfile, CPP_DL_ERROR, paste_op_error_msg); ^ ../../libcpp/macro.c:3106:58: error: format not a string literal and no format arguments [-Werror=format-security] cpp_error (pfile, CPP_DL_ERROR, paste_op_error_msg); ^ Still an issue with gcc 6.5.0 On debian I use 6.3 Maybe try with 5? (5.5.0) Looks like that is working. The fix should be done in libcpp or in buildroot. Probably this is already fixed and I have an old image. Anyway it seems we are going somewhere now. It might still be nix: https://unix.stackexchange.com/questions/356232/disabling-the-security-hardening-options-for-a-nix-shell-environment?rq=1 Nix compiler wrapper has an option to disable format-security. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80895 That is a bug in GCC: GCC couldn't be compiled with `-Werror=format-security`, now it can. The motivation is here (although that post is somewhat outdated): So this is actually mentioned in default.nix It is definitely present in the shell: piksi-env-chrootenv:tom@phoo:~/exo/br$ set |grep hard hardeningDisable=all maybe this is only in a more recent one? i guess it is easy enough to test. Many people mention it, but I don't see this variable defined anywhere. /nix/store/wnr08dppprnf7n1vwyxqag3fd64l99s6-gcc-wrapper-8.3.0/nix-support/add-hardening.sh piksi-env-chrootenv:tom@phoo:~/exo/br$ NIX_DEBUG=1 gcc test.c HARDENING: disabled flags: pie HARDENING: Is active (not completely disabled with "all" flag) HARDENING: enabling fortify HARDENING: enabling stackprotector HARDENING: enabling strictoverflow HARDENING: enabling format HARDENING: enabling pic extra flags before to /nix/store/ihdxv6777cd071l9mnzkp6x1didax0qs-gcc-8.3.0/bin/gcc: -O2 -D_FORTIFY_SOURCE=2 -fstack-protector-strong --param ssp-buffer-size=4 -fno-strict-overflow -Wformat -Wformat-security -Werror=format-security -fPIC -Wl\,-dynamic-linker -Wl\,/nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/ld-linux-x86-64.so.2 original flags to /nix/store/ihdxv6777cd071l9mnzkp6x1didax0qs-gcc-8.3.0/bin/gcc: test.c extra flags after to /nix/store/ihdxv6777cd071l9mnzkp6x1didax0qs-gcc-8.3.0/bin/gcc: -B/nix/store/jrp13k1sl13jl8yqg5f69vikaivbjvm3-gcc-8.3.0-lib/lib -B/nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/ -idirafter /nix/store/sr4253np2gz2bpha4gn8gqlmiw604155-glibc-2.27-dev/include -idirafter /nix/store/ihdxv6777cd071l9mnzkp6x1didax0qs-gcc-8.3.0/lib/gcc/x86_64-unknown-linux-gnu/8.3.0/include-fixed -B/nix/store/wnr08dppprnf7n1vwyxqag3fd64l99s6-gcc-wrapper-8.3.0/bin/ -idirafter /usr/include -idirafter /usr/include -L/usr/lib -L/usr/lib32 -L/usr/lib -L/usr/lib32 -L/nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib -L/nix/store/jrp13k1sl13jl8yqg5f69vikaivbjvm3-gcc-8.3.0-lib/lib test.c: In function 'main': test.c:4:2: error: format not a string literal and no format arguments [-Werror=format-security] printf(argv[1]); ^~~~~~ cc1: some warnings being treated as errors So why is not not responding to hardeningDisable=all ? Some magic is happening here. I want to see exactly where this hardeningDisable variable is used. Ok added it to default.nix This moves it along. It breaks again: make[1]: Entering directory '/home/tom/exo/br/target/i586/build/host-ncurses-6.0/progs' make[1]: *** No rule to make target '../include/curses.h', needed by '../obj_s/tic.o'. Stop. make[1]: Leaving directory '/home/tom/exo/br/target/i586/build/host-ncurses-6.0/progs' make: *** [package/pkg-generic.mk:250: /home/tom/exo/br/target/i586/build/host-ncurses-6.0/.stamp_built] Error 2 make: Leaving directory '/home/tom/exo/deps/buildroot' That directory does have curses.h.in so it looks like something went wrong with the configuration checking for prefix... /home/tom/exo/br/target/i586/host ... checking where we will install curses.h... ${prefix}/include/ncurses Yeah what the fuck ever. menuconfig also doesn't work. maybe host curses setup is not ok. Maybe it needs ncurses 6? I'm trying the bbb variant. Maybe better luck there. Same problem. Maybe start with "make menuconfig" not working? EDIT: Revisit some basics. Entry: nixos generators Date: Sat Oct 26 14:07:33 EDT 2019 https://discourse.nixos.org/t/running-nixos-18-09-in-an-lxd-3-8-container/1804 https://github.com/nix-community/nixos-generators You can now create LXC images from your NixOS config with nixos-generators 16! It might still have some problems, so please report them if you encounter any. lxc image import $(nixos-generate -f lxc-metadata) $(nixos-generate -c configuration.nix -f lxc) So this needs to be internally bootstrapped. I don't really see a need to do anything other than cloning what I have currently running. Entry: do i really want this? Date: Sun Oct 27 08:03:17 EDT 2019 yes. i guess i need to put in the time then. the problem should be simple: a library is not found. ok now menuconfig builds but the other error is still there: make[1]: Entering directory '/home/tom/exo/br/target/i586/build/host-ncurses-6.0/progs' make[1]: *** No rule to make target '../include/curses.h', needed by '../obj_s/tic.o'. Stop. make[1]: Leaving directory '/home/tom/exo/br/target/i586/build/host-ncurses-6.0/progs' make: *** [package/pkg-generic.mk:250: /home/tom/exo/br/target/i586/build/host-ncurses-6.0/.stamp_built] Error 2 make: Leaving directory '/home/tom/exo/deps/buildroot' I think it should be creating that file but somehow it doesn't. Why is it trying to build host ncurses actually? Maybe removing ncurses entirely will fix the problem? No, same problem. I have no other option than to dig into the ncurses build to find out what exactly is going wrong here. Definte the problem: curses.h.in is not turned into curses.h At least, that is what the code seems to expect. I would expect that during configuration, this file is generated. Probably at the end of configure.in AC_OUTPUT( \ include/MKterm.h.awk \ include/curses.head:include/curses.h.in \ ... So waht is that include/curses.head about? First, what does that ":" syntax mean? I think that it meands to generated curses.head instead of default curses.h If I create a link it should work then. So why is this file named curses.head? config.status: creating include/curses.head @ test "$(include_dir)" = "$(prefix)/include" || \ echo '** Include-directory is not in a standard location' so is there a curses.h file anywhere? no So wtf. progs/Makefile.in refers to ../include/curses.h as deps do that file should be there it should be generated somewhere! This mirror has the same config.head line: https://github.com/mirror/ncurses/blob/master/configure.in Why is config.head not referenced anywhere else? The first command that is run is: PATH=/home/tom/exo/br/target/i586/host/bin:/home/tom/exo/br/target/i586/host/sbin:/home/tom/exo/br/scripts/wrappers/bin:/home/tom/exo/br/buildroot/internal/host_output/host/bin:/home/tom/exo/br/buildroot/output/internal/host/bin:/run/wrappers/bin:/usr/bin:/usr/sbin PKG_CONFIG=/home/tom/exo/br/target/i586/host/bin/pkg-config PKG_CONFIG_SYSROOT_DIR=/ PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR=/home/tom/exo/br/target/i586/host/lib/pkgconfig:/home/tom/exo/br/target/i586/host/share/pkgconfig /usr/bin/make -j1 -C /home/tom/exo/br/target/i586/build/host-ncurses-6.0 sources So look at that build target. No explicit target in top Makefile So if I do this, then the curses.h gets generated (see below). But the command above for some reason is not doing anything. It has that "exo +" thing prefixed to it. That's it. So why is buildroot inserting that "echo +" ? If I run that line manually it builds. tom@panda:/i/phoo/home/tom/exo/br/target/i586/build/host-ncurses-6.0$ grep -re "Include-directory" * Makefile: echo '** Include-directory is not in a standard location' Makefile.in: echo '** Include-directory is not in a standard location' tom@panda:/i/phoo/home/tom/exo/br/target/i586/build/host-ncurses-6.0$ ls sources ls: cannot access 'sources': No such file or directory tom@panda:/i/phoo/home/tom/exo/br/target/i586/build/host-ncurses-6.0$ make sources cd include && make - DESTDIR="" RPATH_LIST="/home/tom/exo/br/target/i586/host/lib" sources make[1]: Entering directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/include' cat curses.head >curses.h AWK=gawk /bin/sh ./MKkey_defs.sh ./Caps >>curses.h /bin/sh -c 'if test "chtype" = "cchar_t" ; then cat ./curses.wide >>curses.h ; fi' cat ./curses.tail >>curses.h /bin/sh ./MKhashsize.sh ./Caps >hashsize.h AWK=gawk /bin/sh ./MKncurses_def.sh ./ncurses_defs >ncurses_def.h AWK=gawk /bin/sh ./MKparametrized.sh ./Caps >parametrized.h touch config.h gawk -f MKterm.h.awk ./Caps > term.h /bin/sh ./edit_cfg.sh ../include/ncurses_cfg.h term.h ** edit: HAVE_TCGETATTR 1 ** edit: HAVE_TERMIOS_H 1 ** edit: HAVE_TERMIO_H 1 ** edit: BROKEN_LINKER 0 make[1]: Leaving directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/include' cd ncurses && make - DESTDIR="" RPATH_LIST="/home/tom/exo/br/target/i586/host/lib" sources make[1]: Entering directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/ncurses' gawk -f ./tinfo/MKcodes.awk bigstrings=1 ./../include/Caps >codes.c /usr/bin/gcc -o make_hash -DHAVE_CONFIG_H -DUSE_BUILD_CC -I../ncurses -I. -I../include -I./../include -DHAVE_CONFIG_H -I. -I../include -I/home/tom/exo/br/target/i586/host/include -D_GNU_SOURCE -DNDEBUG -O2 -I/home/tom/exo/br/target/i586/host/include --param max-inline-insns-single=1200 ./tinfo/make_hash.c -L/home/tom/exo/br/target/i586/host/lib -Wl,-rpath,/home/tom/exo/br/target/i586/host/lib /bin/sh -e ./tinfo/MKcaptab.sh gawk 1 ./tinfo/MKcaptab.awk ./../include/Caps > comp_captab.c /bin/sh -e ./tty/MKexpanded.sh "/usr/bin/cpp" -DHAVE_CONFIG_H -I. -I../include -I/home/tom/exo/br/target/i586/host/include -D_GNU_SOURCE -DNDEBUG > expanded.c /bin/sh -e ./tinfo/MKfallback.sh /home/tom/exo/br/target/i586/host/share/terminfo ../misc/terminfo.src /usr/bin/tic >fallback.c /bin/sh -e ./base/MKlib_gen.sh "/usr/bin/cpp -DHAVE_CONFIG_H -I. -I../include -I/home/tom/exo/br/target/i586/host/include -D_GNU_SOURCE -DNDEBUG" "gawk" generated <../include/curses.h >lib_gen.c .. adding -P option to work around /usr/bin/cpp 6.3.0 AWK=gawk /bin/sh ./tinfo/MKkeys_list.sh ../include/Caps | sort >keys.list gawk -f ./base/MKkeyname.awk bigstrings=1 keys.list > lib_keyname.c /bin/sh -e ./base/MKlib_gen.sh "/usr/bin/cpp -DHAVE_CONFIG_H -I. -I../include -I/home/tom/exo/br/target/i586/host/include -D_GNU_SOURCE -DNDEBUG" "gawk" implemented <../include/curses.h >link_test.c .. adding -P option to work around /usr/bin/cpp 6.3.0 gawk -f ./tinfo/MKnames.awk bigstrings=1 ./../include/Caps >names.c echo | gawk -f ./base/MKunctrl.awk bigstrings=1 >unctrl.c /usr/bin/gcc -o make_keys -DHAVE_CONFIG_H -DUSE_BUILD_CC -I../ncurses -I. -I../include -I./../include -DHAVE_CONFIG_H -I. -I../include -I/home/tom/exo/br/target/i586/host/include -D_GNU_SOURCE -DNDEBUG -O2 -I/home/tom/exo/br/target/i586/host/include --param max-inline-insns-single=1200 ./tinfo/make_keys.c -L/home/tom/exo/br/target/i586/host/lib -Wl,-rpath,/home/tom/exo/br/target/i586/host/lib ./make_keys keys.list > init_keytry.h make[1]: Leaving directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/ncurses' cd progs && make - DESTDIR="" RPATH_LIST="/home/tom/exo/br/target/i586/host/lib" sources make[1]: Entering directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/progs' /bin/sh ./MKtermsort.sh gawk ./../include/Caps >termsort.c echo "#ifndef __TRANSFORM_H" >transform.h echo "#define __TRANSFORM_H 1" >>transform.h echo "#include " >>transform.h echo "extern bool same_program(const char *, const char *);" >>transform.h /bin/sh -c 'if test -n "" ; then echo "#define SUFFIX_IGNORED \"\"">>transform.h; fi' echo "#define PROG_CAPTOINFO \"`echo captoinfo| sed 's/$//'|sed 's,x,x,'|sed 's/$//'`\"" >>transform.h echo "#define PROG_INFOTOCAP \"`echo infotocap| sed 's/$//'|sed 's,x,x,'|sed 's/$//'`\"" >>transform.h echo "#define PROG_RESET \"`echo reset| sed 's/$//'|sed 's,x,x,'|sed 's/$//'`\"" >>transform.h echo "#define PROG_INIT \"`echo init| sed 's/$//'|sed 's,x,x,'|sed 's/$//'`\"" >>transform.h echo "#endif /* __TRANSFORM_H */" >>transform.h make[1]: Leaving directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/progs' cd panel && make - DESTDIR="" RPATH_LIST="/home/tom/exo/br/target/i586/host/lib" sources make[1]: Entering directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/panel' rm -f ../include/panel.h cp ./panel.h ../include/panel.h make[1]: Leaving directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/panel' cd menu && make - DESTDIR="" RPATH_LIST="/home/tom/exo/br/target/i586/host/lib" sources make[1]: Entering directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/menu' rm -f ../include/menu.h cp ./menu.h ../include/menu.h rm -f ../include/eti.h cp ./eti.h ../include/eti.h rm -f ../include/mf_common.h cp ./mf_common.h ../include/mf_common.h make[1]: Leaving directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/menu' cd form && make - DESTDIR="" RPATH_LIST="/home/tom/exo/br/target/i586/host/lib" sources make[1]: Entering directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/form' rm -f ../include/form.h cp ./form.h ../include/form.h make[1]: Leaving directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/form' cd test && make - DESTDIR="" RPATH_LIST="/home/tom/exo/br/target/i586/host/lib" sources make[1]: Entering directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/test' make[1]: Nothing to be done for 'sources'. make[1]: Leaving directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/test' cd misc && make - DESTDIR="" RPATH_LIST="/home/tom/exo/br/target/i586/host/lib" sources make[1]: Entering directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/misc' WHICH_XTERM=xterm-new \ XTERM_KBS=BS \ datadir=/home/tom/exo/br/target/i586/host/share \ /bin/sh ./gen_edit.sh >run_tic.sed echo '** adjusting tabset paths' ** adjusting tabset paths sed -f run_tic.sed ../misc/terminfo.src >terminfo.tmp make[1]: Leaving directory '/net/10.1.3.2/phoo/rootfs/home/tom/exo/br/target/i586/build/host-ncurses-6.0/misc' this is in: buildroot/package/ncurses/ncurses.mk define HOST_NCURSES_BUILD_CMDS $(HOST_MAKE_ENV) $(MAKE1) -C $(@D) sources $(HOST_MAKE_ENV) $(MAKE) -C $(@D)/progs tic endef What is MAKE1? Ok so MAKE1 is for non-parallel builds. Print those commands again to see what is actually going on. Maybe the "echo +" is coming from elsewhere? If I add this line it builds: define HOST_NCURSES_BUILD_CMDS echo "MAKE1=$(MAKE1)" $(HOST_MAKE_ENV) $(MAKE1) -C $(@D) sources $(HOST_MAKE_ENV) $(MAKE) -C $(@D)/progs tic endef Something weird is going on here. I guess that line eats the "echo +" part. Replaced it with this: echo "FIXME: this line is a nix build workaround" No I did this!!! tom@phoo:~/exo/deps/buildroot$ git diff diff --git a/package/ncurses/ncurses.mk b/package/ncurses/ncurses.mk index e63b6e4aed..c92a78f3c6 100644 --- a/package/ncurses/ncurses.mk +++ b/package/ncurses/ncurses.mk @@ -140,6 +140,7 @@ NCURSES_POST_INSTALL_TARGET_HOOKS += NCURSES_TARGET_CLEANUP_TERMINFO # ourselves, and use that during installation. # define HOST_NCURSES_BUILD_CMDS + echo "FIXME: this line is a nix build workaround" $(HOST_MAKE_ENV) $(MAKE1) -C $(@D) sources $(HOST_MAKE_ENV) $(MAKE) -C $(@D)/progs tic endef diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index a2a12e7b56..05f8b39486 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -247,6 +247,7 @@ $(BUILD_DIR)/%/.stamp_built:: @$(call step_start,build) @$(call MESSAGE,"Building") $(foreach hook,$($(PKG)_PRE_BUILD_HOOKS),$(call $(hook))$(sep)) + echo +$($(PKG)_BUILD_CMDS) +$($(PKG)_BUILD_CMDS) $(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep)) @$(call step_end,build) Wow... talking about shooting yourself in the foot. Next issue is jack. Entry: jack build error Date: Sun Oct 27 09:57:13 EDT 2019 ImportError: /home/tom/exo/br/target/i586/host/lib/python2.7/lib-dynload/operator.so: undefined symbol: _PyUnicodeUCS2_AsDefaultEncodedString Can this be bypassed? This has the same error: https://github.com/NixOS/nixpkgs/issues/9186 Why is it trying to run host python? Maybe remove python from the nix file? So it seems the buildroot pythong doesn't pick up the correct libs. exo-env-chrootenv:tom@phoo:~/exo/br$ ldd target/i586/host/bin/python2.7 linux-vdso.so.1 (0x00007ffc288fb000) libpython2.7.so.1.0 => /lib/libpython2.7.so.1.0 (0x00007f2fcdd35000) libpthread.so.0 => /lib/libpthread.so.0 (0x00007f2fcdd14000) libdl.so.2 => /lib/libdl.so.2 (0x00007f2fcdd0f000) libutil.so.1 => /lib/libutil.so.1 (0x00007f2fcdd0a000) libm.so.6 => /lib/libm.so.6 (0x00007f2fcdb74000) libc.so.6 => /lib/libc.so.6 (0x00007f2fcd9be000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f2fcd7a6000) /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/ld-linux-x86-64.so.2 => /nix/store/xvj2l5llpmcgx99cl37v5xqpazs38195-glibc-multi-2.27/lib/ld-linux-x86-64.so.2 (0x00007f2fcdf5c000) exo-env-chrootenv:tom@phoo:~/exo/br$ LD_LIBRARY_PATH=$(readlink -f ./target/i586/host/lib) ldd target/i586/host/bin/python2.7 linux-vdso.so.1 (0x00007ffd621c8000) libpython2.7.so.1.0 => /home/tom/exo/br/target/i586/host/lib/libpython2.7.so.1.0 (0x00007f7546a52000) libpthread.so.0 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libpthread.so.0 (0x00007f7546a31000) libdl.so.2 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libdl.so.2 (0x00007f7546a2c000) libutil.so.1 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libutil.so.1 (0x00007f7546a27000) libm.so.6 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libm.so.6 (0x00007f7546891000) libc.so.6 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libc.so.6 (0x00007f75466db000) /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/ld-linux-x86-64.so.2 => /nix/store/xvj2l5llpmcgx99cl37v5xqpazs38195-glibc-multi-2.27/lib/ld-linux-x86-64.so.2 (0x00007f7546c74000) Ok by setting those paths it seems fine except the post build scripts which can be fixed later. Entry: cleanup Date: Sun Oct 27 11:17:55 EDT 2019 - make it generic - fix the profile workaround Entry: profile workaround Date: Sun Oct 27 11:18:28 EDT 2019 fhs doesn't seem to work with "nix-shell --run" (:) tom@phoo:~/exo/br$ nix-shell --run "ls -al /bin" total 4 drwxr-xr-x 1 root root 4 Oct 26 09:35 . drwxr-xr-x 1 root root 234 Oct 26 09:46 .. lrwxrwxrwx 1 root root 63 Oct 26 09:35 sh -> /nix/store/9xg3d7r173zzybgh8v4lhf3zcf0wx51g-bash-4.4-p12/bin/sh The problem might be this: shellHook = '' if [ -z "$PS1" ]; then : else exec exo-env fi ''; So this: fhs = pkgs.buildFHSUserEnv { name = "exo-env"; will eventually set up that "exo-env" program. EDIT: cleaned it up a bit. Entry: nix-build Date: Sun Oct 27 12:45:33 EDT 2019 Now finish all this and wrap it up in a full nix-build. The output should be a package with build tools and root fs. For now it's enough to have "open" environments. Entry: fakeroot Date: Sun Oct 27 15:00:34 EDT 2019 Fakeroot is not working properly. PATH="/home/tom/exo/br/target/i586/host/bin:/home/tom/exo/br/target/i586/host/sbin:/home/tom/exo/br/bin:/run/wrappers/bin:/usr/bin:/usr/sbin" \ /home/tom/exo/br/target/i586/host/bin/fakeroot -- /home/tom/exo/br/target/i586/build/buildroot-fs/fakeroot.fs Fakeroot is hacky, I don't want to fix it. The upstream one seems to work, so add a workaround in fs/common.mk tom@phoo:~/exo/br/buildroot$ grep -re bin/fakeroot * fs/common.mk: PATH=$$(BR_PATH) $$(HOST_DIR)/bin/fakeroot -- $$(FAKEROOT_SCRIPT) Entry: new version different? Date: Sun Oct 27 17:52:55 EDT 2019 i'm still not 100% clear how reproducible this is. tom@rackhub:~/exo/br$ nix --version nix (Nix) 2.3.1 tom@panda:~$ nix --version nix (Nix) 2.2 tom@rackhub:~/exo/br$ nix-shell these derivations will be built: /nix/store/97m5bv8q0a2c4n04cq2znba9haw6wscg-fhs-buildroot-make.drv building '/nix/store/97m5bv8q0a2c4n04cq2znba9haw6wscg-fhs-buildroot-make.drv'... unpacking sources variable $src or $srcs should point to the source builder for '/nix/store/97m5bv8q0a2c4n04cq2znba9haw6wscg-fhs-buildroot-make.drv' failed with exit code 1 error: build of '/nix/store/97m5bv8q0a2c4n04cq2znba9haw6wscg-fhs-buildroot-make.drv' failed now to be fair, nix-build does require src to be set, so it might just be a bugfix making nix-shell and nix-build more similar? anyway, running nix-build on panda should be the error case to track down. I'm going to have to shift in how I think about this. Everything that goes into a nix build will need to be part of sources, so it is really not straightforward to wrap buildroot as it will perform downloads. Maybe best to separate out the download phase. This is going to be a lot of work. It might be simpler to just use it to build the compilers and images, and leave those sitting "open" somewhere. Entry: final word? Date: Sun Oct 27 18:42:00 EDT 2019 - work in "open setting" as much as possible - once a build is stable, rebuild it as closed - don't worry about inefficiencies anything else is either going to be very inefficient (bandwidth-wise) or just complicated to make it faster. buildroot is too big to handle in one piece with nix. the real solution is to build something similar on nix. what is possible though, is to keep an "open" setup. note that a full build is possible, but it will re-download on every build, so we probably want to create a local package mirror to do that. i could in theory create a manual mirror that is bootstrapped from an "open" build, and copy all the files into the temp store for building. the correct way to do it is just inefficient wrt. download bandwidth. that's all really. it would be possible to fix that by making all the sources explicit in buildroot, which is good practice anyway. Entry: multi-user nixos Date: Sun Oct 27 22:51:18 EDT 2019 sh <(curl https://nixos.org/nix/install) --daemon https://nixos.org/nix/manual/#sect-multi-user-installation but this requires systemd newer also need: sysctl kernel.unprivileged_userns_clone=1 what's this about /dev/fd/63: SHA-256 hash mismatch in 'https://nixos.org/releases/nix/nix-2.2/nix-2.2-x86_64-linux.tar.bz2'; expected sha256-R+JUpKAaI7B/bI40beaHpB8tkqPkoQBNE+906QOR4Xk=, got 47e254a4a01a23b07f6c8e346de687a41f2d92a3e4a1004d13ef74e90391e179 tom@rackhub:~$ sh <(curl https://nixos.org/releases/nix/nix-2.2.1/install) bash <(curl https://nixos.org/releases/nix/nix-2.2.1/install) --daemon root@zion:~# mount --bind /vol/raid_2tb/nix /nix Entry: nixos FHS chroots Date: Mon Oct 28 10:12:15 EDT 2019 This is actually quite a powerful tool, an alternative to build vms. Because of the chroot, it is fairly isolated as well from the base system. Entry: revisit basics -- nix FAQ Date: Tue Oct 29 15:29:38 EDT 2019 Time to revisit. Here is a list of things I did not understand properly before writing this down. 1. It is very confusing that nix-shell / nix-build takes one type of default.nix, and nixpkgs' default.nix is something else. One takeaway: default.nix is the nix expression that is associated to a directory. That's pretty much it. The type of this can be arbitrary. For now, focus on what goes into nix-shell, then generalize. 2. nix-shell interprets the nix file as a function that takes arguments to an expression of type Derivation. ( I'm using captial letters to denote types, which are not part of nix syntax but just a way to think about expressions ). 3. what does nix-shell pass in as arguments? is there a way to print the args? TODO: explain. 4. what i see most often is something like this: the derivation is produced by a default nixpkgs.stdenv, but it is possible to override that. { pkgs ? import ../deps/nixpkgs {} }: pkgs.stdenv.mkDerivation { name = "doodle1"; } 5. the link between nix expressions and /nix/store/*.drv is the built-in function "derviation". the .drv file is a simpler format that instructs nix how to build something. 6. nix-shell doesn't actually build the thing described by the derivation. it stops just short of building: it drops you in an environment that has all the build dependencies specified in "buildInputs" and "nativeBuildInputs". (see https://nixos.org/nixpkgs/manual/ for an explanation of those) rule of thumbs: latter is compilers, former is anything else, e.g. run-time libs. 7. how to understand this syntax? targetPkgs = pkgs: with pkgs; [ bash bc ]; it's equivalent to targetPkgs = [ pkgs.bash pkgs.bc ]; Two components: - 'with' introduces a dictionary elements as variables in the lexical scope of an expression, so the list expression can reference those names directly - the 'pgks:' part seems to indicate that the type of targetPkgs is a function that takes a dictionary with packages as an argument. That seems to correspond to what I see in: nixpkgs/pkgs/build-support/build-fhs-userenv/env.nix 8. the attributes that are passed into stdenv.mkDerivation are mapped to shell variables. e.g. "echo $buildInputs" 9. to install a nix file's product into the current environment nix-env -i -f default.nix Entry: perl modules in FHS Date: Tue Oct 29 17:16:22 EDT 2019 So that should be enough to fix buildFHSUserEnv such that it includes the perl modules. Step one, what's the origin of the existing modules? Entry: absolute paths for FHS compiled binaries Date: Tue Oct 29 19:39:45 EDT 2019 e.g.: (top_src_chroot)tom@panda:~/humanetics/top_src$ ldd /home/tom/priv/git-private/humanetics/top_cache/gw_buildroot-d4896d7d0e7bdb9e9b578a3f9904691757d7f2d2/target/bbb/host/bin/../lib/gcc/arm-buildroot-linux-musleabihf/6.4.0/../../../../arm-buildroot-linux-musleabihf/bin/as linux-vdso.so.1 (0x00007ffcb14f4000) libz.so.1 => /usr/lib/libz.so.1 (0x00007f4c5157e000) libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f4c51579000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f4c513c3000) /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/ld-linux-x86-64.so.2 => /nix/store/xvj2l5llpmcgx99cl37v5xqpazs38195-glibc-multi-2.27/lib/ld-linux-x86-64.so.2 (0x00007f4c5159f000) (top_src_chroot)tom@panda:~/humanetics/top_src$ readlink -f /usr/lib/libz.so.1 /nix/store/iiymx8j7nlar3gc23lfkcscvr61fng8s-zlib-1.2.11/lib/libz.so.1.2.11 Those only resolve properly inside the chroot. To make them work outside, the elfs need to be patched to the respective /nix/store paths. Entry: rust on nix Date: Tue Oct 29 20:52:02 EDT 2019 https://github.com/NixOS/nixpkgs/issues/61618 Entry: Building large projects with nixos Date: Wed Oct 30 06:17:40 EDT 2019 The solution should be simple: allow it to use caching. This way you can make a build work inside a nix shell, and once it is final, switch off caching in the config to do a test rebuild. Entry: Producing a properly linked toolchain Date: Wed Oct 30 06:20:16 EDT 2019 I want to limit the effects of the temporary FHS that is used to build a buildroot sdk, such that the sdk can also be used outside. This really should work. Let's give it a try in a toy problem. EDIT: Too much of a toy problem. EDIT: Makde a libusb1.0 toy problem that doesn't have paths set correctly after nix-build tom@panda:~/exo/nix/fhs$ ldd /nix/store/darkwrwklf1f50lnyqa7nj5nh3bn99f2-run-inner/bin/test linux-vdso.so.1 (0x00007ffee85d7000) libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007fc3107cd000) libc.so.6 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libc.so.6 (0x00007fc310a38000) libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fc310a17000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc3105b0000) /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fc3109e6000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc3103a8000) I need to know how this is usually done. I.e. create the same example, but do it the nix way. So here are some ideas: - To produce a buildroot toolchain, something needs to happen still. Either use wrappers that keep the FHS this was built in, or properly fix each binary. The wrappers seem to be a better idea. - It is not a problem for cross compilation targets of course. The question is then: how does nix fix up link paths. Look at the wrapper maybe? which gcc /nix/store/ghzg4kg0sjif58smj2lfm2bdvjwim85y-gcc-wrapper-7.4.0/bin/gcc That doesn't make me much wiser without already knowing some background. I think I understand what the idea is (patchs are patched), but I don't see where it is happening. Entry: how does the library resolution actually work? Date: Wed Oct 30 07:29:56 EDT 2019 I think I'm missing some knowledge about how elf files work. tom@panda:~/exo/nix/usb$ objdump -x test Dynamic Section: NEEDED libusb-1.0.so.0 NEEDED libc.so.6 RUNPATH /nix/store/xvfaxc9nrbr47x4b6n5hjny09nzkbzl0-test_native/lib64:/nix/store/xvfaxc9nrbr47x4b6n5hjny09nzkbzl0-test_native/lib:/nix/store/fa943g2xwhcd7m9dv3y13cpb80rf9y0m-libusb-1.0.22/lib:/nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib:/nix/store/hlnxw4k6931bachvg5sv0cyaissimswb-gcc-7.4.0-lib/lib So a key element here is that the libraries do not have hardcoded paths, and the information is contained in RUNPATH: /nix/store/xvfaxc9nrbr47x4b6n5hjny09nzkbzl0-test_native/lib64 /nix/store/xvfaxc9nrbr47x4b6n5hjny09nzkbzl0-test_native/lib /nix/store/fa943g2xwhcd7m9dv3y13cpb80rf9y0m-libusb-1.0.22/lib /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib /nix/store/hlnxw4k6931bachvg5sv0cyaissimswb-gcc-7.4.0-lib/lib Which, if we look at a binary that is generated inside the FHS emulation, looks like: tom@panda:~/humanetics/gw_buildroot/target/bbb/host/bin$ objdump -x arm-buildroot-linux-musleabihf-gcc Dynamic Section: NEEDED libc.so.6 RUNPATH So it is just not set at all. In another example, it is misses a couple of items: tom@panda:~/exo/nix/fhs$ objdump -x test |grep RUNPATH RUNPATH /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib:/nix/store/hlnxw4k6931bachvg5sv0cyaissimswb-gcc-7.4.0-lib/lib tom@panda:~/exo/nix/fhs$ ldd test linux-vdso.so.1 (0x00007ffd9e9bf000) libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007fbd38f1d000) libc.so.6 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libc.so.6 (0x00007fbd39188000) libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fbd39167000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fbd38d00000) /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fbd39136000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fbd38af8000) Can I try to add this manually? Let's just give it a try. Fopr the libusb example, first make sure the build environment knows the location of libusb. This seems to work: in pkgs.stdenv.mkDerivation rec { ... for_rpath = with pkgs; [ libusb1 ]; } [nix-shell:~/exo/nix/fhs]$ patchelf --set-rpath $for_rpath/lib:$(patchelf --print-rpath test) test [nix-shell:~/exo/nix/fhs]$ echo $(patchelf --print-rpath test) /nix/store/fa943g2xwhcd7m9dv3y13cpb80rf9y0m-libusb-1.0.22/lib:/nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib:/nix/store/hlnxw4k6931bachvg5sv0cyaissimswb-gcc-7.4.0-lib/lib [nix-shell:~/exo/nix/fhs]$ ldd test linux-vdso.so.1 (0x00007ffec1df9000) libusb-1.0.so.0 => /nix/store/fa943g2xwhcd7m9dv3y13cpb80rf9y0m-libusb-1.0.22/lib/libusb-1.0.so.0 (0x00007fd8992f0000) libc.so.6 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libc.so.6 (0x00007fd89913a000) libudev.so.1 => /nix/store/4xmd2nxkdgzjyi7g1v2kfvicjdiqkll8-systemd-239.20190219-lib/lib/libudev.so.1 (0x00007fd899111000) libgcc_s.so.1 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libgcc_s.so.1 (0x00007fd898efb000) libpthread.so.0 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libpthread.so.0 (0x00007fd898eda000) /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/ld-linux-x86-64.so.2 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib64/ld-linux-x86-64.so.2 (0x00007fd89930e000) librt.so.1 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/librt.so.1 (0x00007fd898ece000) libmount.so.1 => /nix/store/l2fmlk757bza3myxfjj55kkbbh3yzsaa-util-linux-2.33.1/lib/libmount.so.1 (0x00007fd898e6f000) libblkid.so.1 => /nix/store/l2fmlk757bza3myxfjj55kkbbh3yzsaa-util-linux-2.33.1/lib/libblkid.so.1 (0x00007fd898e1a000) libuuid.so.1 => /nix/store/l2fmlk757bza3myxfjj55kkbbh3yzsaa-util-linux-2.33.1/lib/libuuid.so.1 (0x00007fd898e11000) Let's try for the buildroot toolchain: tom@panda:~/humanetics/top_cache/gw_buildroot-d4896d7d0e7bdb9e9b578a3f9904691757d7f2d2/target/bbb/host/bin$ ldd arm-buildroot-linux-musleabihf-gcc linux-vdso.so.1 (0x00007ffd858f5000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5fa072f000) /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f5fa0ace000) This seems to just need glibc path. Ok so I think I understand what needs to be done. EDIT: This is odd. New build does have rpath set properly: tom@panda:~/exo/br$ ldd target/i586/host/bin/i586-buildroot-linux-uclibc-gcc linux-vdso.so.1 (0x00007ffd2b9b4000) libc.so.6 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libc.so.6 (0x00007ff33bd69000) /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007ff33bcfe000) tom@panda:~/exo/br$ objdump -x target/i586/host/bin/i586-buildroot-linux-uclibc-as | grep RUNPATH RUNPATH /home/tom/exo/br/target/i586/host/lib:/nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib So it might not be necessary after all. Or buildroot does something weird when it patches things up to run as relocatable sdk. Check later. EDIT: So buildroot "sdk" target does mess with rpath: /home/tom/exo/deps/buildroot/support/scripts/fix-rpath host /home/tom/exo/deps/buildroot/support/scripts/fix-rpath staging It's probably ok to just disable "fix-rpath", since the paths are fine. patch it here: buildroot/Makefile .PHONY: sdk sdk: world @$(call MESSAGE,"Rendering the SDK relocatable") $(TOPDIR)/support/scripts/fix-rpath host $(TOPDIR)/support/scripts/fix-rpath staging $(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location EDIT: Ok, silly. Just don't run that sdk target. Leave the paths as they are. Entry: Installing on debian Date: Wed Oct 30 08:39:42 EDT 2019 # Only use 2.2.1 for now due to nix-shell issue. # Make sure sudo is set up properly sudo sysctl kernel.unprivileged_userns_clone=1 sudo apt-get -y install curl sh <(curl https://nixos.org/releases/nix/nix-2.2.1/install) EDIT: This works on gwprod/gwtest and zoo/panda, but not zion/debnix. WTF? EDIT: It worked after also adding "sandbox = false" in ~/.config/nix/nix.conf So not clear what is the actual limiting factor. The other two probably have that set already? No there's something else that must be diffent. This shit is all getting too fucking complicated. Entry: buildroot Date: Wed Oct 30 10:29:06 EDT 2019 Here's what buildroot says about external toolchains: https://buildroot.org/downloads/manual/manual.html If you want to generate a custom toolchain for your project, that can be used as an external toolchain in Buildroot, our recommendation is definitely to build it with crosstool-NG. We recommend to build the toolchain separately from Buildroot, and then import it in Buildroot using the external toolchain backend. Maybe this is a good next step. For now just keep it all contained. Second: relocatable SDK: run relocate-sdk.sh after! Alternatively, Buildroot can also export the toolchain and the development files of all selected packages, as an SDK, by running the command make sdk. This generates a tarball of the content of the host directory output/host/, named _sdk-buildroot.tar.gz (which can be overriden by setting the environment variable BR2_SDK_PREFIX) and located in the output directory output/images/. This tarball can then be distributed to application developers, when they want to develop their applications that are not (yet) packaged as a Buildroot package. Upon extracting the SDK tarball, the user must run the script relocate-sdk.sh (located at the top directory of the SDK), to make sure all paths are updated with the new location. So to import into nix, generate sdk, and maybe patch that relocate-sdk.sh to point to the /nix/store location. Entry: build cache Date: Wed Oct 30 17:13:50 EDT 2019 Find a generic way to "fake" a nix-build by building in a directory different from source, and hashing "git diff" output as part of what gets hashed to determine nix build. As long as we're honest about all effective source changes, incremental rebuilds should just work. Entry: faking the nix-build src Date: Thu Oct 31 10:18:18 EDT 2019 I'm still a little confused about how nix-build actually works. Maybe not yet completely awake here this morning, but it cannot find scripts I'm referencing. The difference between nix-build and nix-shell, is that the former is executed in a temporary directory which only has the source tree mentioned in the "src" attribute of the derivation. So if that is not set up properly, it is possible that something that works in nix-shell will not work in nix-build. Entry: revisit basics -- nix-build FAQ Date: Thu Oct 31 11:45:48 EDT 2019 1. The builder attribute points to a script that is executed with bash. It does not seem to have PATH variable set. Why? 2. The builder script has almost nothing in its enviroment except for a couple of variables that can be made to point to components. So any explicit builder script is a good reference point to see how a build environment is set up. Entry: rpaths still not ok Date: Thu Oct 31 13:52:39 EDT 2019 I have a nix-build +- working, but still have undefined symbols: tom@panda:~$ ldd /nix/store/9c79arzzc34jhy2h232wdrkvcyndk0kz-buildroot-i586/host/bin/../libexec/gcc/i586-buildroot-linux-uclibc/6.4.0/cc1 linux-vdso.so.1 (0x00007ffecac37000) libmpc.so.3 => /usr/lib/x86_64-linux-gnu/libmpc.so.3 (0x00007f4982dbf000) libmpfr.so.4 => /usr/lib/x86_64-linux-gnu/libmpfr.so.4 (0x00007f4982b58000) libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f49828d5000) libdl.so.2 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libdl.so.2 (0x00007f49831da000) libm.so.6 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libm.so.6 (0x00007f4983044000) libc.so.6 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libc.so.6 (0x00007f498271f000) /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f4982fd7000) Should be easy enough to patch up: tom@panda:/nix/store/9c79arzzc34jhy2h232wdrkvcyndk0kz-buildroot-i586$ find -name '*libmpc*' ./host/lib/libmpc.so.3.0.0 ./host/lib/libmpc.so.3 ./host/lib/libmpc.so ./host/lib/libmpc.la EDIT: Re-use the intelligence inside fix-rpath utility? Anyways it looks like it is working now. Will need some more testing. Maybe first move exo builds to nix? Entry: An incremental build cache Date: Thu Oct 31 16:54:31 EDT 2019 So solve it for the hatd project first. That one has: - current source state, not necessarily in git (commited when done) - a download cache for buildroot (surviving full rebuilds) - current build state (deleted on full rebuild) The final version should have: - build script referencing the source package - full rebuild Or don't cache the source at all. For exo it is pretty much the same. So I wonder: somebody has already done this. Entry: how is a nix hash computed? Date: Thu Oct 31 18:35:50 EDT 2019 Obviously not always using the source, because that would require a source download before a package can be compiled. I think it assumes that the URL of the source is enough to uniquely specify the source. Entry: still not there Date: Thu Oct 31 19:46:56 EDT 2019 >>> Executing post-image script board/beaglebone/post-image.sh genimage: error while loading shared libraries: libconfuse.so.1: cannot open shared object file: No such file or directory make: *** [Makefile:766: target-post-image] Error 127 make: Leaving directory '/home/tom/exo/deps/buildroot' + exit 1 builder for '/nix/store/1ixhhq00qv32rmrlb15gszkgfkpz3qdw-buildroot-bbb.drv' failed with exit code 1 error: build of '/nix/store/1ixhhq00qv32rmrlb15gszkgfkpz3qdw-buildroot-bbb.drv' failed Entry: temporary nix-build cache hack Date: Thu Oct 31 20:33:54 EDT 2019 Here's how to do it: - in the builder script, put a hard-coded "cd" to the directory that has the cache. - when done, remove the "cd" command This is a minimal change that forces you to actually remove the impurity to perform a pristine build, and when that build succeeds you can commit to source control. Entry: fetchgit or different approach? Date: Thu Oct 31 21:24:17 EDT 2019 I have a case that doesn't quite cut it, but let's try it anyway. EDIT: Ok it sort of works. I'll need another day of playing with this though. Entry: why is XML/Parser.pm not showing up? Date: Fri Nov 1 10:16:07 EDT 2019 # Debug perl = pkgs.perl; xmlparser = pkgs.perlPackages.XMLParser; [nix-shell:~/humanetics/top_src/timesys]$ find $xmlparser | grep XML/Parser.pm /nix/store/kdlcgmbcsx4wdc94zg2x3kfzl2pd0csp-perl5.28.1-XML-Parser-2.44/lib/perl5/site_perl/5.28.1/x86_64-linux-thread-multi/XML/Parser.pm It is actually there: tom@panda:/usr/lib/perl5/site_perl/5.28.1/x86_64-linux-thread-multi/XML$ ls -l Parser.pm -r--r--r-- 1 tom tom 27108 Jan 1 1970 Parser.pm So why is it not found? Can't locate XML/Parser.pm in @INC (you may need to install the XML::Parser module) (@INC contains: /nix/store/kiapz3kjvb46jbvswdjkmrzg0s4l2zm7-perl-5.28.1/lib/perl5/site_perl/5.28.1/x86_64-linux-thread-multi /nix/store/kiapz3kjvb46jbvswdjkmrzg0s4l2zm7-perl-5.28.1/lib/perl5/site_perl/5.28.1 /nix/store/kiapz3kjvb46jbvswdjkmrzg0s4l2zm7-perl-5.28.1/lib/perl5/5.28.1/x86_64-linux-thread-multi /nix/store/kiapz3kjvb46jbvswdjkmrzg0s4l2zm7-perl-5.28.1/lib/perl5/5.28.1 ) at ./test.pl line 3. BEGIN failed--compilation aborted at ./test.pl line 3. I guess in general this needs a wrapper that has the paths set up properly. Googling for this I don't really find much. EDIT: So I've created a wrappre and added the path, but apparently timesys insists on using /usr/bin/perl: But the checking for perl... /usr/bin/perl checking for perl >= 5.8.1... 5.28.1 checking for XML::Parser... configure: error: XML::Parser perl module is required for intltool Maybe PATH gets reset? The config.ac uses AC_PATH_PROG AC_PATH_PROG(PERL, perl) if test -z "$PERL"; then AC_MSG_ERROR([perl not found]) fi https://stackoverflow.com/questions/53089239/ac-path-prog-always-return-usr-bin-prog-instead-of-my-own-path-prog It fails on not finding libunistring.so.0 The FHS has libunistring.so.2 from upstream version 0.9.10 build@gwtest-kz:~$ dpkg -l |grep libunistring ii libunistring0:amd64 0.9.3-5.2+b1 amd64 Unicode string library for C How can 0.9.10 be soversion=2 and 0.9.3 be soversion=0 Yeah I don't want to get into this shit. Can I just run a script before sealing up the FHSUserEnv? EDIT: Use extraBuildCommands to create a symlink. Next it does a hickup on fakeroot: make: Entering directory '/home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209' -- Reading configuration and build instructions -- 1572623758 [Fri, 01 Nov 2019 15:55:58 +0000] Please wait, this will take some time... -- packaging toolchain (initial) -- 1572623760 [Fri, 01 Nov 2019 15:56:00 +0000] true cd /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/toolchain && \ PATH=/home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/fake/bin:\ /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/toolchain/ccache:\ /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/toolchain/bin:\ /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/toolchain/usr/bin:\ /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/toolchain/sbin:\ /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/toolchain/usr/sbin:\ /nix/store/axbgfpy80dnpxwx1gkjsjj2iqz9sj9lq-src/fhs-wrappers/bin:\ /run/wrappers/bin:\ /usr/bin:\ /usr/sbin\ fakeroot -- /bin/bash -c "if [ -f /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/toolchain/.pkg.sh ]; then /bin/bash /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/toolchain/.pkg.sh; rm /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/toolchain/.pkg.sh; fi; /usr/bin/tar czf /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/toolchain-initial-armv7l-timesys-linux-gnueabi.tgz ." fakeroot: preload library `libfakeroot.so' not found, aborting. make: *** [include/host.mk:32: /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/toolchain-initial-armv7l-timesys-linux-gnueabi.tgz] Error 1 make: Leaving directory '/home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209' + exit 1 builder for '/nix/store/6fkha5v7dfvcbbm6mqqsj40md7rqqb9q-timesys-vybrid.drv' failed with exit code 1 error: build of '/nix/store/6fkha5v7dfvcbbm6mqqsj40md7rqqb9q-timesys-vybrid.drv' failed It's messing with PATH again. Next error: chmod u+s /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/rfs/bin/busybox chmod: changing permissions of '/home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/rfs/bin/busybox': Operation not permitted make: *** [target/software/System/busybox/busybox.mk:49: /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/busybox-1.22.1/.stamp_init_rfs_installed] Error 1 make: Leaving directory '/home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209' + exit 1 + exit 1 builder for '/nix/store/h3ng02c4ggf1mcmihkl9rn3mld5c6msw-timesys-vybrid.drv' failed with exit code 1 error: build of '/nix/store/h3ng02c4ggf1mcmihkl9rn3mld5c6msw-timesys-vybrid.drv' failed Maybe this is supposed to run inside of fakeroot? Nope it's a nix thing. In debian it works fine: tom@panda:~/top_src/timesys$ chmod u+s /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/rfs/bin/busybox Wait a minute. In the nix shell it works fine too: [nix-shell:~/top_src/timesys]$ [nix-shell:~/top_src/timesys]$ chmod u-s /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/rfs/bin/busybox [nix-shell:~/top_src/timesys]$ [nix-shell:~/top_src/timesys]$ chmod u+s /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/rfs/bin/busybox just not in the chroot? Nope works also. WTF? I've removed the error by wrapping chmod, but this probably should be fixed. chmod u+s /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/rfs/bin/busybox /usr/bin/chmod: changing permissions of '/home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/rfs/bin/busybox': Operation not permitted ERROR: chmod u+s /home/tom/priv/git-private/humanetics/top_src/timesys/factory-20141209/build_armv7l-timesys-linux-gnueabi/rfs/bin/busybox Running into the same issue for gw_buildroot. This was working yesterday... chmod 4755 /home/tom/priv/git-private/humanetics/top_src/gw_buildroot/target/bbb/host/bin/mount chmod: changing permissions of '/home/tom/priv/git-private/humanetics/top_src/gw_buildroot/target/bbb/host/bin/mount': Operation not permitted make[5]: *** [Makefile:12748: install-exec-hook-mount] Error 1 make[5]: *** Waiting for unfinished jobs.... This is something that works ok inside nix-shell, but not inside nix-build. A temporary workaround is to run the build in nix-shell, then package it up in nix-build and fix the issues later. Entry: nix-build doesn't allow setuid Date: Fri Nov 1 14:05:19 EDT 2019 https://github.com/NixOS/nixpkgs/issues/26600 So the problem is that setuid is not allowed inside a nix-build sandbox. Is it really necessary for buildroot? Let's make a wrapper that at least keeps track of what is attempted to be set, and then set it inside fakeroot? https://github.com/NixOS/nix/issues/1429 Entry: cabal2nix Date: Fri Nov 1 14:31:05 EDT 2019 There's a bootstrapping problem with this. Jeez that needs to be fixed as well. EDIT: Just committing files for now. Entry: writable source dir Date: Fri Nov 1 21:00:39 EDT 2019 can i just have a copy of the source dir that is writable? no. Entry: disable sandbox Date: Fri Nov 1 23:40:29 EDT 2019 [master] build@gwtest-kz:~/cache/top_src/gw_src/meta/hs$ nix-build release.nix these derivations will be built: /nix/store/hcfkbrp6x5pnrkz45gihb1v855i9a7ad-gw-meta-1.0.0.drv error: while setting up the build environment: mounting /proc: Operation not permitted [master] build@gwtest-kz:~/cache/top_src/gw_src/meta/hs$ nix-build release.nix --option sandbox false probably also solves the setuid No it doesn't Entry: So, what is a reproducible build? Date: Mon Nov 4 07:40:10 EST 2019 One issue I have with this, is that external inputs (basically, sorce location, and even the presence over time of a source) should be handled. Nix does that using "mirrors" I think. I.e. it doesn't care about where a package comes from, as long as it is the same binary. Entry: Embedded meta system Date: Mon Nov 4 09:27:54 EST 2019 In the end the principle is simple: provide location of libraries at compile time and at run time. It would be good to solve this permanently. I've been trying for a while but just don't get there. Entry: Packaging buildroot Date: Thu Nov 7 15:46:34 EST 2019 I want to do it properly. Buildroot is not the problem. The toolchain is. So how to package the toolchain separately? Or, use crosstool-ng, which is mentioned here: https://buildroot.org/downloads/manual/manual.html That page also mentions the main point: In general, all toolchains that support the _sysroot_ feature should work. Maybe best to start somewhere else? https://nixos.org/nix-dev/2016-August/021423.html Googling a bit. Found this: https://discourse.nixos.org/t/gcc-arm-embedded-gcc-7/1023 Trying: buildInputs = with pkgs; [ gnumake pkgsCross.muslpi.buildPackages.gcc ]; Wich has: [nix-shell:~/exo/nix/arm-embedded]$ readlink $(which arm-none-eabi-gcc) /nix/store/4xdcrlr875y2jlk8nw130z4rvmfp7g68-gcc-arm-embedded-8-2018-q4-major/bin/arm-none-eabi-gcc Pff.. figure this out some other time. Not really documented? Since buildroot mentions crosstool-ng, maybe just use that instead? unpacking source archive /nix/store/pflqnc2zdnxd92rn1659qmgiyb3x13nx-crosstool-ng-1.23.0.tar.bz2 source root is cmdliner-1.0.2 setting SOURCE_DATE_EPOCH to timestamp 1502118649 of file cmdliner-1.0.2/test/test_term_dups.ml patching sources configuring no configure script, doing nothing building build flags: SHELL=/nix/store/cinw572b38aln37glr0zb8lxwrgaffl4-bash-4.4-p23/bin/bash ocamlbuild -use-ocamlfind src/cmdliner.cma /nix/store/cinw572b38aln37glr0zb8lxwrgaffl4-bash-4.4-p23/bin/bash: ocamlbuild: command not found make: *** [Makefile:44: build-byte] Error 127 builder for '/nix/store/zcrlkkhqy41jlfyi6lcf3q3jhk9swhmf-crosstool-ng.drv' failed with exit code 2 error: build of '/nix/store/zcrlkkhqy41jlfyi6lcf3q3jhk9swhmf-crosstool-ng.drv' failed tom@panda:~/exo/nix/crosstool-ng$ ls -al total 12 drwxr-xr-x 1 tom tom 86 Nov 7 16:33 . drwxr-xr-x 1 tom tom 124 Nov 7 16:13 .. -rw-r--r-- 1 tom tom 280 Nov 7 16:33 default.nix lrwxrwxrwx 1 tom tom 28 Nov 7 16:23 .#Makefile -> tom@panda.zoo.977:1572787197 -rw-r--r-- 1 tom tom 29 Nov 7 16:25 #Makefile# -rw-r--r-- 1 tom tom 0 Nov 7 16:22 Makefile~ drwxr-xr-x 1 tom tom 28 Nov 7 16:30 tmp EDIT: Do not just copy hashes! It will try to download it from somewhere else. { pkgs ? import ../../deps/nixpkgs {} }: pkgs.stdenv.mkDerivation { name = "crosstool-ng"; src = pkgs.fetchurl { url = "https://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.24.0.tar.bz2"; # Do not sha256 = "18jqphjiifljlh9jg8zpl6310p3iwyaqphdkmf89acyaix0s4kj1"; }; } EDIT: Wrapping nix is hard. Emulating nix-build just isn't very easy for large builds. these things are really at odds. i'm resisting. Entry: ct-ng build didn't finish properly Date: Fri Nov 8 07:31:26 EST 2019 [INFO ] Finalizing the toolchain's directory: done in 3.02s (at 22:50) [INFO ] Build completed at 20191108.052821 [INFO ] (elapsed: 22:49.85) [INFO ] Finishing installation (may take a few seconds)... + mkdir -p /nix/store/9bmcaszi9q45ppkq0z01zlfyjkd65pcn-crosstool-ng-arm + cp -a .build/arm-unknown-eabi/buildtools/arm-unknown-eabi .build/arm-unknown-eabi/buildtools/bin .build/arm-unknown-eabi/buildtools/include .build/arm-unknown-eabi/buildtools/lib .build/arm-unknown-eabi/buildtools/libexec .build/arm-unknown-eabi/buildtools/share /nix/store/9bmcaszi9q45ppkq0z01zlfyjkd65pcn-crosstool-ng-arm/ ** (process:5872): ERROR **: 00:28:27.175: main: nftw(prefix, nftw_remove, getdtablesize(), FTW_DEPTH | FTW_MOUNT | FTW_PHYS): Permission denied builder for '/nix/store/gzan7f9gp1gqa213d6wb0mv9iqqm3cr1-crosstool-ng-arm.drv' failed due to signal 5 (Trace/breakpoint trap) error: build of '/nix/store/gzan7f9gp1gqa213d6wb0mv9iqqm3cr1-crosstool-ng-arm.drv' failed EDIT: This looks like it wants to remove something, judging from nftw_remove. I wonder if that is inside of cp. Weird eh. https://github.com/coreutils/coreutils/blob/master/src/cp.c No, googling forn "ftw prefix nftw_remove getdtablesize" leads to nix sources. https://github.com/NixOS/nixpkgs/issues/33106 Maybe try again if this really needs chrootenv? Because that's where this points at. EDIT: On zion I noticed ~/x-tools with the cross compiler. I think it puts it there without "asking". Entry: nix-env -i vs -iA Date: Fri Nov 8 08:52:43 EST 2019 what is the difference? the latter picks a toplevel attribute from nixpkgs, which is clear. not sure what the other one does. just use -iA Entry: Building larger systems Date: Sun Nov 10 11:37:17 EST 2019 I run into too many bugs during the building of larger systems. This is a real practical hurdle. Half an hour iteration time is just too much to solve all these accidental complexity issues. The core issue is that I can't make nix-build reproducible. I.e. nix-shell is not similar enough to nix-build. Entry: A practical solution for ct-ng? Date: Sun Nov 10 11:39:57 EST 2019 Maybe it just needs to avoid the FHSUserEnv. EDIT: Objcopy is missing from binutils due to it actually being binutils-wrapper. Nixpkgs is quite complex. Not always well-designed it seems. Designed from a very specific perspective. I'm going to let this go for a bit. Maybe clarity will come later. Entry: Relaxed nixos Date: Sun Nov 10 16:55:52 EST 2019 Basically I need incremental builds. Finer granularity. Entry: I need to understand the crosssytem stuff in nix Date: Sun Nov 10 17:27:54 EST 2019 The question is always where to start looking. Here's a random point: nix-env -iA nixpkgs.stdenv.cc --arg crossSystem '{ config = "arm-linux-gnueabihf"; }' Entry: Wow nothing fucking works Date: Sun Nov 10 20:02:50 EST 2019 "just" compiling something is apparently a problem. Trying to compile a gdb cross compiler. Maybe I need more experience juste working with nixpkgs as is. Entry: nix-shell + genericBuild Date: Sun Nov 10 21:02:27 EST 2019 Just CD to an empty temp dir. Trying that, and it fails for the the same thing. Somehow the install arget for readline doesn't do anything. make[2]: Entering directory '/home/tom/exo/nix/arm-eabi-gdb/tmp/gdb-7.12.1/readline' make[2]: Nothing to be done for 'install'. No that's intentional. The readline packaged with gdb doesn't install. It just doesn't generate the .a If I do it by hand (make libreadline.a) it works. This is odd: make[2]: Entering directory '/home/tom/exo/nix/arm-eabi-gdb/tmp/gdb-7.12.1/readline' make[2]: Nothing to be done for 'all'. The TARGETS variable is empty. The .am contains: TARGETS = @STATIC_TARGET@ @SHARED_TARGET@ Maybe it just needs to be explicitly configured to use system readline instead? Yep that's it. --with-system-readline Ok, so not that dramatic then. Entry: nix-build --keep-failed Date: Sun Nov 10 21:35:48 EST 2019 Do that, then use nix-shell pure to go into the dir and try to patch the build up, then rebuild. Entry: trying to build erlang extensions in nix-shell Date: Sat Nov 16 09:02:17 EST 2019 Problem: make: Entering directory '/home/tom/exo/erl/_build/default/lib/efuse/c_src' gcc efuse.o -lerl_interface -lei -pthread -lnsl -lfuse -lrt -ldl -o efuse /nix/store/0y7jmqnj48ikjh37n3dl9kqw9hnn68nq-binutils-2.31.1/bin/ld: cannot find -lerl_interface /nix/store/0y7jmqnj48ikjh37n3dl9kqw9hnn68nq-binutils-2.31.1/bin/ld: cannot find -lei Which is here: tom@panda:/nix/store/6wf8diszg2v6qjhgdb89v3xv5xfwqaij-erlang-19.3.6.11$ find -name '*erl_interface*' ./lib/erlang/usr/lib/liberl_interface.a ... The problem here is that erlang-efuse's c_src/Makefile calls gcc directly. So it makes more sense to add a wrapper script that adds the proper -L function. Instead of a script, use a shell function, so it can just be included inside the .nix wrapper. gcc is probably already a shell function. No. so I'm missing a fundamental piece of insight: how do you influence the shell environment apart from just setting variables? shellHook Entry: wrappers Date: Sat Nov 16 09:27:43 EST 2019 There is probably a better way, but this seems simplest: - add a shellHook to insert a directory in the path - put wrapper scripts there - let the wrapper script pick locations from the environemnt The shellHook is quite simple this way: shellHook = '' export PATH=${./nix-wrappers}:$PATH ''; Entry: Buildroot on nix Date: Wed Nov 20 09:47:59 EST 2019 I think I've found a way to do this by side-stepping the entire issue. It is not necessary to encapsulate buildroot as nix packages. The reason is mostly that it is much more convenient to be able to use buildroot as an "open" system. E.g. something that has a current build cache such that it can be used for incremental builds. Two things are important to see: - nix-env can be used to create binary products on top of /nix - those binaries could be installed network-wide as long as the /nix dependencies are installed as well It is usually ok to have only a single version in the current "open" development environment. This still gives a well-packaged, well-defined version-pinned tree, but avoids some of the strong nix constraints and inconveniences. Especially for embedded software, the end products will not depend on /nix at all. For a target that supports /nix, use Nix's cross compiler infrastructure. I currently do not have a need for that (Pi, BBB, or the Colibri). Entry: fhs Date: Sun Nov 24 06:21:46 EST 2019 Don't use an "override" directory. It seems that some config scripts will just hardcode things, such as /usr/bin/perl in the Erlang config. Just modify the fhs directly. EDIT: ok this seems to work. Entry: substituters Date: Sun Nov 24 10:48:20 EST 2019 cd /i/tom/exo/nix nix-env -f erlang.nix -i --substituters ssh://panda This gives: warning: substituter 'ssh://panda' does not have a valid signature for path '/nix/store/8kfxvs683f951s19bkjbjk3p93p4vqvr-erlang-19.3' nix-store --generate-binary-cache-key panda /nix/secret-key /nix/public-key I tried a couple of things: generating keys, adding them to nix.conf, using ssh-ng instead of ssh. There are too many parameters here, and it seems things have changed a lot over time so not quite sure where to get the information. This worked, and is ok for now: nix-env -f /i/tom/exo/nix/erlang.nix -i --substituters ssh-ng://tom@10.1.3.29 --no-require-sigs Entry: buildroot on nix Date: Tue Nov 26 06:38:30 EST 2019 If the problem with buildroot is fakeroot and setuid, just don't generate the images? And keep the current intermediate install? This allows the incremental build to be left there. Entry: Current stance on Nix-based dev env Date: Sat Apr 4 16:28:44 EDT 2020 1. If you can use a "closed" package, then do so. 2. If you can't it's ok to just install stuff on top of a nix profile. It is a serious pain to translate "convenience scripts" to the Nix way, so just install these things in an "open" fashon. The main reason I need to use nix is to easily duplicate a dev setup. This is probably OK. Anything that is deployed as application should probably be a "closed" package.