Nixin on bare metal

These are configuration files to build images that can be used to deploy nixin on headless bare metal machines

Image customisation

Before building images, you need to customize some settings in nixin/nixin-install.nix, like initial user account password, custom sshd port and ssh public keys that will be used to later on deploy nixin configuration to the machine.

Images are built using nixos-generators. Some platforms are using a custom niox-generators format. Check the section of platform in this readme to get the exact nixos-generator command to use.

ARM boards

To cross compile from an x86-64 computer, you need to enable binfmt emulation. for exemple, if the target is aarch64-linux, add the following to your configuration.nix :

{
  # Enable binfmt emulation of aarch64-linux.
  boot.binfmt.emulatedSystems = [ "aarch64-linux" ];
}

for more options, check the cross-compiling section of nixos generators documentation

Raspberry Pi boards

Raspberry Pi 4

Work In Progress

Benchmark

[nix-shell:~]$ 7z b

7-Zip [64] 17.05 : Copyright (c) 1999-2021 Igor Pavlov : 2017-08-28
p7zip Version 17.05 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,4 CPUs LE)

LE
CPU Freq: 64000000 - - - - - - - -

RAM size:    3756 MB,  # CPU hardware threads:   4
RAM usage:    882 MB,  # Benchmark threads:      4

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:       3864   327   1151   3760  |      75561   396   1628   6447
23:       3757   344   1113   3829  |      74106   395   1625   6412
24:       3674   353   1120   3951  |      73185   396   1622   6425
25:       3628   364   1139   4143  |      71700   396   1612   6381
----------------------------------  | ------------------------------
Avr:             347   1131   3921  |              396   1622   6416
Tot:             371   1376   5168

[nix-shell:/mnt/root]# cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       308404 iterations per second for 256-bit key
PBKDF2-sha256     546133 iterations per second for 256-bit key
PBKDF2-sha512     448109 iterations per second for 256-bit key
PBKDF2-ripemd160  269141 iterations per second for 256-bit key
PBKDF2-whirlpool  107967 iterations per second for 256-bit key
argon2i       4 iterations, 278284 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id      4 iterations, 284321 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
#     Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b        22.8 MiB/s        78.8 MiB/s
    serpent-cbc        128b        35.0 MiB/s        36.4 MiB/s
    twofish-cbc        128b        53.7 MiB/s        57.8 MiB/s
        aes-cbc        256b        17.4 MiB/s        59.3 MiB/s
    serpent-cbc        256b        35.7 MiB/s        36.4 MiB/s
    twofish-cbc        256b        54.7 MiB/s        57.8 MiB/s
        aes-xts        256b        86.6 MiB/s        76.6 MiB/s
    serpent-xts        256b        36.4 MiB/s        37.9 MiB/s
    twofish-xts        256b        58.3 MiB/s        62.0 MiB/s
        aes-xts        512b        66.4 MiB/s        58.3 MiB/s
    serpent-xts        512b        37.5 MiB/s        38.0 MiB/s
    twofish-xts        512b        59.5 MiB/s        61.9 MiB/s

# Using 1TB NVME SSD with USB interface
[nix-shell:/mnt/root]# dd if=/dev/zero of=./test.random bs=4M count=1024 status=progress conv=fsync oflag=direct
4106223616 bytes (4.1 GB, 3.8 GiB) copied, 15 s, 274 MB/s4294967296 bytes (4.3 GB, 4.0 GiB) copied, 15.6967 s, 274 MB/s

1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 15.698 s, 274 MB/s

[nix-shell:/mnt/root]# echo 3 > /proc/sys/vm/drop_caches

[nix-shell:/mnt/root]# dd if=./test.random of=/dev/null bs=4M count=1024 status=progress
4018143232 bytes (4.0 GB, 3.7 GiB) copied, 13 s, 307 MB/s
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 13.6496 s, 315 MB/s

Power usage

Measures done with nixos running on sd card the ethernet interface connected, a 1TB NVME SSD connected through a USB adapter and no other USD device connected. For the dd bechmark power usage, the file was written to the SSD.

• min idle : 2.4W
• avg idle : 3.7W
• max idle : 4.9W
• min during 7zip benchmark : 4.7 W
• avg during 7zip benchmark : 6.7 W
• max during 7zip benchmark : 8.2 W

Raspberry Pi 5

Work In Progress

HardKernel boards

The images for ODroid boards are build to be booted with HardKernel's Petitboot boot loader that is installed in SPI flash. If the board is brand new, you do have to modify its boot loader. If you deleted petitboot or replaced it with uboot you need to restore petitboot to boot these images. (check hardkernel forum for the procedure)

ODroid M1

Command to build image :

nix-shell -p nixos-generators --run "nixos-generate -c hardkernel/odroid-m1/default.nix --format-path ./nixin/sd-aarch64-kboot-installer-format.nix  --system aarch64-linux -o images/nixin-installer-odroid-m1"

Command to write image to sd card :

# replace /dev/sdX by the device of your sdcard
zstdcat ./images/nixin-installer-odroid-m1/sd-image/*.img.zst | sudo dd of=/dev/sdX bs=4M status=progress conv=fsync oflag=direct

Debug UART :

• UART is set to 1500000n8
• The debug UART port is right next to the sd card slot.

   _______________
  /              |
  |  1  2  3  4  |
  |______________|
             UART
          ___________________
          |     |           |
          |     |           |
          |     |  SD Card  |
          |     |           |
          |_____|___________|
  

• Port pinout :
  • Pin 1 - VCC
  • Pin 2 - TXD
  • Pin 3 - RXD
  • Pin 4 - GND
• It is using 3.3V TTL levels

Installing to SD Card

ToDo : write a script to automate this

Installing to EMMC module

ToDo : write a script to automate this

Installing to NVME SSD

ToDo : write a script to automate this

Installing to SATA SSD

ToDo : write a script to automate this

Installing to both SATA and NVME SSDs configured in RAID1

ToDo : write a script to automate this

Crypted root install

ToDo : add an option to the above scripts to crypt the filesystem

Benchmark

[nix-shell:~]$ 7z b

7-Zip [64] 17.05 : Copyright (c) 1999-2021 Igor Pavlov : 2017-08-28
p7zip Version 17.05 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,4 CPUs LE)

LE
CPU Freq: - - - 64000000 128000000 256000000 - - -

RAM size:    7414 MB,  # CPU hardware threads:   4
RAM usage:    882 MB,  # Benchmark threads:      4

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:       2968   342    845   2888  |      86183   389   1892   7353
23:       2866   352    830   2920  |      84216   389   1871   7287
24:       2824   364    833   3036  |      82137   390   1850   7211
25:       2752   375    838   3143  |      80490   394   1819   7163
----------------------------------  | ------------------------------
Avr:             358    837   2997  |              390   1858   7253
Tot:             374   1347   5125

[nix-shell:~]$ cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       334367 iterations per second for 256-bit key
PBKDF2-sha256     668734 iterations per second for 256-bit key
PBKDF2-sha512     356173 iterations per second for 256-bit key
PBKDF2-ripemd160  235741 iterations per second for 256-bit key
PBKDF2-whirlpool   98847 iterations per second for 256-bit key
argon2i       4 iterations, 299251 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id      4 iterations, 299251 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
#     Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b       456.8 MiB/s       522.0 MiB/s
    serpent-cbc        128b        41.3 MiB/s        44.9 MiB/s
    twofish-cbc        128b        63.1 MiB/s        67.7 MiB/s
        aes-cbc        256b       410.6 MiB/s       486.6 MiB/s
    serpent-cbc        256b        42.5 MiB/s        44.9 MiB/s
    twofish-cbc        256b        64.5 MiB/s        67.6 MiB/s
        aes-xts        256b       490.7 MiB/s       490.2 MiB/s
    serpent-xts        256b        41.9 MiB/s        45.5 MiB/s
    twofish-xts        256b        65.5 MiB/s        68.9 MiB/s
        aes-xts        512b       458.4 MiB/s       458.1 MiB/s
    serpent-xts        512b        43.8 MiB/s        45.5 MiB/s
    twofish-xts        512b        67.5 MiB/s        69.0 MiB/s

# SATA SSD
[root@cocotte:/mnt]# dd if=/dev/zero of=./test.random bs=4M count=1024 status=progress conv=fsync oflag=direct
4043309056 bytes (4.0 GB, 3.8 GiB) copied, 10 s, 404 MB/s4294967296 bytes (4.3 GB, 4.0 GiB) copied, 10.6654 s, 403 MB/s

1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 12.4913 s, 344 MB/s

[root@cocotte:/mnt]# echo 3 > /proc/sys/vm/drop_caches

[root@cocotte:/mnt]# dd if=./test.random of=/dev/null bs=4M count=1024 status=progress
4043309056 bytes (4.0 GB, 3.8 GiB) copied, 10 s, 404 MB/s
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 10.6264 s, 404 MB/s

# NVME SSD
[root@cocotte:~]# dd if=/dev/zero of=./test.random bs=4M count=1024 status=progress conv=fsync oflag=direct
3716153344 bytes (3.7 GB, 3.5 GiB) copied, 5 s, 743 MB/s4294967296 bytes (4.3 GB, 4.0 GiB) copied, 5.79092 s, 742 MB/s

1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 6.04984 s, 710 MB/s

[root@cocotte:~]# echo 3 > /proc/sys/vm/drop_caches

[root@cocotte:~]# dd if=./test.random of=/dev/null bs=4M count=1024 status=progress
4072669184 bytes (4.1 GB, 3.8 GiB) copied, 7 s, 582 MB/s
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 7.40545 s, 580 MB/s

Power usage

Measures done with nixos running on a 1TB NVME SSD, ethernet interface connected, no USB device connected, no sd card, no emmc module, no SATA drive. 8 GB RAM model

• min idle : 1.7W
• avg idle : 4.1W
• max idle : 6.5W
• min during 7zip benchmark : 3.1 W
• avg during 7zip benchmark : 6.7 W
• max during 7zip benchmark : 10.6 W

ODroid M1S

We do not have any ODroid M1s to make an installation image for it and test it. Contribution welcome

ODroid N2+

Work in progress

ODroid HC4

Work in progress

ODroid C4

We do not have any ODroid C4 to make an installation image for it and test it. But it should be fairly easy to adapt the HC4 configuration to this board. Contribution welcome

Orange Pi

Orabge Pi 5 Plus

Command to build image

nix build ./orangepi/orangepi-5plus#nixosConfigurations.nixos.sdImage --extra-experimental-features "nix-command flakes" -o ./images/nixin-installer-orangepi-5plus

Command to write image to sd card

# replace /dev/sdX by the device of your sdcard
zstdcat ./images/nixin-installer-orangepi-5plus/sd-image/*.img.zst | sudo dd of=/dev/sdX bs=4M status=progress conv=fsync oflag=direct

To install nixos to the NVME SSD you nedd to write UEFI firmware to the SPI flash :

1. Download the UEFI firmware from pre-build for the orange pi 5 plus from EDK2 latest release.
2. Boot the board with the installation image written to a sdcard
3. Then use dd to flash the UEFI firmware you downloaded to the SPI flash:

   sudo dd if=./xxx-UEFI-xxx.img of=/dev/mtdblock0
   

4. Reboot the board (taking the SD card out), and you should see the UEFI boot menu.

   NOTE: On orangePi5plus which have 2 HDMI output, the UEFI will only be displayed on the first HDMI output, be sure to plug your monitor to the middle HDMI socket

5. In the UEFI boot menu
   1. Enter [Device Manager] => [Rockchip Platform Configuration] => [ACPI / Device Tree]
   2. Change [Config Table Mode] to UEFI.
   3. Change [Support DTB override & overlays] to Enabled.

Then, when installing to the nvme SSD, base your configuration.nix on the sample in this repo.

Benchmark

[nix-shell:~]# 7z b

7-Zip [64] 17.05 : Copyright (c) 1999-2021 Igor Pavlov : 2017-08-28
p7zip Version 17.05 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs LE)

LE
CPU Freq: 64000000 - - - - - - - -

RAM size:   31718 MB,  # CPU hardware threads:   8
RAM usage:   1765 MB,  # Benchmark threads:      8

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:      17233   764   2195  16765  |     217197   679   2729  18526
23:      15659   727   2194  15955  |     211924   678   2704  18339
24:      15384   735   2250  16541  |     207594   679   2683  18220
25:      14674   764   2194  16755  |     202399   680   2648  18013
----------------------------------  | ------------------------------
Avr:             747   2208  16504  |              679   2691  18275
Tot:             713   2450  17389

[nix-shell:~]$ cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1      1161213 iterations per second for 256-bit key
PBKDF2-sha256    2340571 iterations per second for 256-bit key
PBKDF2-sha512     967321 iterations per second for 256-bit key
PBKDF2-ripemd160  618994 iterations per second for 256-bit key
PBKDF2-whirlpool  260321 iterations per second for 256-bit key
argon2i       4 iterations, 772035 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id      4 iterations, 759452 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
#     Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b       990.1 MiB/s      1728.9 MiB/s
    serpent-cbc        128b        66.6 MiB/s        76.5 MiB/s
    twofish-cbc        128b       104.3 MiB/s       113.9 MiB/s
        aes-cbc        256b       823.1 MiB/s      1446.4 MiB/s
    serpent-cbc        256b        69.0 MiB/s        76.5 MiB/s
    twofish-cbc        256b       108.2 MiB/s       113.9 MiB/s
        aes-xts        256b      1434.9 MiB/s      1432.2 MiB/s
    serpent-xts        256b        71.9 MiB/s        79.9 MiB/s
    twofish-xts        256b       119.2 MiB/s       123.7 MiB/s
        aes-xts        512b      1224.4 MiB/s      1223.3 MiB/s
    serpent-xts        512b        73.3 MiB/s        79.9 MiB/s
    twofish-xts        512b       121.1 MiB/s       123.6 MiB/s

[operator@orange:~]$ dd if=/dev/zero of=./test.random bs=4M count=1024 status=progress conv=fsync oflag=direct
3577741312 bytes (3.6 GB, 3.3 GiB) copied, 2 s, 1.8 GB/s4294967296 bytes (4.3 GB, 4.0 GiB) copied, 2.40462 s, 1.8 GB/s

1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 2.61636 s, 1.6 GB/s

[operator@orange:~]$ echo 3 > /proc/sys/vm/drop_caches
-bash: /proc/sys/vm/drop_caches: Permission denied

[operator@orange:~]$ dd if=./test.random of=/dev/null bs=4M count=1024 status=progress
3133145088 bytes (3.1 GB, 2.9 GiB) copied, 2 s, 1.6 GB/s
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 2.73825 s, 1.6 GB/s

Power usage

Measures done with nixos running on a 1TB NVME SSD, one ethernet interface connected, no USB device connected, no sd card, no emmc module. 32 GB RAM model

• min idle : 5.0 W
• avg idle : 5.2 W
• max idle : 6.3 W
• min during 7zip benchmark : 5.8 W
• avg during 7zip benchmark : 11.0 W
• max during 7zip benchmark : 13.4 W

Note: the idle power usage seems a bit higher than it should be on this board.

RISC-V boards

StarFive

Vision Five 2

Work in progress

Command to build image

nix build ./starfive/visionfive2# --extra-experimental-features "nix-command flakes" -o ./images/nixin-installer-visionfive2

Command to write image to sd card

# replace /dev/sdX by the device of your sdcard
cat images/nixin-installer-visionfive2/sd-image/*.img | sudo dd of=/dev/sdX bs=4M status=progress conv=fsync oflag=direct

Notes

To boot the installation image, the boot mode needs to be set to SDIO (RGPIO_0 = 1(H), RGPIO_1 = O(L))

To boot from NVME the boot mode needs to be set to QSPI. It is recommended to update the firmware the board in QSPI flash to the latest version. Refer to the VisionFive 2 documention for how to do it.

If the boot fails with an "Unknown command" error of the bootloader, try to reset the bootloader environment like this :

StarFive # env default -a -f
## Resetting to default environment
StarFive # env save
Saving Environment to SPIFlash... Erasing SPI flash...Writing to SPI flash...done
OK

In case of out of memory error during nixos-install/nixos-rebuild on the 4GB model, you can add temporary swap like this :

fallocate -l 8G /tmp/swap; mkswap /tmp/swap; swapon /tmp/swap

7zip benchmark (4GB RAM model)

root@starfive:~# 7z b

7-Zip 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,4 CPUs LE)

LE
CPU Freq: 64000000 64000000 64000000 - - - 512000000 1024000000 -

RAM size:    3874 MB,  # CPU hardware threads:   4
RAM usage:    882 MB,  # Benchmark threads:      4

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:       2638   334    769   2566  |      66701   399   1425   5691
23:       2581   343    767   2630  |      65110   399   1412   5634
24:       2539   353    775   2730  |      63749   400   1401   5596
25:       2117   308    784   2417  |      61870   399   1379   5506
----------------------------------  | ------------------------------
Avr:             334    773   2586  |              399   1404   5607
Tot:             367   1089   4096

[root@glove:~]# dd if=/dev/zero of=./test.random bs=4M count=1024 status=progress conv=fsync oflag=direct
4148166656 bytes (4.1 GB, 3.9 GiB) copied, 23 s, 180 MB/s4294967296 bytes (4.3 GB, 4.0 GiB) copied, 23.8205 s, 180 MB/s

1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 23.9832 s, 179 MB/s

[root@glove:~]# echo 3 > /proc/sys/vm/drop_caches

[root@glove:~]# dd if=./test.random of=/dev/null bs=4M count=1024 status=progress
4156555264 bytes (4.2 GB, 3.9 GiB) copied, 23 s, 181 MB/s
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 23.7604 s, 181 MB/s

Power usage

Measures done with nixos running on a 1TB NVME SSD, one ethernet interface connected, no USB device connected, no sd card, no emmc module. 32 GB RAM model

• min idle : 3.0 W
• avg idle : 3.5 W
• max idle : 3.9 W
• min during 7zip benchmark : 4.0 W
• avg during 7zip benchmark : 5.2 W
• max during 7zip benchmark : 6.0 W

Virtualization

Proxmox VM image

An image for deploying a KVM virtual machine on a Proxmox VE server ca be built using :

nix-shell -p nixos-generators --run "nixos-generate -c virtualization/proxmox-vm/default.nix -f proxmox -o images/nixin-installer-proxmox-vm"

Then to deploy the image you can do :

• upload the resulting file images/nixin-installer-proxmox-vm/*.vma.zst to the backup dump directory of the proxmox server
• in the web interface, navigate to the backup folder, select the uploaded file and use the restore feature of proxmox to create a VM from it.
• before starting the VM :
  • rename the VM to the name you like
  • go to its hardware properties, and change the network bridge that is linked to the VM network interface.
  • then go to the cloud init section, set the address and gateway of the network interface and click on regenerate image
• start the VM, connect to it using ssh and check that it has access to internet.
• if everything is OK, before further configuration of the VM, you can stop the VM, clone it and transform the clone into a template for easier creation of more VMs later on.
• If you stopped it at th eprevious step, rerestart the VM, connect back to it with ssh, and run nixos-generate-config to generate the hardware-configuration.nix file that will correctly mount the VM disk.
• discard the generated configuration.nix file and build your own, taking as exemple the sample one in virtualization/proxmox-lxc and use nixos-rebuild switch to activate it.
• you can also have a look in the nixin-krops git repository for a sample configurations that can be remotely deployed usign krops instead of the previous step. Look at the configuration of the ldl-proxy-mpl-1 server and to the krops.nix file.

Proxmox LXC template

Deploying containers from LXC templates is not straightforward. It is a bit of a hit and miss as we do not have yet devised any foolproof nixos configuration and procedure to do it. Prefer using virtual machines as described in the previous section if you want to avoid headackes. If you like chalenges, do continue and please contribute your experience to the NixiN project by sending a pull with better procedure/configurations than what we currently have.

^·^

An template for deploying LXC containers to a Proxmox VE server can be built using :

nix-shell -p nixos-generators --run "nixos-generate -c virtualization/proxmox/default.nix -f proxmox-lxc -o images/nixin-installer-proxmox-lxc"

Then upload the resulting file images/nixin-installer-proxmox-lxc/tarball/nixos-system-x86_64-linux.tar.xz to the proxmox server LXC templates directory and you can create containers from it. The image is configured to not manage its network configuration itself and let the proxmox server defining it. See the sample configuration.nix in virtualization/proxmox-lxc for how to maintain this behavior when doing nixos-rebuild in the container.