Testing procedure used by the DistriLab for benchmarking the performances of SBC boards
## Introduction
These benchmarking are done in the context of choosing a hardware platform for self hosting open source web applications.
The testing procedure mainly focusses on energy consuption and CPU/memory/disk speed.
The full test procedure during which the energy consumption is measured takes one hour.
During that period the CPU is heavily stressed for around 40 minutes and is mostly idle during the remaining 20 minutes.
By default the disk speed is measured for sequential reads, sequential writes and a mix of random reads and writes with a 2 to 1 ratio (2 reads for 1 write) with a block size of 1024 KiB over 512 files of 128 MiB each (64 GiB total)
The total disk space used is adapted depending on the target board specs. The goal is to use a disk space that is at least 2 times as large as the system memory size to avoid having the results biased by the Linux file cache.
## Credits
The performance measurements and system stressing are mainly done thanks to [sysbench](https://github.com/akopytov/sysbench)
[hdparm](https://sourceforge.net/projects/hdparm/) is also used to get disk information and to measure raw disk read speed
[lm_sensors](https://github.com/lm-sensors/lm-sensors) is used to measure CPU temperature
System information is gathered with [smartctl](https://www.smartmontools.org/), though the Linux /proc filesystem or using some other small command line utilities like lsblk, uname, grep, cut, ...
## Testing procedure
1. Plug the power supply of the target machine through a Watt meter
1. Install a debian GNU/Linux based distribution on the target machine and make sure to upgrade to the latest version (`apt update && apt dist-upgrade && reboot`)
1. Adjust the number of threads and/or the disk space used by the test for boards that have less or more than 4 CPU cores or less than 64 GB disk free space or more than 32 GB of RAM (some documentation needs to be written on this step)
1. Take note of the power consumption of the taget machine in idle state on the Watt meter
1. Reset the energy consumption of the watt meter
1. Run the full benchmarking script : `~/sbc-performance-test-kit/bin/run-full-benchmark.sh some-test-name`
1. Around 15 minutes after the start of the test, during the warm-up step, make note of the max power consumption
1. Exactly 60 minutes after the beginning of the test, that should have ended around the 50 minutes mark, take note of the total energy consumption on the Watt meter
1. Make a backup copy of the log file generated by the test in the current directory
If there are additional drives to test on the same board the quick test script can be used :
1. Change directory to the mount point of another disk that should be tested : `cd /some/where`
1. Run the quick benchmarking script : `~/sbc-performance-test-kit/bin/run-quick-benchmark.sh some-test-name`
1. Make a backup copy of the log file generated by the test in the current directory
