SBC Performance Test Kit

Testing procedure used by the DistriLab for benchmarking the performances of SBC boards

Introduction

These benchmarking are done in the context of choosing a hardware platform for self hosting open source web applications. The testing procedure mainly focusses on energy consuption and CPU/memory/disk speed.

The full test procedure during which the energy consumption is measured takes one hour. During that period the CPU is heavily stressed for around 40 minutes and is mostly idle during the remaining 20 minutes.

By default the disk speed is measured for sequential reads, sequential writes and a mix of random reads and writes with a 2 to 1 ratio (2 reads for 1 write) with a block size of 1024 KiB over 512 files of 128 MiB each (64 GiB total)

The total disk space used is adapted depending on the target board specs. The goal is to use a disk space that is at least 2 times as large as the system memory size to avoid having the results biased by the Linux file cache.

Credits

The performance measurements and system stressing are mainly done thanks to sysbench

hdparm is also used to get disk information and to measure raw disk read speed

lm_sensors is used to measure CPU temperature

System information is gathered with smartctl, though the Linux /proc filesystem or using some other small command line utilities like lsblk, uname, grep, cut, ...

Testing procedure

1. Plug the power supply of the target machine through a Watt meter
2. Install a debian GNU/Linux based distribution on the target machine and make sure to upgrade to the latest version (apt update && apt dist-upgrade && reboot)
3. Login on the physical console or though ssh
4. Install git : apt install git
5. Clone this git repository : cd ~ && git clone https://gitlab.com/distrilab/sbc-performance-test-kit.git
6. Install dependencies : sudo ~/sbc-performance-test-kit/bin/install-dependencies.sh
7. Adjust the number of threads and/or the disk space used by the test for boards that have less or more than 4 CPU cores or less than 64 GB disk free space or more than 32 GB of RAM (some documentation needs to be written on this step)
8. Take note of the power consumption of the taget machine in idle state on the Watt meter
9. Reset the energy consumption of the watt meter
10. Run the full benchmarking script : ~/sbc-performance-test-kit/bin/run-full-benchmark.sh some-test-name
11. Around 15 minutes after the start of the test, during the warm-up step, make note of the max power consumption
12. Exactly 60 minutes after the beginning of the test, that should have ended around the 50 minutes mark, take note of the total energy consumption on the Watt meter
13. Make a backup copy of the log file generated by the test in the current directory

If there are additional drives to test on the same board the quick test script can be used :

1. Change directory to the mount point of another disk that should be tested : cd /some/where
2. Run the quick benchmarking script : ~/sbc-performance-test-kit/bin/run-quick-benchmark.sh some-test-name
3. Make a backup copy of the log file generated by the test in the current directory

Sample test results and how to interpret them

To Be Done