SBC Performance Test Kit

Testing procedure used by the DistriLab for benchmarking the performances of SBC boards

Introduction

These benchmarking are done in the context of choosing a hardware platform for self hosting open source web applications. The testing procedure mainly focusses on energy consuption and CPU/memory/disk speed.

The full test procedure during which the energy consumption is measured takes one hour. During that period the CPU is heavily stressed for around 40 minutes, the system is idle for 10 minutes and the remaining 10 minutes are taken by the CPU/Memory/Disk tests (these tests can actually last less or more than 10 minutes depending on the performance of the machine being tested but the nergy consumption is done on the 60 first minutes after the styart of the test).

By default the disk speed is measured for sequential reads, sequential writes and a mix of random reads and writes with a 2 to 1 ratio (2 reads for 1 write) with a block size of 1024 KiB over 512 files of 128 MiB each (64 GiB total) But the total disk space that is used is adapted depending on the target board specs. The goal is to use a disk space that is at least 2 times larger than the system memory size to avoid having the results biased by the system file cache.

This is a work in progress. Do not hesitate to propose your improvements by creating issues or pull requests on this gitlab project.

Credits

The performance measurements and system stressing are mainly done thanks to sysbench

hdparm is also used to get disk information and to measure raw read disk speed

lm_sensors is used to measure CPU temperature

System information is gathered with smartctl, though the Linux /proc filesystem or using some other small command line utilities like lsblk, uname, grep, cut, ...

The script used to automate the tests is run with bash on Debian GNU/Linux or a derivative distribution.

Testing procedure

1. Plug the power supply of the target machine through a Watt meter
2. Install a debian GNU/Linux based distribution on the target machine and make sure to upgrade to the latest version (apt update && apt dist-upgrade && reboot)
3. Login on the physical console or though ssh
4. Install git : apt install git
5. Clone this git repository : cd ~ && git clone https://gitlab.com/distrilab/sbc-performance-test-kit.git
6. Install dependencies : sudo ~/sbc-performance-test-kit/bin/install-dependencies.sh
7. Adjust the number of threads and/or the disk space used by the test for boards that have less or more than 4 CPU cores or less than 64 GB disk free space or more than 32 GB of RAM (some documentation needs to be written on this step)
8. Take note of the power consumption of the taget machine in idle state on the Watt meter
9. Reset the energy consumption of the watt meter
10. Run the full benchmarking script : ~/sbc-performance-test-kit/bin/run-full-benchmark.sh some-test-name
11. Around 15 minutes after the start of the test, during the warm-up step, make note of the max power consumption
12. Exactly 60 minutes after the beginning of the test, that should have ended around the 50 minutes mark, take note of the total energy consumption on the Watt meter
13. Make a backup copy of the log file generated by the test in the current directory

If there are additional drives to test on the same board the quick test script can be used :

1. Change directory to the mount point of another disk that should be tested : cd /some/where
2. Run the quick benchmarking script : ~/sbc-performance-test-kit/bin/run-quick-benchmark.sh some-test-name
3. Make a backup copy of the log file generated by the test in the current directory

Sample test results and how to interpret them

To Be Done