The main test system in the MariaDB open source database project is the mariadb-test-run script (inherited from mysql-test-run). It is easy to run to and does not require you to compile any source code.
While writing MTR tests is relevant only for MariaDB developers, knowing how to run MTR is useful for any database administrator running MariaDB, as it is a quick way to validate that MariaDB can run correctly on your hardware and operating system version.
# TL;DR for Debian/Ubuntu users
Run the full 6000+ test suite as current user in temporary directory with multiple workers in parallel and with detailed logging on failures, typically taking over 30 minutes on modern laptop:
apt install -y mariadb-test mariadb-backup mariadb-plugin-* patch gdb cd /usr/share/mysql/mysql-test export MTR_PRINT_CORE=detailed ./mtr --force --parallel=auto --vardir=$(mktemp -d) \ --skip-test-list=unstable-tests.amd64 --big-testFor full details, read the whole article.
# Install ‘mariadb-test’ package in Debian/Ubuntu
To avoid polluting your actual system with new packages, it is convenient to run MTR in a throwaway container. Start one with some RAM-memory-backed disk allocated with --shm-size=1G so running MTR with --mem later on is possible:
docker run --interactive --tty --rm --shm-size=1G debian:sid bashThis example uses Docker, but the principle is the same with any Linux container tool, such as Podman.
Next, install the MariaDB test suite package. This will also pull in the MariaDB server and all the necessary dependencies. Additionally, also install the GNU Debugger (gdb) for automatic stack traces and patch.
apt update
apt install -y mariadb-test mariadb-backup mariadb-plugin-* patch gdbThe mariadb-test-run is not intended to be run as root and will skip some tests if run as root. Therefore, create a new user inside the container and grant it permissions to the test directory:
adduser --disabled-password mariadb-test-runner
chown -R mariadb-test-runner /usr/share/mysql/mysql-testAt minimum, the test runner user needs to be able to write to the path /usr/share/mysql/mysql-test/var (unless some other path is defined with --vardir and --tmpdir), but some tests also run patch to modify the test files on-the-fly, so just grant permissions to the whole test directory. As this is a throwaway container anyway, there is no need to be prudent with the permissions.
Next, switch to the test user and test directory and start the run with default settings:
$ su - mariadb-test-runner
$ cd /usr/share/mysql/mysql-test
$ ./mariadb-test-run
Logging: ./mariadb-test-run
VS config:
vardir: /usr/share/mysql/mysql-test/var
Creating var directory '/usr/share/mysql/mysql-test/var'...
Checking supported features...
MariaDB Version 10.11.2-MariaDB-1
- SSL connections supported
- binaries built with wsrep patch
Using suites: main-,archive-,atomic-,binlog-,binlog_encryption-,client-,csv-,compat/oracle-,compat/mssql-,compat/maxdb-,encryption-,federated-,funcs_1-,funcs_2-,gcol-,handler-,heap-,innodb-,innodb_fts-,innodb_gis-,innodb_i_s-,innodb_zip-,json-,maria-,mariabackup-,multi_source-,optimizer_unfixed_bugs-,parts-,perfschema-,plugins-,roles-,rpl-,stress-,sys_vars-,sql_sequence-,unit-,vcol-,versioning-,period-,sysschema-,disks,func_test,metadata_lock_info,query_response_time,sequence,sql_discovery,type_inet,type_uuid,user_variables
Collecting tests...
...
main.subselect_innodb 'innodb' w4 [ pass ] 3359
main.subselect_sj2 'innodb' w1 [ pass ] 2737
main.subselect_sj2_jcl6 'innodb' w3 [ pass ] 2933
main.parser_bug21114_innodb 'innodb' w8 [ pass ] 15364
innodb_gis.rtree_search 'innodb' w5 [ pass ] 52379
--------------------------------------------------------------------------
The servers were restarted 1736 times
Spent 8527.863 of 1659 seconds executing testcases
Completed: All 5131 tests were successful.
995 tests were skipped, 280 by the test itself.# Defining what tests to run
If no test suite is selected, MTR will run about 6000 tests, which on my laptop takes about 30 minutes. If MTR is started with additional --big-test parameter, it will run additional tests that are resource intensive and consume, for example, a lot of RAM memory and take a long time to run, totalling to a test run that has 6100 tests and takes 43 minutes to complete (on my laptop). To only run big tests, use --big --big.
If there is a need to limit the scope, such as in build systems that want to validate that the built binary works without running all tests, typically --suite=main --skip-rpl is used. This results in about 1000 tests being run, which on my laptop takes about 3½ minutes.
Even when running without any limitations on what tests are run, many tests have code that makes them opt out automatically based on some condition being missing, and on my laptop about 1000 tests end up being skipped. Some examples:
main.connect2 [ skipped ] Requires debug build
main.mysql_client_test_comp [ skipped ] No IPv6
main.connect-abstract [ skipped ] Need Linux
main.grant_cache_ps_prot [ skipped ] Need ps-protocol
main.fix_priv_tables [ skipped ] Test need MYSQL_FIX_PRIVILEGE_TABLES
main.no-threads [ skipped ] Test requires: 'one_thread_per_connection'
main.udf_skip_grants [ skipped ] Need udf example
main.lowercase_mixed_tmpdir_innodb [ skipped ] Test requires: 'lowercase2'
main.innodb_load_xa [ skipped ] Need InnoDB plugin
mariabackup.alter_copy_excluded [ skipped ] No mariabackup
binlog.binlog_expire_warnings [ skipped ] Test needs --big-test
encryption.innodb-spatial-index [ skipped ] requires patch executable
plugins.pam_cleartext [ skipped ] Not run as user owning auth_pam_tool_dir
rpl.rpl_gtid_mdev4474 'innodb,row' [ skipped ] Neither MIXED nor STATEMENT binlog formatIf you are interested in one particular test, just give it as the last argument:
$ ./mariadb-test-run main.connect
Logging: ./mariadb-test-run main.connect
Creating var directory '/usr/share/mysql/mysql-test/var'...
Checking supported features...
MariaDB Version 10.11.2-MariaDB-1
- SSL connections supported
- binaries built with wsrep patch
Collecting tests...
Installing system database...
==============================================================================
TEST RESULT TIME (ms) or COMMENT
--------------------------------------------------------------------------
main.connect [ pass ] 14206
--------------------------------------------------------------------------
The servers were restarted 0 times
Spent 14.206 of 20 seconds executing testcases
Completed: All 1 tests were successful.# Optimize the MTR run for speed
There are three parameters can greatly improve how quickly MTR runs. The primary one is --parallel=auto, which will run MTR in parallel with as many workers as there are CPUs (by default MTR runs with just one worker). On my laptop, going from one MTR worker to 8 in parallel reduced the total run time for the main suite from 17 to about 3 minutes.
Another parameter is --fast, which will make the MTR kill all server processes violently without waiting for them to gracefully shutdown. The test run restarts the MariaDB server hundreds of times, so saving half a second on every shutdown results in the main suite completing 20 seconds faster.
The parameter --mem instructs MTR to make the directory var/ a symbolic link to a subdirectory on the shared memory device (/dev/shm). This works if the container is started with --shm-size and has at least 350MB of space on the ramdisk. On my laptop, this further reduced the main test suite duration down to 2½ minutes.
# Optimize the MTR run for logging
When running a single test, one might use --verbose as an additional argument to see the commands that run in the test. It is also possible to define --verbose --verbose twice, but that makes the test run so verbose that is it unusable.
When running a suite of tests, you only want to have extra output visible if the test fails. If the server fails to start and a particular test doesn’t run at all, using --verbose-restart might be beneficial. If running the test in an automated system, one might want to save results in a JUnit-compatible XML file that can, for example, be rendered by GitLab CI.
This is the command that several CI systems run MTR with:
export MTR_PRINT_CORE=detailed
eatmydata perl -I. ./mariadb-test-run \
--force --testcase-timeout=120 --suite-timeout=540 --retry=3 \
--verbose-restart --max-save-core=1 --max-save-datadir=1 \
--parallel=auto --skip-rpl --suite=main \
--xml-report=mariadb-test-run-junit.xmlThe command above also showcases use of eatmydata, which makes fsync and similar system calls to skip memory-to-disk guarantees, but in my testing with MTR, it didn’t affect speed.
# Running more tests for MariaDB
The above summarizes everything one typically needs to know for running the mariadb-test-run.
For a DBA, there are also several other tools that ship with MariaDB, which can help with testing and validating ones own environment, such as mariadb-stress-test and mariadb-slap.
# Writing MTR tests for MariaDB
If you want to write a new test or fix an existing test, there are many more additional parameters to learn, such as --record and --gcov. The official contribution docs at mariadb.org list some of the most useful MTR parameters. The mariadb.com knowledge base article lists them all. As with all commands in Linux, there is also the main page for mariadb-test-run.
The structure of the tests is actually quite easy, and requires no C/C++ skills to write. Each test file (suffix .test) consists mainly of SQL code which is executed by mariadb-test-run (MTR), with the output compared (with diff) to the corresponding file with the expected output in text format (suffix .result).
In my development workflow, writing a test and running MTR might look something like this:
If you want to contribute to the MariaDB open source project, extending the test coverage is a great place to start. To scratch your own itch, think about a MariaDB bug you encountered while using it, and think if that can be reproduced as a test and submitted upstream so that it becomes part of the body of tests and thus easy to catch if it ever regresses again.
To learn more about writing mariadb-test-run tests, read the test case authoring guide at mariadb.org.