While writing MTR tests is relevant only for MariaDB developers, knowing how to run MTR is useful for any database administrator running MariaDB as it is a quick way to validate that MariaDB can run correctly on your hardware and operating system version.
TL;DR for Debian/Ubuntu users
Run the full 6000+ test suite as current user in temporary directory with multiple workers in parallel and with detailed logging on failures, typically taking over 30 minutes on modern laptop:
apt install -y mariadb-test mariadb-backup mariadb-plugin-* patch gdb cd /usr/share/mysql/mysql-test export MTR_PRINT_CORE=detailed ./mtr --force --parallel=auto --vardir=$(mktemp -d) \ --skip-test-list=unstable-tests.amd64 --big-test
For full details, read the whole article.
Install ‘mariadb-test’ package in Debian/Ubuntu
To avoid polluting your actual system with new packages it is convenient to run MTR in a throwaway container. Start one with some RAM memory backed disk allocated with
--shm-size=1G so running MTR with
--mem later on is possible:
docker run --interactive --tty --rm --shm-size=1G debian:sid bash
Next install the MariaDB test suite package. This will pull in also the MariaDB server and all the necessary dependencies. Additionally also install gdb for automatic stack traces and patch.
apt update apt install -y mariadb-test mariadb-backup mariadb-plugin-* patch gdb
The mariadb-test-run is not intended to be run as root and will skip some tests if run as root. Therefore create a new user inside the container and grant if permissions to the test directory:
adduser --disabled-password mariadb-test-runner chown -R mariadb-test-runner /usr/share/mysql/mysql-test
At minimum the test runner user needs to be able to write to the path
/usr/share/mysql/mysql-test/var (unless some other path is defined with
--tmpdir) but some tests also run
patch to modify the test files on-the-fly, so just grant permissions to the whole test directory. As this is a throwaway container anyway there is no need to be prudent with the permissions.
Next switch to the test user and test directory and start the run with default settings:
$ su - mariadb-test-runner $ cd /usr/share/mysql/mysql-test $ ./mariadb-test-run Logging: ./mariadb-test-run VS config: vardir: /usr/share/mysql/mysql-test/var Creating var directory '/usr/share/mysql/mysql-test/var'... Checking supported features... MariaDB Version 10.11.2-MariaDB-1 - SSL connections supported - binaries built with wsrep patch Using suites: main-,archive-,atomic-,binlog-,binlog_encryption-,client-,csv-,compat/oracle-,compat/mssql-,compat/maxdb-,encryption-,federated-,funcs_1-,funcs_2-,gcol-,handler-,heap-,innodb-,innodb_fts-,innodb_gis-,innodb_i_s-,innodb_zip-,json-,maria-,mariabackup-,multi_source-,optimizer_unfixed_bugs-,parts-,perfschema-,plugins-,roles-,rpl-,stress-,sys_vars-,sql_sequence-,unit-,vcol-,versioning-,period-,sysschema-,disks,func_test,metadata_lock_info,query_response_time,sequence,sql_discovery,type_inet,type_uuid,user_variables Collecting tests... ... main.subselect_innodb 'innodb' w4 [ pass ] 3359 main.subselect_sj2 'innodb' w1 [ pass ] 2737 main.subselect_sj2_jcl6 'innodb' w3 [ pass ] 2933 main.parser_bug21114_innodb 'innodb' w8 [ pass ] 15364 innodb_gis.rtree_search 'innodb' w5 [ pass ] 52379 -------------------------------------------------------------------------- The servers were restarted 1736 times Spent 8527.863 of 1659 seconds executing testcases Completed: All 5131 tests were successful. 995 tests were skipped, 280 by the test itself.
Defining what tests to run
If no test suite is selected MTR will run about 6000 tests, which on my laptop takes about 30 minutes. If MTR is stared with additional
--big-test parameter, it will run additional tests that are resource intensive and consume for example a lot of RAM memory and take a long time to run, totaling to a test run that has 6100 tests and takes 43 minutes to complete (on my laptop). To only run big tests use
If there is a need to limit the scope, such as in build systems that want to validate that the built binary works without running all tests, typically
--suite=main --skip-rpl is used. This results in about 1000 tests being run, which on my laptop takes about 3½ minutes.
Even when running without any limitations on what tests are run, many tests have code that makes them opt out automatically based on some condition missing and on laptop about 1000 tests end up being skipped. Some examples:
main.connect2 [ skipped ] Requires debug build main.mysql_client_test_comp [ skipped ] No IPv6 main.connect-abstract [ skipped ] Need Linux main.grant_cache_ps_prot [ skipped ] Need ps-protocol main.fix_priv_tables [ skipped ] Test need MYSQL_FIX_PRIVILEGE_TABLES main.no-threads [ skipped ] Test requires: 'one_thread_per_connection' main.udf_skip_grants [ skipped ] Need udf example main.lowercase_mixed_tmpdir_innodb [ skipped ] Test requires: 'lowercase2' main.innodb_load_xa [ skipped ] Need InnoDB plugin mariabackup.alter_copy_excluded [ skipped ] No mariabackup binlog.binlog_expire_warnings [ skipped ] Test needs --big-test encryption.innodb-spatial-index [ skipped ] requires patch executable plugins.pam_cleartext [ skipped ] Not run as user owning auth_pam_tool_dir rpl.rpl_gtid_mdev4474 'innodb,row' [ skipped ] Neither MIXED nor STATEMENT binlog format
If you are interested in one particular test, just give it as the last argument:
$ ./mariadb-test-run main.connect Logging: ./mariadb-test-run main.connect Creating var directory '/usr/share/mysql/mysql-test/var'... Checking supported features... MariaDB Version 10.11.2-MariaDB-1 - SSL connections supported - binaries built with wsrep patch Collecting tests... Installing system database... ============================================================================== TEST RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- main.connect [ pass ] 14206 -------------------------------------------------------------------------- The servers were restarted 0 times Spent 14.206 of 20 seconds executing testcases Completed: All 1 tests were successful.
Optimize the MTR run for speed
There are three parameters can can greatly improve how quickly MTR runs. The primary one is
--parallel=auto which will run MTR in parallel with as many workers as there are CPUs (by default MTR runs with just one worker). On my laptop going from one MTR worker to 8 in parallel reduced the total run time for the main suite from 17 to about 3 minutes.
Another parameter is
--fast, which will make the MTR kill all server processes violently without waiting for them to gracefully shutdown. The test run restarts the MariaDB server hundreds of times, so saving half a second on every shutdown results in the main suite completing 20 seconds faster.
--mem instructs MTR to make the directory
var/ a symbolic link to a subdirectory on the shared memory device (
/dev/shm). This works if the container was started with
--shm-size and has at least 350MB of space on the ramdisk. On my laptop this further reduced the main test suite duration down to 2½ minutes.
Optimize the MTR run for logging
When running a single test one might use
--verbose as an additional argument to see the commands that ran in the test. It is also possible to define
--verbose --verbose twice, but that makes the test run so verbose that is it unusable.
When running a suite of tests, you only want to have extra output visible if the test failed. If the server failed to start and a particular test didn’t run at all, using
--verbose-restart might be beneficial. If running the test in an automated system one might want to save results in a JUnit compatible XML file that can for example be rendered by Gitlab CI.
This is the command several CI systems run MTR with:
export MTR_PRINT_CORE=detailed eatmydata perl -I. ./mariadb-test-run \ --force --testcase-timeout=120 --suite-timeout=540 --retry=3 \ --verbose-restart --max-save-core=1 --max-save-datadir=1 \ --parallel=auto --skip-rpl --suite=main \ --xml-report=mariadb-test-run-junit.xml
Running more tests for MariaDB
The above summarizes everything one typically needs to know for running the mariadb-test-run.
Writing MTR tests for MariaDB
If you want to write a new test or fix an existing test, there are many more additional parameters to learn, such as
--gcov. The official contribution docs at mariadb.org lists some of the most useful MTR parameters to be familiar. The mariadb.com knowledge base article lists them all as with all commands in Linux, there is also the man page for mariadb-test-run.
The structure of the tests is actually quite easy, and requires no C/C++ skills to write. Each test file (suffix
.test) consists mainly of SQL code which is executed by
mariadb-test-run (MTR) and output compared (with diff) to the corresponding file with the expected output in text format (suffix
In my development workflow writing a test and running MTR might look something like this:
If you want to contribute to the MariaDB open source project, extending the test coverage is a great place to start. To scratch your own itch, think about a MariaDB bug you encountered while using it, and think if that can be reproduced as a test and submitted upstream so that it becomes part of the body of tests and thus easy to catch if it ever regresses again.
To learn more about writing mariadb-test-run tests, read the test case authoring guide at mariadb.org.