Featured image of post Stop the senseless killing

Stop the senseless killing

In over 22 years of working with Linux systems, I have often seen people use excessive force to kill various computer programs, causing unnecessary suffering.

A typical example would be:

$ killall -9 mysqld

Please stop using this command. Both killall and the parameter -9 are harmful when dealing with programs that store data, like the MySQL or MariaDB database, and better alternatives exist.

Avoid killing in vain

The 9 means signal 9, which is SIGKILL. All standard Linux signals can easily be listed with:

$  kill -L

The Linux kernel sends most of these signals to the program itself, and we expect programs to have code to handle various signals. One exception is signal number 9 (SIGKILL). We use the SIGKILL signal to terminate the program immediately. The program cannot handle block or ignore it. Therefore, it is always fatal. If kill is issued without parameters, it will default to signal 15 (SIGTERM) and the Linux kernel will request the running process to terminate itself and clean file pointers and other resources it was using. Thus, always prefer SIGTERM and only use SIGKILL as a last resort.

Violently killing programs that deal with data storage might cause writing into a file to abruptly stop halfway, leading to inconsistent data or even unusable corrupted files. For example, even if the program is not a database or similar, senseless immediate killing leads to everything dropping mid-flight, potentially causing unnecessary harm to other programs communicating over the same network. In some cases, albeit rare, poorly terminated programs may turn into zombie processes, showing up in the Linux process listing as <defunct>. If that happens, the only way to clean them is to restart the entire operating system. So, please, do not kill without trying other means first.

Don’t be senseless

Let’s dive into the first part of the command in the opening paragraph, which is also wrong.

The command killall has been in Unix and all its descendants, such as Linux, since 1983. It does the job. However, we also have modern tools, which I prefer, like pkill from 1998.

The program pkill is superior because it has the parameter -e to display what is killed (or terminated). Those who still use killall but don’t want to be senseless also use variations of ps ax | grep <processname> to see which processes are running before and after the killing.

Compare the clarity in these identical examples:

$ killall mysqld
$ killall mysqld
mysqld: no process found

The killall command remains silent if it did something and only reports errors. In contrast, pkill reports which process it terminated and is only silent if it didn’t do anything.

pkill -e mysqld
mysqld_safe killed (pid 3911)
mysqld killed (pid 4011)
$ pkill -e mysqld

The bonus of being sensible

A bonus for system administrators using pkill is they could learn something valuable by observing the effects of their commands. For example, a database sysadmin could encounter the following:

$ pkill -9 -e mariadbd
mariadbd killed (pid 5281)
$ pkill -9 -e mariadbd
mariadbd killed (pid 5383)
$ pkill -9 -e mariadbd
mariadbd killed (pid 5412)
$ pkill -e mariadbd
mariadbd killed (pid 5497)
$ pkill -e mariadbd

With MariaDB, if the main server abruptly dies (as with signal 9/SIGKILL), the wrapper mysqld_safe (if used) will detect that and automatically restart the server because it assumes it died during a crash and knows getting it back up and running is desirable. However, when the proper signal 15/SIGTERM is used (default in pkill when no signal is defined), the mariadbd process intentionally shuts itself down. The wrapper respects that and understands that automatically restarting it would not make sense.

The procps package

The command pkill is part of the procps suite. You can find it by default on most Linux systems, but if you don’t have it, run apt install procps or yum install procps to get it. The suite also includes the command pgrep, which replaces manual ps ax | grep <processname> invocations, as mentioned above.

An example of pgrep in action:

$ pgrep -af mysqld
5577 /bin/sh /usr/bin/mysqld_safe
5676 /usr/sbin/mariadbd --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-log-error --pid-file=/run/mysqld/mysqld.pid --socket=/run/mysqld/mysqld.sock
5677 logger -t mysqld -p daemon error

To see the complete list of options, run pkill --help and pgrep --help or man pkill to read the man page. I recommend all system administrators make using --help and man <command> from the command line as a general habit to act more sensibly in all command-line actions.

Most importantly, don’t use signal 9 (SIGKILL) on databases other than as the last resort. Please avoid any senseless killings!

Hey if you enjoyed reading the post, please share it on social media and subscribe for notifications about new posts!

comments powered by Disqus