The Linux kernel can spawn processes on its own

2022-06-11

While investigating the aftermath of a potential security breach, I came across a somewhat “odd” process looking like this:

root@ubuntu2004:~# ls -al /proc/1026/exe
lrwxrwxrwx 1 root root 0 Jun 11 05:26 /proc/1026/exe -> '/ (deleted)'

This virtual symlink normally points to the binary being run, for example:

root@ubuntu2004:~# ls -al /proc/self/exe
lrwxrwxrwx 1 root root 0 Jun 11 05:27 /proc/self/exe -> /usr/bin/ls

A common pattern of malicious code is to start a binary and then delete it from disk to avoid being picked up by simple antivirus software, which leads to output like this:

$ ls -al /proc/5194/exe
lrwxrwxrwx 1 void users 0 Jun 11 07:29 /proc/5194/exe -> '/tmp/tmp/bla (deleted)'

So, what we often do is to search for processes whose exe symlink points to something that no longer exists – or rather, something that says “deleted”. As a result, the first process shown above (PID 1026) turned up during my search.

It doesn’t really fit the pattern, though: exe points to a directory. Was this some kind of trickery that I didn’t know about?

Long story short, the process in question was started directly by the kernel itself – it was not fork()ed nor posix_spawn()ed by some other existing process, thus breaking a bunch of assumptions in my mind. Yes, it’s not a big surprise that “the kernel can spawn processes”, but the vast majority of them are children of some other processes that already ran.

What I saw, was bpfilter_umh.

The basic idea is that this kernel module creates a “usermode driver”, i.e. an “ordinary” process, and then communicates with it via pipes.

If you dig a bit more, you’ll come across usermode_driver.c, the history of that file leads you to umh.c, and that history in turn leads you to this commit which contains this comment:

The usermode helper has a provenance from the old usb code which first required a usermode helper.

And this part of the diff of kmod.c:

Unblock all signals when we exec a usermode process. Shuu Yamaguchi December 2000

call_usermodehelper wait flag, and remove exec_usermodehelper. Rusty Russell Jan 2003

In other words, this mechanism (i.e., spawning new processes outside of the normal process hierarchy) has been around for quite a while. It predates the switch to Git and I was not motivated enough to go all the way back to the beginning, but you can find mentions of usermode helpers in the history of kmod.c.

Then again, bpfilter_umh introduces new concepts after all.

What has already existed in the past are “usermode helpers”. For some tasks, the kernel can spawn a new usermode process (i.e., it’s not fork()ed from something else). A prominent example appears to be loading kernel modules (initiated by the kernel, not userspace), which calls out to the normal modprobe utility (see kmod.c from 2005). This page in the KernelNewbies wiki lists more examples.

bpfilter_umh, however, is a new “type” of kernel module which not only contains a normal kernel module but also some other code which will then be run as a userspace process – a “usermode driver”. This approach has only been around since bpfilter in Linux 4.18. The code that is being run in the userspace process is embedded in a kernel module. That’s quite a bit different than just calling the normal /sbin/modprobe that sits around on your drive. This also explains why the exe symlink cannot show something meaningful – there simply is no corresponding binary in the file system. (The code blob is copied from the kernel module to a file in an “anonymous” tmpfs, which doesn’t show up in the output of mount.)

On a side note: When I see such a process with exe being marked as deleted, I restore the binary by doing cp /proc/$pid/exe foo.bin. You can then, for example, ask your distribution’s package manager if it knows a file that has the same SHA256 sum. If it does, it increases the chances of this process being legitimate. This approach isn’t possible with usermode drivers like bpfilter_umh, since there is no single file that contains just that code.

What I could not find (yet) is a way to determine whether a given userspace process belongs to a kernel module or not. So far, I only know of these clues:

/proc/$pid/exe points to / (deleted) (a directory, not a file).
File descriptors 0 and 1 are pipes and no other running process holds a reference to them.
File descriptor 2 points to /dev/kmsg.
When you unload the kernel module (probably has the same name as the process, minus the _umh suffix), the process should quit.

That’s a bit wonky, though. Ideally, I would have hoped that some magic file in /proc or /sys holds the PID(s) of usermode drivers. That would allow us to exclude certain PIDs when searching for processes whose program binaries are marked as “deleted”.

– Update: A reader wondered which parent PID theses processes have. Good idea! They appear to be children of kthreadd, just like kernel threads:

root@ubuntu2004:~# ps ax -o pid,ppid,cmd | grep -e bpfilter -e kthreadd
      2       0 [kthreadd]
   1025       2 bpfilter_umh

So this is probably the best thing to look out for. The PPID itself doesn’t tell you which kernel module they belong to, but it’s a very strong indicator that this is not an “ordinary” userspace process – which means it’s probably safe to trust the process’s title and, well, that says bpfilter_umh, so there you have it. (Just saying: Don’t trust process titles in general. They’re easy to manipulate.)

(Fun fact: This effectively hides bpfilter_umh from the output of pstree.)

– Update: If you grep the kernel sources for fork_usermode_driver, you’ll see that bpfilter is the only place where this function is called (as of Linux 5.18), so this is probably the only kernel module that currently employs this technique (third-party modules aside).

Next blog post on this topic: Trying to verify the running blob of bpfilter_umh.

Comments?