While investigating the aftermath of a potential security breach, I came across a somewhat “odd” process looking like this:
root@ubuntu2004:~# ls -al /proc/1026/exe lrwxrwxrwx 1 root root 0 Jun 11 05:26 /proc/1026/exe -> '/ (deleted)'
This virtual symlink normally points to the binary being run, for example:
root@ubuntu2004:~# ls -al /proc/self/exe lrwxrwxrwx 1 root root 0 Jun 11 05:27 /proc/self/exe -> /usr/bin/ls
A common pattern of malicious code is to start a binary and then delete it from disk to avoid being picked up by simple antivirus software, which leads to output like this:
$ ls -al /proc/5194/exe lrwxrwxrwx 1 void users 0 Jun 11 07:29 /proc/5194/exe -> '/tmp/tmp/bla (deleted)'
So, what we often do is to search for processes whose
points to something that no longer exists – or rather, something that
says “deleted”. As a result, the first process shown above (PID 1026)
turned up during my search.
It doesn’t really fit the pattern, though:
exe points to a
directory. Was this some kind of trickery that I didn’t know about?
Long story short, the process in question was started directly by the
kernel itself – it was not
posix_spawn()ed by some
other existing process, thus breaking a bunch of assumptions in my mind.
Yes, it’s not a big surprise that “the kernel can spawn processes”, but
the vast majority of them are children of some other processes that
What I saw, was
The basic idea is that this kernel module creates a “usermode driver”, i.e. an “ordinary” process, and then communicates with it via pipes.
The usermode helper has a provenance from the old usb code which first required a usermode helper.
And this part of the diff of
Unblock all signals when we exec a usermode process. Shuu Yamaguchi December 2000
call_usermodehelperwait flag, and remove
exec_usermodehelper. Rusty Russell Jan 2003
In other words, this mechanism (i.e., spawning new processes outside of
the normal process hierarchy) has been around for quite a while. It
predates the switch to Git and I was not motivated enough to go all the
way back to the beginning, but you can find mentions of usermode helpers
in the history of
bpfilter_umh introduces new concepts after all.
What has already existed in the past are “usermode helpers”. For some
tasks, the kernel can spawn a new usermode process (i.e., it’s not
fork()ed from something else). A prominent example appears to be
loading kernel modules (initiated by the kernel, not userspace), which
calls out to the normal
modprobe utility (see
2005). This page in the KernelNewbies wiki lists more
bpfilter_umh, however, is a new “type” of kernel module which not only
contains a normal kernel module but also some other code which will then
be run as a userspace process – a “usermode driver”. This approach has
only been around since bpfilter in Linux 4.18. The code
that is being run in the userspace process is embedded in a kernel
module. That’s quite a bit different than just calling the normal
/sbin/modprobe that sits around on your drive. This also explains why
exe symlink cannot show something meaningful – there simply is no
corresponding binary in the file system. (The code blob is copied from
the kernel module to a file in an “anonymous”
tmpfs, which doesn’t
show up in the output of
On a side note: When I see such a process with
exe being marked as
deleted, I restore the binary by doing
cp /proc/$pid/exe foo.bin.
You can then, for example, ask your distribution’s package manager if it
knows a file that has the same SHA256 sum. If it does, it increases the
chances of this process being legitimate. This approach isn’t possible
with usermode drivers like
bpfilter_umh, since there is no single file
that contains just that code.
What I could not find (yet) is a way to determine whether a given userspace process belongs to a kernel module or not. So far, I only know of these clues:
/ (deleted)(a directory, not a file).
_umhsuffix), the process should quit.
That’s a bit wonky, though. Ideally, I would have hoped that some magic
/sys holds the PID(s) of usermode drivers. That
would allow us to exclude certain PIDs when searching for processes
whose program binaries are marked as “deleted”.
– Update: A reader wondered which parent PID theses processes
have. Good idea! They appear to be children of
kthreadd, just like
root@ubuntu2004:~# ps ax -o pid,ppid,cmd | grep -e bpfilter -e kthreadd 2 0 [kthreadd] 1025 2 bpfilter_umh
So this is probably the best thing to look out for. The PPID itself
doesn’t tell you which kernel module they belong to, but it’s a very
strong indicator that this is not an “ordinary” userspace process –
which means it’s probably safe to trust the process’s title and, well,
bpfilter_umh, so there you have it. (Just saying: Don’t
trust process titles in general. They’re easy to manipulate.)
(Fun fact: This effectively hides
bpfilter_umh from the output of
– Update: If you
grep the kernel sources for
you’ll see that bpfilter is the only place where this function is
called (as of Linux 5.18), so this is probably the only kernel module
that currently employs this technique (third-party modules aside).
Next blog post on this topic: Trying to verify the running blob of