Revealing hidden processes

[tclahr]

Discussed in #258

^{Originally posted by halpomeranz July 23, 2024}
I recently submitted a pull request to collect /proc/*/mounts to help detect when somebody is using a bind mount to hide processes (see https://dfir.ch/posts/slash-proc/). However, UAC could go farther and actively unmount and bind mounts found under /proc in order to reveal the hidden process information. Of course, this would actively change the state of the remote system and I wasn't sure if UAC wanted to go there.

Another idea I was considering was to implement something similar to chkproc from the chkrootkit distro (chkrootkit.org). Essentially write a small script that tries all /proc directories from {1..$(cat /proc/sys/kernel/pid_max)}. If it ends up in a directory that isn't visible under /proc, then you have a hidden process and probably a rootkit. Obviously, we would want to collect this process' info and probably call out the hidden process in some special way.

Thoughts on either of these ideas?

[mumuwawa]

The idea of actively unmounting bind mounts under /proc to reveal hidden processes could be very effective, but I understand the concern about potentially altering the remote system's state. Perhaps implementing this with caution, such as adding warnings or requiring user confirmation, might be a good way to mitigate risks while enhancing detection capabilities.

Additionally, creating a script similar to chkproc from chkrootkit sounds like a solid approach. Iterating over all possible PIDs and checking their visibility in /proc could help identify hidden processes.

[tclahr]

I will add a property (modifier: true) to the artifacts that have the ability to change the system state. They will be disabled by default, and only executed if the --enable-modifiers is used in the command line.

Also, I was thinking about adding an artifact to live_response/process/procfs_information.yaml that would compare all PIDs in /proc vs PIDs in ps. I don't think we need to brute force all PIDs, but only checking the ones that have a directory in /proc and are not listed in ps.

  -
    description: List all PIDs with a directory in /proc but hidden for ps command.
    supported_os: [linux]
    collector: command
    foreach: for pid in /proc/[0-9]*; do echo ${pid} | sed -e 's:/proc/::'; done
    command: if ps ax | awk '{print $1}' | grep -q %line%; then true else echo %line%; fi
    output_file: hidden_pids_for_ps_command.txt

Additionally, a new artifact could be created in live_response/modifiers/revel_hidden_processes.yaml to collect the current system state, and them umount any bind mounted directories under /proc.

This artifact would run prior any other one as we want to revel hidden processes before UAC runs over the artifacts in live_response/process/*

version: 1.0
modifier: true
output_directory: /live_response/modifiers
artifacts:
  -
    description: Lists all mounted filesystems before changing the system state.
    supported_os: [linux]
    collector: command
    command: mount
    output_file: mount.txt
  -
    description: Report a snapshot of the current processes before changing the system state.
    supported_os: [linux]
    collector: command
    command: ps
    output_file: ps.txt
  -
    description: Report a snapshot of the current processes before changing the system state.
    supported_os: [linux]
    collector: command
    command: ps auxwww
    output_file: ps_auxwww.txt
  -
    description: Report a snapshot of the current processes before changing the system state.
    supported_os: [linux]
    collector: command
    command: ps auxwwwf
    output_file: ps_auxwwwf.txt
  -
    description: Report a snapshot of the current processes before changing the system state.
    supported_os: [linux]
    collector: command
    command: ps -ef
    output_file: ps_-ef.txt
  -
    description: List all PIDs with a directory in /proc but hidden for ps command.
    supported_os: [linux]
    collector: command
    foreach: for pid in /proc/[0-9]*; do echo ${pid} | sed -e 's:/proc/::'; done
    command: if ps ax | awk '{print $1}' | grep -q %line%; then true else echo %line%; fi
    output_file: hidden_pids_for_ps_command.txt
  -
    description: Umount all bind mounted directories to /proc/PID.
    supported_os: [linux]
    collector: command
    foreach: mount | awk 'BEGIN { FS=" on "; } { print $2; }' | grep "/proc/[0-9]" | awk '{print $1}'
    command: umount %line%
    output_file: umount_%line%.txt

[tclahr]

@halpomeranz

[tclahr]

Code was pushed to add_modifiers branch.

[tclahr]

Code has been merged into develop.