sshd scoping

[michalbiesek]

There are two issues related to scoping sshd.

• After attaching to sshd we don't see that libscope.so is loaded in child process.

• This is related to fact that sshd use execv for starting a new process

• Issue when scoping sshd and connecting via ssh.

ldscope /usr/sbin/sshd -ddd
## I am unable to connect from another shell via `ssh` to `sshd` server

Log message from sshd:

debug3: preauth child monitor started
debug3: privsep user:group 105:65534 [preauth]
debug1: permanently_set_uid: 105/65534 [preauth]
debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
debug3: ssh_sandbox_child: attaching seccomp filter program [preauth]
debug3: append_hostkey_type: ssh-rsa key not permitted by HostkeyAlgorithms [preauth]
debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug3: send packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
`monitor_read_log: child log fd closed`

The last message comes from the following logic:

https://github.com/openssh/openssh-portable/blob/715c892f0a5295b391ae92c26ef4d6a86ea96e8e/atomicio.c#L81-L83

[michalbiesek]

Issue status:

With amazing help from @jrcheli and @iapaddler we find the root cause of the issue.

Thee root cause is described below:

• sshd at the beginning close all file descriptors
• In AppScope we don't close our file descriptors by purpose
• sshd process with following logic will set the process and file limit for the child process.
• the limits will impact us - we will not be able to write to a file and create the process on the AppScope side

Current limitation:
Sshd uses execv to create & execute the child process - with execv we are missing to pass environment variable. This can result to don't use the configuration based on SCOPE_ environment variables.
The workaround is based on putting proper configuration in scope.yml

[michalbiesek]

There is one limitation that I encounter while working on the sshd integration test.
On execv, execve we are trying to find the scope executable to use our loader.

First execv - root

execv /usr/sbin/sshd uid 0

HOSTNAME=707e6dfa0417
PWD=/opt/test-runner
HOME=/root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
TERM=xterm
SHLVL=1
PATH=/usr/local/scope:/usr/local/scope/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBIAN_FRONTEND=noninteractive
OLDPWD=/opt/appscope
_=/usr/local/scope/bin/ldscope
SCOPE_EXEC_PATH=/opt/appscope/bin/linux/x86_64/ldscope
SCOPE_LIB_PATH=/tmp/libscope-1.1.0/libscope.so
SCOPE_PID=18
SCOPE_APP_TYPE=native
LD_PRELOAD=/tmp/libscope-1.1.0/libscope.so
SCOPE_EXEC_TYPE=dynamic
JAVA_TOOL_OPTIONS=-agentpath:/tmp/libscope-1.1.0/libscope.so

Second exec - root

execve /usr/bin/env uid 0

HOSTNAME=707e6dfa0417
PWD=/opt/test-runner
HOME=/root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
TERM=xterm
SHLVL=1
PATH=/usr/local/scope:/usr/local/scope/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBIAN_FRONTEND=noninteractive
OLDPWD=/opt/appscope
_=/usr/local/scope/bin/ldscope
SCOPE_EXEC_PATH=/opt/appscope/bin/linux/x86_64/ldscope
SCOPE_LIB_PATH=/tmp/libscope-1.1.0/libscope.so
SCOPE_PID=23
SCOPE_APP_TYPE=native
LD_PRELOAD=/tmp/libscope-1.1.0/libscope.so
SCOPE_EXEC_TYPE=dynamic
JAVA_TOOL_OPTIONS=-agentpath:/tmp/libscope-1.1.0/libscope.so

Third exec - ssh session user

These environment variables will be used to execute (execve) new session for a new user.

execve /bin/bash uid 1000

USER=sshuser
LOGNAME=sshuser
HOME=/home/sshuser
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SHELL=/bin/bash
MOTD_SHOWN=pam
SSH_CLIENT=127.0.0.1 46430 22
SSH_CONNECTION=127.0.0.1 46430 127.0.0.1 22

In the scenario above during calling execve we are not able to see any SCOPE_ environment variables. Additionally, we will not be able to find ldscope if the PATH for root is different than another user.
Assuming that ldscope is in /usr/local/scope/bin/ldscope:
root:

PATH=/usr/local/scope:/usr/local/scope/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

other user:

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

[michalbiesek]

Update regarding passing environment variables for forked process executed by execv.
The scenario used for the test is:

#Terminal 1
ldscope /usr/sbin/sshd

#Terminal 2
sshpass -p "sshuser" ssh -o StrictHostKeyChecking=no sshuser@localhost

Without passing ldscope to path available for all users:

root          22  1.7  0.0  37524  6296 ?        Ssl  07:17   0:00 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups
root          26  3.2  0.0  38496 12668 ?        Ssl  07:17   0:00  \_ sshd: sshuser [priv]
sshuser       39  1.6  0.0  38736  8124 ?        Sl   07:17   0:00      \_ sshd: sshuser@pts/3
sshuser       41  0.0  0.0   4248  3592 pts/3    Ss+  07:17   0:00          \_ -bash

• libscope.so is in proc 22, 26, 39 mappings
• SCOPE_PID, SCOPE_APP_TYPE, SCOPE_EXEC_TYPE are in the 22, 26, 39 env variables there are no other environment variables despite the fact I have defined additional one like SCOPE_CRIBL_ENABLE

Without passing ldscope to path available for all users and adding to execv logic with writing pid.env file

• libscope.so is in proc 22, 26, 39 mappings
• SCOPE_PID, SCOPE_APP_TYPE, SCOPE_EXEC_TYPE are in the 22 env variables (this coming from that sshd works as a daemon by default ??)
• SCOPE_PID, SCOPE_APP_TYPE, SCOPE_EXEC_TYPE and SCOPE_CRIBL_ENABLE are in the 26, 39 env variables

With passing ldscope to path available for all users and adding to execv logic with writing pid.env file

We see the following state

root          22  1.7  0.0  37524  6296 ?        Ssl  07:17   0:00 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups
root          26  3.2  0.0  38496 12668 ?        Ssl  07:17   0:00  \_ sshd: sshuser [priv]
sshuser       39  1.6  0.0  38736  8124 ?        Sl   07:17   0:00      \_ sshd: sshuser@pts/3
sshuser       40  0.0  0.0   3024  1720 pts/3    Ss   09:54   0:00          \_ /tmp/libscope-web/ldscopedyn /bin/bash
sshuser       41  2.4  0.0  29888  7804 pts/3    Sl+  09:54   0:00              \_ /bin/bash

• libscope.so is in proc 22, 26, 39, 40, 41 mappings
• SCOPE_PID, SCOPE_APP_TYPE, SCOPE_EXEC_TYPE are in the 22, 40, 41 env variables
• SCOPE_PID, SCOPE_APP_TYPE, SCOPE_EXEC_TYPE and SCOPE_CRIBL_ENABLE are in the 26, 39 env variables

The main problem with passing env variables I think is related with:

sshuser       40  0.0  0.0   3024  1720 pts/3    Ss   09:54   0:00          \_ /tmp/libscope-web/ldscopedyn /bin/bash

The bash session is started here:
https://github.com/openssh/openssh-portable/blob/25bd659cc72268f2858c5415740c442ee950049f/session.c#L1699

When we intercept it on execve we don't have any SCOPE_ or LD_PRELOAD we find our ldscope loader in the PATH env variable

appscope/src/utils.c

Lines 140 to 141 in 456ce2f

	// try to resolve the cmd from PATH env variable
	char *path_env_ptr = getenv("PATH");

And we start to run bin/bash with our loader with ldscope found in the PATH:

appscope/src/wrap.c

Line 2679 in 100734d

return g_fn.execve(pathname, argv, envp);

[michalbiesek]

Here is the summary of work.
There are two problems here:

1. When we sshd server we should put ldscope in the PATH location available for all the users like in /etc/environment if not we will miss loading AppScope for ssh session.
2. The SCOPE_ environment variables used to the configuration are not correctly passed to
   Example use cases:

Scenario 1

-ldscope is placed in /usr/local/bin//usr/bin

• the desired configuration is placed in /etc/scope/scope.yml

Not impacted - everything works fine.

Scenario 2

-ldscope is placed in non standard location e.g./usr/local/scope/ldscope

• the desired configuration is placed in /etc/scope/scope.yml

The possible workaround is to pass the ldscope location to sshd process /usr/sbin/sshd -o "SetEnv PATH=$PATH:/usr/local/scope"

Scenario 3

-ldscope is placed in non standard location e.g./usr/local/scope/ldscope

• the desired configuration is passed via environment variables e.g. SCOPE_CRIBL=tcp://127.0.0.1:1234

The possible workaround is to pass the ldscope location to sshd process /usr/sbin/sshd -o "SetEnv SCOPE_CRIBL=$SCOPE_CRIBL"

The one thing worth mentioning is that ldscope should have proper file permissions to use by all users.