1.4 Release Checklist

[seanvaleo]

Checklist

• filter -> rules Update filter to use new rules sub-command #1520
• wait for 10s on reattach with --inspect flag ; detach with --wait flag
• remove support for a filter file in /tmp Remove support for a filter file in /tmp #1537
• config is not getting set correctly in filter file or in attach when using filter --add Config is not getting set correctly in rules file or in attach when using filter --add #1568
• add support for placing(cli) and retrieving(libscope) a filter file in CRIBL_HOME
• do we mount and env in all new containers after a filter add
• do we mount and env in all existing containers after a filter add
• SIGUSR2 [Bug]: Postgres fails to be preloaded #1529
• memory leak
• process leak
• ld.so.preload when scope is in the path
• accept a "unixpath" argument in the rules --add command and put it in the rules file under "source"
• remove support for CRIBL_HOME for determining libscope and filter path
• allow removal of a rules entry by specifying a proc name and an arg Allow removal of a rules entry by specifying a proc name and an arg #1558
  ----------- if time permits -----------
• signal on arm go Go signals on aarch64 #1469
• add existing container support to filter --add and filter --remove Add mounts to existing containers #1523

[iapaddler]

Issues & Questions

• should we have a -f option for filter --add?
  answer; use user config option

• an empty filter list returns an empty allow list. should it present a message instead?
  from within a container:
  ./scope filter --rootdir /hostfs/
  allow: []
  This results when a rules file exists and is empty
  We probably want to check for an empty list and emit the same message as when no rules file exists

• memfd is inherited from a parent proc and causes the ps command not to work

• host filter remove error
  sudo ./scope filter --add top
  sudo ./scope filter
  sudo ./scope filter --remove top
  results in this error: fork/exec /memfd:: no such file or directory
  seanvaleo: i tried the 'host filter remove error' on my machine and i did not see the memfd error you saw.

• Error mounting the socket file
  With this in the rules file -

"source": {
           "id": "in_appscope",
           "enableUnixPath": true,
           "unixSocketPath": "/var/run/appscope/appscope.sock",
           "tls": {
              "disabled": true
           },
          "host": "0.0.0.0",
          "port": 10090,
          "authToken": ""
       }

We get this error when starting a container
docker run -it --rm ubuntu:latest bash
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/run/appscope/appscope.sock" to rootfs at "/var/run/appscope/appscope.sock": stat /var/run/appscope/appscope.sock: no such file or directory: unknown.

The dir /var/run/appscope exists. However, there is no appscope.sock file yet.
Do we want to mount the socket file or the dir where the socket file exists.
Several docs describe that the dir should be mounted. here and there.

• This release is not intended to automatically scope lxd or podman containers. By that we mean when a scope rules command is executed the env and mount points are not added to new lxd or podman containers. This ability has been tested with docker and containerd.

• rules add with arg segfaults
  Can't reproduce this. Not sure what caused this.
  $ sudo scope rules --add 'sleep infinity'
  Segmentation fault
  $ sudo scope rules --add sleep infinity
  Segmentation fault

• rules add with arg (branch feat/remove-proc-arg)
  sudo scope rules --add sleep --arg infinity
  sleep 60
  sudo scope ps
  ID PID USER COMMAND
  1 89047 donn sleep 60

Why is sleep 60 scoped when it seems like sleep infinity was defined in the rules file?

• rules remove (branch feat/remove-proc-arg)
  sudo scope rules --remove sleep --arg infinity
  Usage:
  scope rules [flags]
  .........
  fork/exec /memfd:: no such file or directory

• rules remove (branch feat/remove-proc-arg)
  sudo scope rules --remove sleep
  sleep was not removed from scope_rules

[iapaddler]

Test Status

• host - host

• docker - host

• docker - docker

• docker - containerd (new C's)

• docker - lxc (new C's)
  does not work. ref Running new lxc containers does not automatically set env or mounts #1552

• host - docker
  see issue above

• docker - docker (existing container)
  holding off for now - will add if time permits

• docker - podman
  no LD_PRELOAD env set in the new container
  not supporting for now

• use CRIBL_HOME and validate paths
  note that CRIBL_HOME is defined in the Edge docs for a host install, not a container install
  refer to comment below

[iapaddler]

Tests
3 scenarios:

• host-to-host
• container-to-host (argument --rootdir /hostfs)
• container-to-container (argument --rootdir /hostfs/proc//root)

Tip: start "edge" containers like this:
docker run -it --rm -e CRIBL_EDGE=1 -p 9420:9420 -v /<path_to_code>/appscope:/opt/appscope -v /var/run/appscope:/var/run/appscope -v /var/run/docker.sock:/var/run/docker.sock -v /:/hostfs:ro --privileged --name cribl-edge ubuntu

end to end tests:
scope filter --add
# Is the process attached?
# Is ld.so.preload set?
scope filter
# Does the filter file look correct?
scope filter --add
<start a new process>
# Is the process scoped?
scope filter --remove
# Is the process detached?
scope filter --remove
# Is the process detached?
scope filter
# Is the file empty?
# Is ld.so.preload unset?

! Important
Also test the use of $CRIBL_HOME to define the path for libscope, filter, unix socket, etc. See below comment for intended bevaviour

[seanvaleo]

Helpful Tips
Desired behaviour:

• if scope run as root
  • library: /usr/lib/appscope/dev/libscope.*
  • rules: /usr/lib/appscope/scope_rules
• if scope run as nonroot
  • library: /tmp/appscope/dev/libscope.*
  • (not possible to add rules file as nonroot)
• if version is dev, always overwrite contents of dev.
• dev true/false has nothing to do with the base path.

NOTE:

When you are using scope rules and the resulting libscope in /usr/lib/appscope you MUST execute a scope command with sudo in order to cause libscope to be updated after a build. Example; sudo ./bin/linux/x86_64/scope ls will copy an updated libscope to /usr/lib/appscope//.

[seanvaleo]

Scenarios we intend to support

WHATEVER YOU DO; CHMOD ON LD.SO.PRELOAD TO ENABLE USER REMOVAL BEFORE TESTING

Scenario: Edge running in a container

Start edge (as defined in cribl documentation):

docker run -it --rm -e CRIBL_EDGE=1  -p 9420:9420 -v /<path_to_code>/appscope:/opt/appscope  -v /var/run/appscope:/var/run/appscope  -v /var/run/docker.sock:/var/run/docker.sock  -v /:/hostfs:ro  --privileged  --name cribl-edge cribl/cribl:latest bash
/opt/cribl/bin/cribl start

Tests:

<start edge container>
<run top on the host>
<start a container>
<run top in that container>
scope rules --add top --sourceid A --rootdir /hostfs --unixpath /var/run/appscope
scope rules --rootdir /hostfs
### Does the rules file contain an entry for top?
scope ps --rootdir /hostfs
### Are two top processes scoped by attach?
<run top on the host>
<start a new container>
<run top in the new container>
scope ps --rootdir /hostfs
### Are four top processes scoped (2 by attach, 2 by preload)?
### Is data flowing into edge from 3 processes (2 on host, 1 in new container)?
scope rules --remove top --sourceid A --rootdir /hostfs
scope rules --rootdir /hostfs
### Is the rules file empty?
scope ps --rootdir /hostfs
### Are 0 top processes scoped?
<run top on the host>
scope ps --rootdir /hostfs
### Are 0 top processes scoped?
### A unix sock path is supported on the rules add command line. it will place the unix path in the rules file where the config from Edge is placed. 
sudo scope rules --add top --unixpath /var/run/appscope
at the end of the rules file we will see this:
source:
  unixSocketPath: /var/run/appscope
  authToken: ""
the result is that /var/run/appscope is mounted in new containers.

Scenario: Edge running on the host

Start edge (as defined in cribl documentation):

<switch the user to root>
curl https://cdn.cribl.io/dl/4.1.3/cribl-4.1.3-15457782-linux-x64.tgz -o ~/Downloads/cribl.tgz
cd /opt/
tar xvzf ~/Downloads/cribl.tgz
mv /opt/cribl/ /opt/cribl-edge
export CRIBL_HOME=/opt/cribl-edge # note: $CRIBL_HOME is set only in the cribl process (and cli children)
cd /opt/cribl-edge/bin
./cribl mode-edge
chown root:root /opt/cribl-edge/bin/cribl
./cribl start

Tests:

<start edge on host>
<run top on the host>
<start a container>
<run top in that container>
scope rules --add top --sourceid A --unixpath /var/run/appscope
scope rules
### Does the rules file contain an entry for top?
scope ps
### Are two top processes scoped by attach?
<run top on the host>
<start a new container>
<run top in the new container>
scope ps
### Are four top processes scoped (2 by attach, 2 by preload)?
### Is data flowing into edge from three processes (2 on host, 1 in new container)?
scope rules --remove top --sourceid A
scope rules
### Is the rules file empty?
scope ps
### Are 0 top processes scoped?
<run top on the host>
scope ps
### Are 0 top processes scoped?

Where files will be created

Host processes:

• libscope: should end up in /usr/lib/appscope/<ver>/ on the host
• scope: should end up in /usr/lib/appscope/<ver>/ on the host
• scope_rules: should end up in /usr/lib/appscope/scope_rules on the host
• unix socket:
  • edge running in container: will be in /var/run/appscope/ on the host by default (edge documentation describes that /var/run/appscope is mounted from the host into the container).
  • edge running on host: will be in $CRIBL_HOME/state/ by default

Existing container processes:

• libscope: installed into /usr/lib/appscope// in all existing containers (/etc/ld.so.preload points to this)
• scope: not required
• scope_rules: /usr/lib/appscope should be mounted into all existing containers into /usr/lib/appscope/
• unix socket: the dirpath defined in scope_rules should be mounted into all existing containers ($CRIBL_HOME/state/ note that the env var will be resolved in the scope_rules file)

New container processes:

• libscope: extracted into /opt/appscope in all new containers
• scope: /usr/lib/appscope should be mounted into all new containers into /usr/lib/appscope/
• scope_rules: /usr/lib/appscope should be mounted into all new containers
• unix socket: the dirpath defined in scope_rules should be mounted into all new containers (default /var/run/appscope/)