(#1038) Add `scope start` command #1049

michalbiesek · 2022-07-19T08:09:00Z

TODO:

Add custom file handling in case of attach and service
Handling the error/status to the Edge
Stdin support
Required (#1061) Use SCOPE_CONF_PATH during attach from host to container #1086

Question:

In case of setting custom yml configuration should service be still using the following path /etc/scope/<service_name>/scope.yml or point to a new yml file location

overwrite the file

Next steps:

Interactive process: docker exec -it local/remote login/command run Ensure that our library is loaded

Future Improvements

Handle communication layer from fs namespace after scope attach from host perspective?
Identify User in Namespace/ Container ID in command like scope ps

Found multiple processes matching that name...
ID	PID    	USER    	SCOPED	COMMAND
1 	846727 	-       	true  	memcached
2 	877968 	-       	true  	/usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1 -P /var/run/memcached/memcached.pid
3 	1198535	memcache	false 	/usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1 -P /var/run/memcached/memcached.pid

michalbiesek · 2022-07-20T09:59:43Z

Progress for interactive commands and loading the libscope

For single cmd run

docker exec --user foo -e LD_PRELOAD=/usr/local/lib/libscope.so -it ubuntu_test top
docker exec -e LD_PRELOAD=/usr/local/lib/libscope.so alpine_test top

The libscope.so is loaded in top

The disadvantage is that the command must be prepared on the host side.
Tested on Ubuntu and Alpine.

For interactive login shell

docker exec -it --user foo ubuntu_test bash -l

Steps required to do it:

Create the script in /etc/profile.d/scope.sh

export LD_PRELOAD=/usr/local/lib/libscope.so

All commands executed in the shell result in preloading the libscope.so excluding the shell itself.
Tested on Ubuntu and Alpine.

For interactive shell

For the bash

docker exec -it --user foo ubuntu_test bash

Append to the ~/.bashrc -> export LD_PRELOAD=/usr/local/lib/libscope.so
With the change above commands executed in the shell result in preloading the libscope.so excluding the shell itself.

For the ash (Alpine)
I did not yet found a viable way to load libscope.so for children process in case of the following command:

docker exec -it --user foo alpine_test ash

The solution present in a single cmd run works.

Credits to @iapaddler for investigation.

michalbiesek · 2022-07-20T15:39:11Z

Progress for namespace

explore namespace API to see if we can run scope filter in the container context.

Let's consider the following example:

We want to scope the process on the container by focusing on running most of the action on the host machine.

flowchart LR
  subgraph Host 
    subgraph Container A
       id1["Process A"]
    end
    subgraph Container B
       id2["Service B"]
    end
    id3["Process C"]
  end

flowchart TB
    A[Host] -->|scope filter config.file| C{Check if Process belongs to host namespace}
    C -->|Process is on host namespace| D[Perform scope filter action]
    C -->|Process is not on host namespace| E[Switch namespace]
    E --> D

scope filter depends on passed configuration, but internally it will call actions like:

scope service ?
scope attach ?
Perform action like described here - My current goal is to focus on switching namespace and perform attach operation

Scope attach with switching namespaces at first glance seems to work with Docker containers

michalbiesek · 2022-07-26T08:01:10Z

Scope attach in different namespace progress:

Attach to docker process Ubuntu
Attach to docker process Alpine
Passing configuration file from the host
Passing configuration parameter from host
Add information about namespace switch in log

michalbiesek · 2022-07-27T11:35:40Z

Scope attach in different namespace progress:

Attach to LXC container

While attaching functionality to the Docker container works fine - the LXC container is not eager to cooperate.

Several observations are mentioned below:

with the current implementation, the attach fails on the namespace side in:

appscope/src/scope_static.c

Lines 1100 to 1104 in 431fb7a

    
           int fd = scope_shm_open(path, O_RDWR|O_CREAT, S_IRUSR|S_IRGRP|S_IROTH); 
        
           if (fd == -1) { 
        
               scope_perror("shm_open() failed"); 
        
               return EXIT_FAILURE; 
        
           }

returning the EOVERFLOW

the loader in lxc namespace context has access to the dev/shm path
I see that files created from CLI in the context of namespace (/root/.scope/) have different owner than a root

drwxr-xr-x  3 nobody nogroup    5 Jul 27 11:29 .scope

In the case of Docker container, the same directory and its content are owned by root.
I cannot remove (/root/.scope/) being a root inside the LXC namespace:

root@ubuntuLcTest:~/.scope# ls -la ldscope 
-rwxr-xr-x 1 nobody nogroup 11411456 Jul 27 10:58 ldscope
root@ubuntuLcTest:~/.scope# rm -f ldscope 
rm: cannot remove 'ldscope': Permission denied

This can come from a different approach in user and gid mappings

In Docker:

proc/<proc_running_in_docker_namespace_pid_from_host_perspective>/uid_map 
         0          0 4294967295

proc/<proc_running_in_docker_namespace_pid_from_host_perspective>/gid_map 
         0          0 4294967295

In Lxc:

proc/<proc_running_in_lxc_namespace_pid_from_host_perspective>/uid_map 
         0    1000000 1000000000

proc/<proc_running_in_lxc_namespace_pid_from_host_perspective>/gid_map 
         0    1000000 1000000000

michalbiesek · 2022-07-28T11:09:04Z

Scope attach in different namespace progress:

LXC seems to work - it required a GID and UID translation during switching namespace to avoid file permissions issue mentioned in the previous comment

I have observed timing-related? the issue that sometimes during attach I fail during:

appscope/src/scope_static.c

Lines 1071 to 1074 in 431fb7a

    
           if (scope_readlink("/proc/self/exe", execPath, sizeof(execPath) - 1) == -1) { 
        
               scope_perror("readlink(/proc/self/exe) failed"); 
        
               return EXIT_FAILURE; 
        
           }

Setting SCOPE_EXEC_PATH in cli solves the issue

I am not sure if it is worth mentioning somewhere that the --cap-add CAP_SYS_PTRACE must be added in the case of Docker container even if we perform attach as a root from the host
The one challenge which I did not resolve yet is the following scenario:

 lxc launch ubuntu:20.04 ubuntuLcTest
 lxc exec ubuntuLcTest bash
 apt update -y && apt install -y redis

After installation redis will be run as a service - but we cannot attach to it - even from the container itself. scope service will work in this case though.
The issue is with stricter systemd setting for the redis service.
E.g. setting

PrivateTmp=true

Prevent us from attach

error: dlopen() failed, library could not be injected

- add support for passing configuration file from host into the container - in `ldscope` copy the configuration file from host to container Closes #1061

cli/run/start.go

seanvaleo · 2022-09-14T15:31:46Z

cli/run/start.go

+
+	// Validate user has root permissions
+	if err := util.UserVerifyRootPerm(); err != nil {
+		log.Fatal().


log.Fatal will actually cause an os.Exit(). might be better to stick to log.Error and return err , to maintain our pattern

I intend to call os. exit here please see that there is no point to run it further if you are not an root.

then the return err afterwards is not needed

I think Your suggestion with Log.error instead of Log.fatal is better. With Log.fatal I call Os.Exit with the same value. With Log.error I can distinguish the error with the return value. Long story short changed

cli/run/start.go

- `Verfy` -> `Verify` - `exit` -> `exist` - `alllow` -> `allow`

- Add TODO - Fix formatting

src/ns.c

iapaddler · 2022-09-14T16:55:00Z

src/ns.c

+    }
+    if (setNamespace(hostPid, "mnt") == FALSE) {
+        goto cleanupMem;
+    }


seems like we need a net NS and probably ipc as well?

TBH for my internal testing mnt and pid namespace was sufficent

iapaddler · 2022-09-14T17:18:28Z

src/loaderop.c

+static int g_debug = 0;
+
+static void
+setEnvVariable(char *env, char *value)


we have a utility func that sets an env var. utils.c:fullSetenv(). can that be used?

I do not think so: FullSetenv is doing the operation based on g_fn structure.
setEnvVariable (or more generally loaderop module) is doing the operations in the context of our loader (ldscope) where we g_fn is not available.

iapaddler · 2022-09-14T17:20:03Z

src/loaderop.c

+ */
+
+static int
+get_dir(const char *path, char *fres, size_t len) {


should this be a generally avail func in utils.c, like getpath()?

TBH the code here was moved from scope_static.c with the intention to share some logic between modules e.g. function loaderOpPatchLibrary used in ns.c

See commit d5badd8 for details

website/src/pages/docs/cli-reference.md

src/libdir.c

src/setup.c

iapaddler · 2022-09-14T17:48:39Z

src/loaderop.c

+}
+
+char *
+loaderOpGetLoader(const char *exe) {


why are all of these funcs moved? are they used in multiple places?

With scope start: one of the first operations which I perform is copying the libscope.so to the well-known location on the host and on the containers.
For the container part, I use the code presented in the ldscope as a --configure. If the container is a musl-based distro (e.g. Alpine) I need to perform patching of the libscope.so.
Therefore after copying the libscope.so from a host into a container I need to perform a patching operation. I do not perform patching via ldscope --patch. I patch the libscope.so with function loaderOpPatchLibrary.
The function loaderOpPatchLibrary is used in multiple places: in ldscope and in setup.c. The rest of the code is moved here since it is needed for the loaderOpPatchLibrary logic.
I limit the visibility of loaderOpSetLibrary and loaderOpSetLibrary

src/setup.c

- `loaderOpSetLibrary` and `loaderOpSetLibrary`

michalbiesek force-pushed the feature/1038/scope-filter branch 3 times, most recently from 9e54b38 to 9ae1f2b Compare July 20, 2022 08:35

iapaddler mentioned this pull request Jul 20, 2022

Interpose services based on a filter #1038

Closed

michalbiesek force-pushed the feature/1038/scope-filter branch 2 times, most recently from 6b467ce to a276757 Compare July 25, 2022 10:41

jrcheli changed the base branch from master to release/1.2 July 28, 2022 17:18

michalbiesek force-pushed the feature/1038/scope-filter branch 2 times, most recently from e356137 to 65db197 Compare July 29, 2022 15:01

michalbiesek changed the title ~~[WIP] (#1038) Add scope filter command~~ [WIP] (#1038) Add scope start command Aug 23, 2022

michalbiesek force-pushed the feature/1038/scope-filter branch 6 times, most recently from ec38cdd to 0157253 Compare August 26, 2022 12:38

michalbiesek force-pushed the feature/1038/scope-filter branch 3 times, most recently from 97a628b to 21e0715 Compare September 2, 2022 08:58

michalbiesek changed the title ~~[WIP] (#1038) Add scope start command~~ (#1038) Add scope start command Sep 2, 2022

michalbiesek marked this pull request as ready for review September 2, 2022 09:09

michalbiesek force-pushed the feature/1038/scope-filter branch 2 times, most recently from 21e897e to 7b81cc0 Compare September 2, 2022 10:05

michalbiesek linked an issue Sep 2, 2022 that may be closed by this pull request

Interpose services based on a filter #1038

Closed

(#1061) Use SCOPE_CONF_PATH during attach from host to container

b88ab6b

- add support for passing configuration file from host into the container - in `ldscope` copy the configuration file from host to container Closes #1061

seanvaleo reviewed Sep 14, 2022

View reviewed changes

cli/run/start.go Outdated Show resolved Hide resolved

seanvaleo reviewed Sep 14, 2022

View reviewed changes

cli/run/start.go Outdated Show resolved Hide resolved

seanvaleo reviewed Sep 14, 2022

View reviewed changes

cli/run/start.go Outdated Show resolved Hide resolved

seanvaleo reviewed Sep 14, 2022

View reviewed changes

cli/run/start.go Outdated Show resolved Hide resolved

seanvaleo reviewed Sep 14, 2022

View reviewed changes

cli/run/start.go Outdated Show resolved Hide resolved

michalbiesek added 4 commits September 14, 2022 18:01

Restore SilenceUsage flag to default value

a647610

Rename scope loader instance ld -> sL

c693d6d

Fix typos

21dd0f4

- `Verfy` -> `Verify` - `exit` -> `exist` - `alllow` -> `allow`

Fix comments

c1925e0

- Add TODO - Fix formatting

iapaddler reviewed Sep 14, 2022

View reviewed changes

src/ns.c Show resolved Hide resolved

iapaddler reviewed Sep 14, 2022

View reviewed changes

src/ns.c Outdated Show resolved Hide resolved

iapaddler reviewed Sep 14, 2022

View reviewed changes

michalbiesek added 2 commits September 14, 2022 19:14

Fix coding style issue

875efee

setNamespace remove goto solution

3c925bf

iapaddler reviewed Sep 14, 2022

View reviewed changes

seanvaleo reviewed Sep 14, 2022

View reviewed changes

website/src/pages/docs/cli-reference.md Show resolved Hide resolved

seanvaleo reviewed Sep 14, 2022

View reviewed changes

src/libdir.c Show resolved Hide resolved

seanvaleo reviewed Sep 14, 2022

View reviewed changes

src/setup.c Outdated Show resolved Hide resolved

iapaddler reviewed Sep 14, 2022

View reviewed changes

src/setup.c Outdated Show resolved Hide resolved

michalbiesek added 5 commits September 15, 2022 10:38

Replace log.Fatal to log.Error

c84d1c5

Limit visibility of loaderop functions

6ada40d

- `loaderOpSetLibrary` and `loaderOpSetLibrary`

Replae magic number with macro

fc0cf88

Fix coding style issue

32b6dfe

Rename variable for container PID

b31ab7d

michalbiesek merged commit 2818ca8 into release/1.2 Sep 16, 2022

michalbiesek mentioned this pull request Sep 16, 2022

Handle empty data in scope start #1106

Merged

michalbiesek deleted the feature/1038/scope-filter branch November 15, 2022 08:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(#1038) Add `scope start` command #1049

(#1038) Add `scope start` command #1049

michalbiesek commented Jul 19, 2022 •

edited

Loading

michalbiesek commented Jul 20, 2022 •

edited

Loading

michalbiesek commented Jul 20, 2022 •

edited

Loading

michalbiesek commented Jul 26, 2022 •

edited

Loading

michalbiesek commented Jul 27, 2022 •

edited

Loading

michalbiesek commented Jul 28, 2022 •

edited

Loading

seanvaleo Sep 14, 2022

michalbiesek Sep 14, 2022

seanvaleo Sep 14, 2022

michalbiesek Sep 15, 2022

iapaddler Sep 14, 2022

michalbiesek Sep 14, 2022

iapaddler Sep 14, 2022

michalbiesek Sep 14, 2022

iapaddler Sep 14, 2022

michalbiesek Sep 14, 2022

iapaddler Sep 14, 2022

michalbiesek Sep 15, 2022

(#1038) Add scope start command #1049

(#1038) Add scope start command #1049

Conversation

michalbiesek commented Jul 19, 2022 • edited Loading

TODO:

Question:

Next steps:

Future Improvements

michalbiesek commented Jul 20, 2022 • edited Loading

For single cmd run

For interactive login shell

For interactive shell

michalbiesek commented Jul 20, 2022 • edited Loading

michalbiesek commented Jul 26, 2022 • edited Loading

michalbiesek commented Jul 27, 2022 • edited Loading

michalbiesek commented Jul 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

(#1038) Add `scope start` command #1049

(#1038) Add `scope start` command #1049

michalbiesek commented Jul 19, 2022 •

edited

Loading

michalbiesek commented Jul 20, 2022 •

edited

Loading

michalbiesek commented Jul 20, 2022 •

edited

Loading

michalbiesek commented Jul 26, 2022 •

edited

Loading

michalbiesek commented Jul 27, 2022 •

edited

Loading

michalbiesek commented Jul 28, 2022 •

edited

Loading