ecapture in container fail to locate pthread lib #69

vincentmli · 2022-06-05T16:52:30Z

Describe the bug

In k8s environment, run ecapture in k8s node normally with --pthread argument works fine

 [root@cilium-dev home]# ecapture tls --libssl=/lib64/libssl.so.1.1.1g --pthread=/lib64/libpthread.so.0 --hex
2022/06/05 12:26:22 pid info :3292288
2022/06/05 12:26:22 start to run EBPFProbeOPENSSL module
2022/06/05 12:26:22 start to run EBPFProbeGNUTLS module
2022/06/05 12:26:22 stat /usr/lib/libgnutls.so.30: no such file or directory
2022/06/05 12:26:22 start to run EBPFProbeNSPR module
2022/06/05 12:26:22 stat /usr/lib/libnspr4.so: no such file or directory
2022/06/05 12:26:22 HOOK type:2, binrayPath:/lib64/libssl.so.1.1.1g
2022/06/05 12:26:22 libPthread so Path:/lib64/libpthread.so.0 <===libPhread
2022/06/05 12:26:22 target all process.

but if run ecapture in pod/container with --pthread in the same k8s node, ecapture fail to locate libpthread, the pod mounted node / in pod /mnt directory for ecapture to locate the node /lib64/libpthread.so.0 library

[root@cilium-dev home]# kubectl exec -it netshoot-ecap -- /bin/bash

bash-5.1# ecapture tls --pthread=/mnt/lib64/libpthread.so.0 --libssl=/mnt/lib64/libssl.so.1.1.1g --hex

2022/06/05 16:46:48 pid info :3305486
2022/06/05 16:46:48 start to run EBPFProbeOPENSSL module
2022/06/05 16:46:48 start to run EBPFProbeGNUTLS module
2022/06/05 16:46:48 lstat /etc/ld.so.conf: no such file or directory <===
2022/06/05 16:46:48 invalid argument <====

bash-5.1# ls -l /mnt/lib64/libpthread.so.0 /mnt/lib64/libssl.so.1.1.1g
lrwxrwxrwx    1 root     root            18 Jul 20  2020 /mnt/lib64/libpthread.so.0 -> libpthread-2.28.so
-rwxr-xr-x    1 root     root        615576 Mar 30  2021 /mnt/lib64/libssl.so.1.1.1g

To Reproduce
Steps to reproduce the behavior:

deploy netshoot pod image that includes ecapture binary

see netshoot diff vincentmli/netshoot@72633ba to build your netshoot
or deploy the netshoot pod from docker hub vli39/netshoot:ecap with following yaml

netshoot-ecap yaml with privilege and mount k8s node / in pod /mnt

apiVersion: v1
kind: Pod
metadata:
 name: netshoot-ecap
spec:
 hostPID: true
 hostNetwork: true
 nodeSelector:
    dedicated: master
 containers:
   - name: netshoot-ecap
     image: vli39/netshoot:ecap
     command:
       - /bin/sleep
       - "3600"
     volumeMounts:
       - mountPath: /mnt
         name: host-slash
     securityContext:
       privileged: true
 volumes:
   - name: host-slash
     hostPath:
       path: /
       type: ''

kubectl exec into netshoot pod

[root@cilium-dev home]# kubectl exec -it netshoot-ecap -- /bin/bash

bash-5.1# ecapture tls --pthread=/mnt/lib64/libpthread.so.0 --libssl=/mnt/lib64/libssl.so.1.1.1g --hex

2022/06/05 16:46:48 pid info :3305486
2022/06/05 16:46:48 start to run EBPFProbeOPENSSL module
2022/06/05 16:46:48 start to run EBPFProbeGNUTLS module
2022/06/05 16:46:48 lstat /etc/ld.so.conf: no such file or directory
2022/06/05 16:46:48 invalid argument

bash-5.1# ls -l /mnt/lib64/libpthread.so.0 /mnt/lib64/libssl.so.1.1.1g
lrwxrwxrwx    1 root     root            18 Jul 20  2020 /mnt/lib64/libpthread.so.0 -> libpthread-2.28.so
-rwxr-xr-x    1 root     root        615576 Mar 30  2021 /mnt/lib64/libssl.so.1.1.1g

Expected behavior

expect ecapture in container environment with --pthread argument to find the right libpthread lib for connect and ignore looking for libthread through ld.so.conf

Screenshots

Linux Server/Desktop (please complete the following information):

OS: [e.g. iOS]

  cat /etc/os-release 
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"

Kernel Version [e.g. 5.10]

uname -a
Linux cilium-dev 5.18.0+ #1 SMP PREEMPT_DYNAMIC Wed Jun 1 21:37:44 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

Version [e.g. 0.1.3-20220313-69c1e0]
0.1.8-20220525-c030d80-prev-84a358e

Additional context

The text was updated successfully, but these errors were encountered:

vincentmli · 2022-06-06T15:04:07Z

@cfc4n I am actually curious why ecapture requires SEC("uprobe/connect") https://github.com/ehids/ecapture/blob/master/kern/openssl_kern.c#L285-L291, pardon me that I am not familiar with how bpf uprobe works, I thought SEC("uprobe/SSL_read") and SEC("uprobe/SSL_write") alone would give us the SSL payload, no? it must require SEC("uprobe/connect")?

vincentmli · 2022-06-07T04:25:47Z

[root@cilium-dev home]# kubectl exec -it netshoot-ecap -- /bin/bash

bash-5.1# ecapture tls --pthread=/mnt/lib64/libpthread.so.0 --libssl=/mnt/lib64/libssl.so.1.1.1g --hex

2022/06/05 16:46:48 pid info :3305486
2022/06/05 16:46:48 start to run EBPFProbeOPENSSL module
2022/06/05 16:46:48 start to run EBPFProbeGNUTLS module
2022/06/05 16:46:48 lstat /etc/ld.so.conf: no such file or directory <===
2022/06/05 16:46:48 invalid argument <====

@cfc4n after add log in ecapture, I know what's wrong, got to read the output carefully, from above output, the error actually is caused by start to run EBPFProbeGNUTLS module, when runs ecapture in container with --libssl and --pthread, ecapture still tries to auto-probing libs for gnutls and result in lstat /etc/ld.so.conf: no such file or directory, after I made the change like below to test, the problem went away.

diff --git a/cli/cmd/tls.go b/cli/cmd/tls.go
index dc6135c..69a4f7d 100644
--- a/cli/cmd/tls.go
+++ b/cli/cmd/tls.go
@@ -57,7 +57,8 @@ func openSSLCommandFunc(command *cobra.Command, args []string) {
        }
        log.Printf("pid info :%d", os.Getpid())
 
-       modNames := []string{user.MODULE_NAME_OPENSSL, user.MODULE_NAME_GNUTLS, user.MODULE_NAME_NSPR}
+       modNames := []string{user.MODULE_NAME_OPENSSL}

cfc4n · 2022-06-07T06:17:33Z

@cfc4n I am actually curious why ecapture requires SEC("uprobe/connect") https://github.com/ehids/ecapture/blob/master/kern/openssl_kern.c#L285-L291, pardon me that I am not familiar with how bpf uprobe works, I thought SEC("uprobe/SSL_read") and SEC("uprobe/SSL_write") alone would give us the SSL payload, no? it must require SEC("uprobe/connect")?

HOOK SEC("uprobe/connect") to capture IP ADDRESS, because HOOK SEC("uprobe/SSL_write")、SEC("uprobe/SSL_read") can only capture plaintext buf of SSL, not include IP ADDRESS info.

vincentmli · 2022-06-07T16:01:44Z

@cfc4n after add log in ecapture, I know what's wrong, got to read the output carefully, from above output, the error actually is caused by start to run EBPFProbeGNUTLS module, when runs ecapture in container with --libssl and --pthread, ecapture still tries to auto-probing libs for gnutls and result in lstat /etc/ld.so.conf: no such file or directory, after I made the change like below to test, the problem went away.

I think we could introduce another argument something like single-mod to only trace one tls type connection user.MODULE_NAME_OPENSSL, user.MODULE_NAME_GNUTLS, user.MODULE_NAME_NSPR, I could send a PR if you agree @cfc4n

vincentmli · 2022-06-07T16:31:31Z

following diff does the work, I will send a PR.

diff --git a/cli/cmd/global.go b/cli/cmd/global.go
index 0d7c5f3..3c72e7e 100644
--- a/cli/cmd/global.go
+++ b/cli/cmd/global.go
@@ -11,9 +11,10 @@ import (
 // GlobalFlags are flags that defined globally
 // and are inherited to all sub-commands.
 type GlobalFlags struct {
-       IsHex bool
-       Debug bool
-       Pid   uint64 // PID
+       IsHex        bool
+       Debug        bool
+       Pid          uint64 // PID
+       IsSingleMode bool   // only trace one TLS type connection
 }
 
 func getGlobalConf(command *cobra.Command) (conf GlobalFlags, err error) {
@@ -31,5 +32,10 @@ func getGlobalConf(command *cobra.Command) (conf GlobalFlags, err error) {
        if err != nil {
                return
        }
+
+       conf.IsSingleMode, err = command.Flags().GetBool("single-mode")
+       if err != nil {
+               return
+       }
        return
 }
diff --git a/cli/cmd/root.go b/cli/cmd/root.go
index 7bc7bfc..c236438 100644
--- a/cli/cmd/root.go
+++ b/cli/cmd/root.go
@@ -73,5 +73,6 @@ func init() {
        //rootCmd.Flags().BoolP("toggle", "t", false, "Help message for toggle")
        rootCmd.PersistentFlags().BoolVarP(&globalFlags.Debug, "debug", "d", false, "enable debug logging")
        rootCmd.PersistentFlags().BoolVar(&globalFlags.IsHex, "hex", false, "print byte strings as hex encoded strings")
+       rootCmd.PersistentFlags().BoolVar(&globalFlags.IsSingleMode, "single-mode", false, "single tls mode trace")
        rootCmd.PersistentFlags().Uint64VarP(&globalFlags.Pid, "pid", "p", defaultPid, "if pid is 0 then we target all pids")
 }
diff --git a/cli/cmd/tls.go b/cli/cmd/tls.go
index dc6135c..b7132aa 100644
--- a/cli/cmd/tls.go
+++ b/cli/cmd/tls.go
@@ -108,7 +108,13 @@ func openSSLCommandFunc(command *cobra.Command, args []string) {
                                return
                        }
                }(mod)
+
                runMods++
+
+               if gConf.IsSingleMode {
+                       log.Printf("single mode\n")
+                       break
+               }
        }
 
        // needs runmods > 0

test

 kubectl exec -it netshoot-ecap -- /bin/bash
bash-5.1# ecapture tls --pthread=/mnt/lib64/libpthread.so.0 --libssl=/mnt/lib64/libssl.so.1.1.1g --hex --single-mode=true
2022/06/07 16:27:39 pid info :962631
2022/06/07 16:27:39 start to run EBPFProbeOPENSSL module
2022/06/07 16:27:39 single mode
2022/06/07 16:27:39 HOOK type:2, binrayPath:/mnt/lib64/libssl.so.1.1.1g
2022/06/07 16:27:39 libPthread so Path:/mnt/lib64/libpthread.so.0
2022/06/07 16:27:39 target all process. 

2022/06/07 16:28:25 PID:963154, Comm:curl, TID:963154, Version:TLS1_2_VERSION, Send 76 bytes to 10.169.72.34:443, Payload:
GET / HTTP/1.1
Host: 10.169.72.34
User-Agent: curl/7.61.1
Accept: */*


2022/06/07 16:28:25 PID:963154, Comm:curl, TID:963154, Version:TLS1_2_VERSION, Recived 538 bytes from 10.169.72.34:443, Payload:
HTTP/1.1 200 OK
Date: Tue, 07 Jun 2022 16:28:30 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=16070400; includeSubDomains
Last-Modified: Sat, 04 Jun 2022 11:42:36 GMT

When run ecapture in Kubernetes pod, ecapture is given --libssl, --pthread to specific lib location, but ecapture continues to auto detect gnutls lib and result in error 2022/06/05 16:46:48 pid info :3305486 2022/06/05 16:46:48 start to run EBPFProbeOPENSSL module 2022/06/05 16:46:48 start to run EBPFProbeGNUTLS module <==== 2022/06/05 16:46:48 lstat /etc/ld.so.conf: no such file or directory <=== 2022/06/05 16:46:48 invalid argument <==== see gojue#69 add single-mode to stop the auto detection of gnutls lib when we only want to trace libssl connection Signed-off-by: Vincent Li <vincent.mc.li@gmail.com>

When run ecapture in Kubernetes pod, ecapture is given --libssl, --pthread to specific lib location, but ecapture continues to search gnutls lib and result in error 2022/06/05 16:46:48 pid info :3305486 2022/06/05 16:46:48 start to run EBPFProbeOPENSSL module 2022/06/05 16:46:48 start to run EBPFProbeGNUTLS module <==== 2022/06/05 16:46:48 lstat /etc/ld.so.conf: no such file or directory <=== 2022/06/05 16:46:48 invalid argument <==== see gojue#69 add nosearch argument to require specifying lib path for each TLS modules when nosearch is true. this solves problems when auto search lib path failed for ecapture running in container that standard lib path does not exist. Signed-off-by: Vincent Li <vincent.mc.li@gmail.com>

fixed: #69 ecapture in container fail to locate pthread lib. output with "invalid argument" Signed-off-by: CFC4N <cfc4n.cs@gmail.com>

cfc4n · 2022-06-09T04:47:20Z

@vincentmli PR merged, please test it again.

vincentmli mentioned this issue Jun 5, 2022

RFE: trace TLS in container #65

Closed

cfc4n added the 🐞 bug Something isn't working label Jun 6, 2022

cfc4n self-assigned this Jun 6, 2022

vincentmli mentioned this issue Jun 7, 2022

Add nosearch argument to skip auto search lib path #70

Merged

cfc4n added a commit that referenced this issue Jun 8, 2022

user : bug fixed.

97d8216

fixed: #69 ecapture in container fail to locate pthread lib. output with "invalid argument" Signed-off-by: CFC4N <cfc4n.cs@gmail.com>

cfc4n closed this as completed in 58c7d62 Jun 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ecapture in container fail to locate pthread lib #69

ecapture in container fail to locate pthread lib #69

vincentmli commented Jun 5, 2022

vincentmli commented Jun 6, 2022

vincentmli commented Jun 7, 2022

cfc4n commented Jun 7, 2022

vincentmli commented Jun 7, 2022 •

edited

Loading

vincentmli commented Jun 7, 2022

cfc4n commented Jun 9, 2022

ecapture in container fail to locate pthread lib #69

ecapture in container fail to locate pthread lib #69

Comments

vincentmli commented Jun 5, 2022

vincentmli commented Jun 6, 2022

vincentmli commented Jun 7, 2022

cfc4n commented Jun 7, 2022

vincentmli commented Jun 7, 2022 • edited Loading

vincentmli commented Jun 7, 2022

cfc4n commented Jun 9, 2022

vincentmli commented Jun 7, 2022 •

edited

Loading