Kernel config read failed, error:Config not found #117

vincentmli · 2022-07-02T16:42:40Z

Describe the bug

the most recent ecapture build from master branch breaks with error

# ./bin/ecapture 
2022/07/02 12:37:53 Kernel config read failed, error:Config not found,  item:CONFIG_BPF.

To Reproduce
Steps to reproduce the behavior:

git clone the ecapture master branch
make
./bin/ecapture

Expected behavior

should work

Screenshots

Linux Server/Android (please complete the following information):

OS: centos 8 with custom kernel build
Arch: x86
Kernel Version: 5.18.0-rc3+

Additional context
ecapture works before on this same machine with same kernel build

The text was updated successfully, but these errors were encountered:

cfc4n · 2022-07-03T07:08:05Z

Can old version works? like v0.1.10 .

I think that CONFIG_BPF config was disabled. you can check it with cat /boot/config

cfc4n@vm-server:~$ cat /boot/config-`uname -r`|grep CONFIG_BPF
CONFIG_BPF=y
CONFIG_BPF_LSM=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
# CONFIG_BPF_PRELOAD is not set
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=

vincentmli · 2022-07-03T14:56:15Z

0.1.10 works, here is my kernel config, I suspect commit c425dea41a0d introduced a bug

egrep 'BPF|BTF' /usr/src/bpf-next/.config

CONFIG_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
# BPF subsystem
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
# CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set
# CONFIG_BPF_PRELOAD is not set
# CONFIG_BPF_LSM is not set
# end of BPF subsystem
CONFIG_CGROUP_BPF=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
# CONFIG_BPFILTER is not set
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
# CONFIG_BPF_STREAM_PARSER is not set
# CONFIG_LWTUNNEL_BPF is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_PAHOLE_HAS_SPLIT_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y
# CONFIG_MODULE_ALLOW_BTF_MISMATCH is not set
CONFIG_BPF_EVENTS=y
# CONFIG_BPF_KPROBE_OVERRIDE is not set
# CONFIG_TEST_BPF is not set

cfc4n · 2022-07-03T15:07:16Z

maybe.
kernel config filepath is one of https://github.com/ehids/ecapture/blob/e083d55e2ef52b1232ad92e4684907118fa1da9e/pkg/util/ebpf/bpf_linux.go#L31-L35

but , why is this ?

vincentmli · 2022-07-03T15:14:54Z

my kernel is custom build, I don't have /boot/config* for my running kernel, I do have /sys/kernel/btf/vmlinux

 configPaths = []string{ 
 	"/proc/config.gz", 
 	"/boot/config", 
 	"/boot/config-%s", 
 }

ls -l /boot/conf* old kernel config from centos distro

-rw-r--r--. 1 root root 187648 May  8  2020 /boot/config-4.18.0-193.el8.x86_64
-rw-r--r--. 1 root root 192095 Jun  1  2021 /boot/config-4.18.0-305.3.1.el8.x86_64

my custom running kernel [root@centos-dev ecapture]# uname -a

Linux centos-dev.localdomain 5.18.0-rc3+ #90 SMP PREEMPT_DYNAMIC Mon May 23 12:55:31 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

vincentmli · 2022-07-03T16:25:04Z

@cfc4n to resolve my problem, I found the fix with diff below, my custom kernel has config located here /lib/modules/$(uname -r)/build/.config, in general linux distribution should have either /proc/config.gz or /boot/config-%s, so I guess my custom kernel build is kind of special, I can send a PR if that is ok for you.

diff --git a/pkg/util/ebpf/bpf_linux.go b/pkg/util/ebpf/bpf_linux.go
index a8dad21..396cb9b 100644
--- a/pkg/util/ebpf/bpf_linux.go
+++ b/pkg/util/ebpf/bpf_linux.go
@@ -32,6 +32,8 @@ var (
                "/proc/config.gz",
                "/boot/config",
                "/boot/config-%s",
+               "/boot/config-%s",
+               "/lib/modules/%s/build/.config",
        }
 )

cfc4n · 2022-07-03T16:41:08Z

wait...

/lib/modules/%s/build/.config is a standard path? do you have official documentation in linux handle? or a link?

vincentmli · 2022-07-03T16:57:14Z

"/lib/modules/%s/build/.config",

I could not find official documentation yet, according to my understanding, when I build custom kernel from kernel source, the /lib/modules/%s/build is a symbolic link to the kernel source tree build directory, for example, I always run most recent bpf-next kernel tree and cloned to /usr/src/bpf-next directory, after I make modues_install and reboot into my custom kernel build, I can see the symbolic link:

ls -l /lib/modules/uname -r/build

lrwxrwxrwx 1 root root 17 May  6 00:49 /lib/modules/5.18.0-rc3+/build -> /usr/src/bpf-next

this is for my centos 8 with custom kernel build

I also checked ubuntu 22.04 machine that is upgraded from ubuntu 20.04, so the kernel is upgraded from the distro, I don't have custom kernel build on this machine, but it has the same symbolic link

ls -l /lib/modules/uname -r/build

lrwxrwxrwx 1 root root 40 Jun  9 14:36 /lib/modules/5.15.0-39-generic/build -> /usr/src/linux-headers-5.15.0-39-generic

it seems to be safe to assume /lib/modules/`uname -r`/build/.config could exist for running kernel

vincentmli · 2022-07-03T17:11:41Z

I think I speak too early, the fix i mentioned only fix ecapture running in the host VM machine, if I put the ecapture in container like netshoot pod, it failed to run with same error. so the fix is not good, ecapture 0.1.10 works in container fine. maybe we need to think about if the configPaths is needed? does andriod require configPaths?

cfc4n · 2022-07-03T17:17:15Z

please wait, I'm release new version. something need to debug..

vincentmli · 2022-07-04T01:37:21Z

fyi, i recompiled the kernel with /proc/config.gz support, ecapture still failed with same error, both failed in host and in container

[root@cilium-dev var]# uname -a
Linux cilium-dev 5.18.0+ #3 SMP PREEMPT_DYNAMIC Sun Jul 3 20:57:54 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

[root@cilium-dev var]# ls -l /proc/config.gz 
-r--r--r-- 1 root root 34436 Jul  3 21:09 /proc/config.gz

vincentmli · 2022-07-04T02:45:45Z

fyi, i recompiled the kernel with /proc/config.gz support, ecapture still failed with same error, both failed in host and in container
[root@cilium-dev var]# uname -a
Linux cilium-dev 5.18.0+ #3 SMP PREEMPT_DYNAMIC Sun Jul 3 20:57:54 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

[root@cilium-dev var]# ls -l /proc/config.gz 
-r--r--r-- 1 root root 34436 Jul  3 21:09 /proc/config.gz

there is problem with the code, line 47 append kernel version number to /proc/config.gz or /boot/config-%s, but /proc/config.gz does not have kernel version number, so even if user has /proc/config.gz supported in kernel config, /proc/config.gz will not be found, this is the first problem I found

 31         configPaths = []string{
 32                 "/proc/config.gz",
 33                 "/boot/config",
 34                 "/boot/config-%s",
 35         }
 36 )
 37 
 38 func GetSystemConfig() (map[string]string, error) {
 39         var KernelConfig = make(map[string]string)
 40 
 41         i, e := getOSUnamer()
 42         if e != nil {
 43                 return KernelConfig, e
 44         }
 45 
 46         for _, system_config_path := range configPaths {
 47                 bootConf := fmt.Sprintf(system_config_path, i.Release)
 48                 KernelConfig, e = getLinuxConfig(bootConf)
 49                 if e != nil {
 50                         continue
 51                 }
 52 
 53                 if len(KernelConfig) > 0 {
 54                         break
 55                 }
 56         }

the second problem i found when I hacked the code to bypass line 47 bootConf := fmt.Sprintf(system_config_path, i.Release) for /proc/config.gz, the line 48 getLinuxConfig(bootConf) still failed to parse /proc/config.gz for some reason and KernelConfig is empty

cfc4n · 2022-07-04T04:57:17Z

yes ,the code here is flawed. I'll send a PR later. That is my fault, thannnnkkkksssss.

Signed-off-by: CFC4N <cfc4n.cs@gmail.com>

pkg : fix Kernel config read failed, error:Config not found #117

vincentmli · 2022-07-05T14:10:34Z

@cfc4n I am sorry to tell you that I tested again with your fix #123, it did not fix the problem on my VM with custom kernel build with /proc/config.gz support :(.

[root@centos-dev ecapture]# ls -l /proc/config.gz 
-r--r--r-- 1 root root 34270 Jul  4 02:21 /proc/config.gz

./bin/ecapture 
2022/07/05 10:07:11 Kernel config read failed, error:KernelConfig not found.

also I am concerned even if we fixed /proc/config.gz for ecapture running in host, it may not work for ecapture in k8s container environment, /proc/config.gz is not supported in Ubuntu or Centos distribution, and in container environment, the container may has no access to /boot/config* neither, just a head up

now my workaround is to replace log.Fatalf with log.Printf in main.go when error:KernelConfig not found :)

cfc4n · 2022-07-05T14:49:32Z

eh.... That is scene I didn't expect while use on k8s container.

I'll try a new idea.

vincentmli · 2022-07-05T15:41:59Z

maybe add a new flag switch or something like that for andriod environment, restore the previous behavior that worked in container environment?

cfc4n · 2022-07-05T16:43:47Z

I don't think so.
New detection methods are necessary for container.

remove detection of BPF config when running at container. In Github Action release.yml , qemu in docker ,do not have BPF config , eCapture cant works with ecapture -h. other issue : #117 Signed-off-by: CFC4N <cfc4n.cs@gmail.com>

cfc4n added a commit that referenced this issue Jul 5, 2022

pkg : fix Kernel config read failed, error:Config not found #117

6de35fa

Signed-off-by: CFC4N <cfc4n.cs@gmail.com>

cfc4n linked a pull request Jul 5, 2022 that will close this issue

pkg : fix Kernel config read failed, error:Config not found #117 #123

Merged

cfc4n closed this as completed in #123 Jul 5, 2022

cfc4n added a commit that referenced this issue Jul 5, 2022

Merge pull request #123 from ehids/config-read-failed

c96bffd

pkg : fix Kernel config read failed, error:Config not found #117

cfc4n mentioned this issue Jul 7, 2022

remove detection of BPF config when running at container #127

Closed

cfc4n mentioned this issue Jul 7, 2022

ebpf: remove detection of BPF config when running at container #127 #128

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kernel config read failed, error:Config not found #117

Kernel config read failed, error:Config not found #117

vincentmli commented Jul 2, 2022

cfc4n commented Jul 3, 2022 •

edited

Loading

vincentmli commented Jul 3, 2022

cfc4n commented Jul 3, 2022

vincentmli commented Jul 3, 2022 •

edited

Loading

vincentmli commented Jul 3, 2022 •

edited

Loading

cfc4n commented Jul 3, 2022

vincentmli commented Jul 3, 2022

vincentmli commented Jul 3, 2022

cfc4n commented Jul 3, 2022

vincentmli commented Jul 4, 2022

vincentmli commented Jul 4, 2022

cfc4n commented Jul 4, 2022

vincentmli commented Jul 5, 2022 •

edited

Loading

cfc4n commented Jul 5, 2022

vincentmli commented Jul 5, 2022

cfc4n commented Jul 5, 2022

Kernel config read failed, error:Config not found #117

Kernel config read failed, error:Config not found #117

Comments

vincentmli commented Jul 2, 2022

cfc4n commented Jul 3, 2022 • edited Loading

vincentmli commented Jul 3, 2022

cfc4n commented Jul 3, 2022

vincentmli commented Jul 3, 2022 • edited Loading

vincentmli commented Jul 3, 2022 • edited Loading

cfc4n commented Jul 3, 2022

vincentmli commented Jul 3, 2022

vincentmli commented Jul 3, 2022

cfc4n commented Jul 3, 2022

vincentmli commented Jul 4, 2022

vincentmli commented Jul 4, 2022

cfc4n commented Jul 4, 2022

vincentmli commented Jul 5, 2022 • edited Loading

cfc4n commented Jul 5, 2022

vincentmli commented Jul 5, 2022

cfc4n commented Jul 5, 2022

cfc4n commented Jul 3, 2022 •

edited

Loading

vincentmli commented Jul 3, 2022 •

edited

Loading

vincentmli commented Jul 3, 2022 •

edited

Loading

vincentmli commented Jul 5, 2022 •

edited

Loading