Issue with building lttng version 2.10 from source using 4.14.91-rt49-v7+ Preempt_rt kernel

[NickSadjoli]

As stated in title, I'm currently having issues building lttng (https://lttng.org/docs/v2.10/#doc-building-from-source) from source using this specific kernel.

For the details, I installed the kernel by following the blogpost by leMariva on the installation for this specific PREEMPT_RT kernel: https://lemariva.com/blog/2018/07/raspberry-pi-preempt-rt-patching-tutorial-for-kernel-4-14-y.

Please do note that this meant that I was building the rt kernel in an external computer before transferring it and using it on the Raspberry Pi.

As I am aware that many have experienced problems using the Debian packages that you can get using apt-get (https://lttng.org/docs/v2.10/#doc-debian) on the RPi, as described by some forums like this one (https://stackoverflow.com/questions/52386448/linux-trace-tool-lttng-on-raspbian-kernel-tracer-not-available).

Since this is the case, I decided to try and build the lttng installation from source. However I then encountered the following error when executing the 'make' command in one of their cloned repositories:

pi@raspberrypi:~/BangEco_rvmscan/stagelight_project/lttng/lttng-modules-2.9.12 $ make make -C /lib/modules/4.14.91-rt49-v7+/build M=/home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules-2.9.12 CONFIG_LTTNG=m CONFIG_LTTNG_CLOCK_PLUGIN_TEST=m modules make[1]: *** /lib/modules/4.14.91-rt49-v7+/build: No such file or directory. Stop. Makefile:111: recipe for target 'modules' failed make: *** [modules] Error 2

Based on the error, I then checked the availability of the build directory that seems to be missing, only to find that the rt-version of the kernel seems to be creating a symbolic link back to my external PC, where it was built, as per seen below:

pi@raspberrypi:~/BangEco_rvmscan/stagelight_project/lttng/lttng-modules-2.9.12 $ ls -l /lib/modules/4.14.91-rt49-v7+/build lrwxrwxrwx 1 root root 35 Apr 4 20:46 /lib/modules/4.14.91-rt49-v7+/build -> /home/nick-sadjoli/rpi-kernel/linux

pi@raspberrypi:~/BangEco_rvmscan/stagelight_project/lttng/lttng-modules-2.9.12 $ ls -l /lib/modules/4.14.91-rt49-v7+/source lrwxrwxrwx 1 root root 35 Apr 4 20:46 /lib/modules/4.14.91-rt49-v7+/source -> /home/nick-sadjoli/rpi-kernel/linux
I then tried to install the raspberrypi-kernel-headers package via 'sudo apt-get install rasbperrypi-kernel-headers' command, only to then find out that there aren't any linux headers directory of the 4.14.91-rt49-v7+ that has been built, as the command only installs headers for the previous kernels that are still available within my RPi (I opted to keep the original kernel that was available with Rasbpian Stretch, to enable switching between them)

With all of these problems listed, are there any advice that you can give me for overcoming this issue? I would like to still use the current version of the PREEMPT_RT-patched kernel if possible, and avoid any kind of reverts that you might need to make this work.

Otherwise, is there an easy way to make sure that the current 4.14.91-rt49-v7+ directory can be stripped down to its basic header components that I can just stick into the /lib/modules in the RPi?

Thank you for your attention and looking forward to your feedback/replies!

[pelwell]

Have you tried compiling natively on the Pi? On a 3B or a 3B+ it doesn't take as long as you might think, and it would almost certainly solve this build problem.

[NickSadjoli]

Alright, I'll try to natively compile the RT-kernel on the Pi now. Will post the results here.

[NickSadjoli]

Alright after trying to build it locally it seems that there were some improvements: I was able to bypass the issue that I initially put in this thread.

However, now a new issue is popping up from the lttng modules, in the following error message below:
`In file included from /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/lttng-kprobes.c:31:0:
/home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/../blacklist/kprobes.h:19:4: error: #error "Your kernel is known to have buggy optimized kprobes implementation. Fixed by commit 0ac569b "ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction" in Linux. Disable CONFIG_OPTPROBES or upgrade your kernel."

error "Your kernel is known to have buggy optimized kprobes implementation. Fixed by commit 0ac569b "ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction" in Linux. Disable CONFIG_OPTPROBES or upgrade your kernel."

`
It seems that CONFIG_OPTPROBES is not really compatible with the lttng modules. Were there previous known issues related to this for the PREEMPT_RT version of the RPi Kernel that was reported?

I will be trying to change the .config file as per given by the error message. However, if there were previous known solutions around this issue, then it would be greatly appreciated.

Thanks!

[NickSadjoli]

@pelwell Sorry to bother you. But may I know.whether you have any suggestions for the errors this time? Thanks!

[pelwell]

From what I can see, the commit mentioned in the error message (0ac569b) is very simple and can just be cherry-picked from the rpi-5.0.y branch:

$ git cherry-pick 0ac569b

If, however, you've done a minimal clone (e.g. --depth 1) you may need to download the patch file for the commit and apply it manually:

$ git am whatever_you_called_it.patch

Then rebuild your kernel (it should be very quick this time) and try again. This counts as upgrading your kernel, so you shouldn't need to disable CONFIG_OPTPROBES.

[NickSadjoli]

Hello @pelwell,

Thank you so much for your reply. I'll try out the patch as you suggested and will report back with the results. May I confirm that this does not disable the RT functionality of the kernel?

In the meantime, may I confirm whether the CONFIG_OPTPROBES option is indeed unable to be turned off when doing a 'make menuconfig'? Thank you and looking forward to your reply

[pelwell]

May I confirm that this does not disable the RT functionality of the kernel?

No, not at all - it's just a bug fix.

In the meantime, may I confirm whether the CONFIG_OPTPROBES option is indeed unable to be turned off when doing a 'make menuconfig'?

Neither CONFIG_OPTPROBES nor CONFIG_HAVE_OPTPROBES have label strings, which seems to mean they can't be manually enabled or disabled - they can only be selected by other options or disabled by missing dependencies.

[NickSadjoli]

Sorry for the late reply.

Thank you for the replies once again @pelwell!

No, not at all - it's just a bug fix.

Thank you for your confirmation of this, as the RT functionality is going to be very crucial for my RPi application. If the patch is indeed only a bug fix then this is fine and I'll try this solution instead.

Noted on the CONFIG_OPTPROBES situation. That's quite unfortunate. Hope future versions of the RT kernel would allow turning off of this feature without affecting the other like it does now.

[NickSadjoli]

Hi again @pelwell!

I have tried your solution of cherry-picking. It initially seemed to work well, but then I got hit with this following error again, this time asking for an upgrade to commit numbered 0ac569b,

`In file included from /home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/lttng-kprobes.c:31:0:
/home/pi/BangEco_rvmscan/stagelight_project/lttng/lttng-modules/probes/../blacklist/kprobes.h:19:4: error: #error "Your kernel is known to have buggy optimized kprobes implementation. Fixed by commit 0ac569b "ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction" in Linux. Disable CONFIG_OPTPROBES or upgrade your kernel."

error "Your kernel is known to have buggy optimized kprobes implementation. Fixed by commit 0ac569b "ARM: 8834/1: Fix: kprobes: optimized kprobes illegal instruction" in Linux. Disable CONFIG_OPTPROBES or upgrade your kernel."

^~~~~

`
I have tried to do a git cherry-pick to this commit version, and even tried to redo the entire process via a head reset followed by another cherry-pick session, performing a 'make clean' to completely restart the entire make process, before cherry-picking again.... It still unfortunately failed with the same error for the lttng-module.

It is to be noted however, that when I tried to re-do the cherry-pick but this time utilizing the already mentioned kernel, it returned me the below message:
`pi@raspberrypi:~/rpi-kernel/linux $ git cherry-pick 0ac569b
On branch rpi-4.14.y-rt
Your branch is ahead of 'origin/rpi-4.14.y-rt' by 1 commit.
(use "git push" to publish your local commits)
You are currently cherry-picking commit 0ac569b.

nothing to commit, working tree clean
The previous cherry-pick is now empty, possibly due to conflict resolution.
If you wish to commit it anyway, use:

git commit --allow-empty

Otherwise, please use 'git reset'
pi@raspberrypi:~/rpi-kernel/linux $ git status
On branch rpi-4.14.y-rt
Your branch is ahead of 'origin/rpi-4.14.y-rt' by 1 commit.
(use "git push" to publish your local commits)
You are currently cherry-picking commit 0ac569b.
(all conflicts fixed: run "git cherry-pick --continue")
(use "git cherry-pick --abort" to cancel the cherry-pick operation)
`
Should I just do a 'git cherry-pick --continue' directly, or is there something that I need to do for the conflict resolution first?

Thank you and looking forward to your reply again @pelwell !

[pelwell]

git cherry-pick (like git am) will add a commit to your tree. The same commit will not apply a second time - the state has changed, and a dumb system would simply say that it doesn't matcg, but git is smart enough to say that all changes included in the commit have already been applied, leaving you with an empty commit.

I think that the problem isn't with how you applied the back-ported patch, but with the detection of the problem by lttng. It seems that the obvious course of action is not sufficient to fix the problem, and I can't spare the time to start building lttng just to debug it for you - sorry.

[NickSadjoli]

Sorry for the late reply @pelwell! I see, noted with the empty commit.

That's an unfortunate case on the lttng part. For what it's worth, I did ask about this issue with the lttng folks as well, and they said that the detection/kernel blacklist is actually working fine. See the link here:

https://lists.lttng.org/pipermail/lttng-dev/2019-April/028730.html

It also seems that other versions of the RT kernel not designed for the RPi (i.e. those that were created for Intel Platforms, like this one: https://kernel.googlesource.com/pub/scm/linux/kernel/git/rt/linux-stable-rt/+/refs/heads/v4.14-rt) work quite well with lttng. I guess this means either the RPi RT kernel needs to be properly updated to support lttng, or lttng needs to add proper drivers for RPi's Broadcomm architecture??

I'll be keeping this issue forum open until someone is able to help resolve this issue I guess. Hopefully this can be resolved in the future.

[NickSadjoli]

Hi @pelwell and anyone who's been reading this issue thread: An update on my end.

So I was ready to abandon the lttng module for my RPi due to these issues and thought of using another tool or other methods to do the tracing/testing of RT applications. But then I thought of trying out the previous version of lttng (i.e. version 2.9). I actually tried this out previously with just the vanilla 4.14.91-rt49-v7+ RT kernel for the RPi, which also gave a similar warning of buggy kernel implementation.

But when I re-tried the lttng 2.9 installation with the 4.14rt kernel that is patched with @pelwell's recommended patches, it worked smoothly and now I have a working installation of the lttng with my RPi's RT kernel.

This does mean that there seems to be a problem of installing the 2.10 version of the lttng onto the RPi Kernel, and thus the change of the issue thread to reflect the problem with this specific version instead. I'll still keep this issue thread open for now so that anyone who is looking into development of the 4.14rt RPi kernel might be able to look into this as well in the future.

For the rest who is trying to also use lttng with RPi's PREEMPT_RT kernel, I suggest to use the 2.9 version instead for now, with the patched version of the 4.14rt kernel if similar issues to this thread are encountered.

Thanks again @pelwell for your assistance so far and looking forward to any solutions or fixes from anyone in the future that can hopefully resolve this issue.

[JamesH65]

@NickSadjoli Are you happy for this issue to be closed?

[NickSadjoli]

Hi @JamesH65 .

Yes for now I think you can close the issue. If anyone is having a similar problem in the future then they should go ahead to this thread and re-open the issue a bit