Custom kernel module #589
Comments
|
+1 After attempting several avenues. I think I can nail this down to: If the dev image that is created as a result of the SDK guide had the ability to compile the linux kernel. I would be able to build my drivers. Currently, my attempts at this failed because of a lack of bc and perl (and that's as far as I got) inside the dev VM. Perhaps if these (and whatever else is needed) were added, we could use the VM as a spot to at least build .ko's that we could insmod via the OEM interfaces. Thoughts? |
|
I should add bc and perl to the dev image and container by default but for now you can install them and other things with the following sequence: ( Thereafter a usual |
|
I need this too. The NVIDIA driver to run cuda apps. A walk through example of adding that and building the developer image would be wonderful. I'm still trying to figure it out. |
|
A good blog post on compiling kernel modules (focused on out of tree, but the initial steps work for in-tree as well): http://tleyden.github.io/blog/2014/11/04/coreos-with-nvidia-cuda-gpu-drivers/ |
|
+1 I also need this because I run PPPoE inside containers and I need the kernel module. |
|
any update on this? coreos now is not using the linux repo. coreos kernel version is now >4 |
|
ok found a hint here http://coreos-dev.narkive.com/n9Yz1JzE/coreos-linux-kernel-tree-v3-19-0 |
|
@majidaldo - that does work and I've been using it for a while. The hard part is getting the right compiler version. It has to match exactly when you compile modules. |
|
@hookenz what are you referring to? how could it work since the old coreos linux repo does not go up to the current version 4+? i'm experimenting with the linux unmodified linux kernel repo based on the tleyden post..but no luck. do you have more particular steps? |
|
@majidaldo - It works from the official linux repo. Have a look at my Dockerfile here: This worked on CoreOS 681.2.0. It's really messy and difficult to get just the right build of compiler. I'm fully expecting to get stuck in that regard soon. |
|
@majidaldo - Can you tell me how I can do this? > why is everyone downloading the whole linux repo?? i'd just get the linux-x.x.x.tar.xz. It used to be possible but changes to github server in the last few years and I can't find that feature anymore. It annoyed me that I had to clone 1GB + repository when a 30+MB compressed archive is all I really needed. |
|
apparently a cuda thing. |
|
@majidaldo - Ahh, you're using CUDA too. You have to create the device files manually at boot. The driver doesn't create those. You should be able to put something together based on this: |
|
@majidaldo - Sorry I misread that. I don't know what you're doing different than I am, but that build script I linked to creates an nvidia_uvm module. As the saying goes these days - "works on my machine" :) sorry I can't really further. Maybe the guys from CoreOS will give us a better overview of the SDK and better still, use the NVidia driver as an example! I tried reading the ChromeOS docs about it and didn't get very far at all. |
|
@hookenz thx i can get nvidia_uvm to load if i do a deviceQuery from cuda samples. BUT i can't get the driver version from CUDA to install. i CAN install the nvidia kernel module in a way similar to your Dockerfile. i thought docker would alleviate 'works on my machine' problems :( |
|
success! see my fork . although it has nvidia's stuff the non-nvidia steps should be the same for any kernel module. https://github.com/majidaldo/coreos-nvidia/blob/master/Dockerfile i gave up on trying to figure out why the driver didn't install from cuda. but i think it might hve to do with the new 4.x linux kernel |
|
@majidaldo - Cool. I found earlier versions just needed the kernel driver. Unfortunately, the new kernel module gets somehow tied up with the driver versions so CUDA and the kernel drivers get tied to the base... it's frustrating. My hope is to have a container that will run on most base systems with the kernel driver baked in. I'm netbooting and managed to bake the driver into the image after unpacking it all and copying it in. It's a messy process. |
|
it's on my plate, but I haven't made progress on it yet. |
|
The latest I found is https://gist.github.com/marineam/9914debc25c8d7dc458f My end goal is to build NVIDIA drivers automatically as soon as a new CoreOS release (or kernel, really) is pushed, then store them somewhere in the cluster. I don't want to build things inside a bloated Ubuntu container and, ideally, I'd start from just the toolchain linked above and a tarball with the sources/headers. I don't want to build the whole OS, either. :-) I can contribute PRs and testing, especially if you can provide guidance on where you see things headed, namely what the builder looks like and what your recommended storage/delivery mechanism would be. I definitely wouldn't store drivers in etcd and would rather use something P2P-like. Maybe packaging them as containers, allowing |
|
I'm getting weird failures at the end of Any ideas? |
|
That sounds like it might be related to #1216. |
|
Maybe related, maybe not. I narrowed my case down to |
|
Running |
|
I even gave the CoreOS VM 3GB of RAM, still the same thing. From running |
|
I am currently doing things the manual way, mounting the image manually, etc. I had found that the KILL came from systemd-nspawn itself, in its main loop (because it even emitted a newline to stdout if that wasn't the last character input by the user). Anyway, it's still spewing a lot of emerge-related errors, although it eventually compiles a driver: |
|
This is even more broken now. |
|
I take back all that I said. Things have been working quite reliably with at least 1082 and 1097. I suspect part of my issues were caused by reusing developer containers across multiple attempts. I can now build NVIDIA modules with a couple of scripts (which we plan to release soon and would probably benefit from your feedback). There is life at the end of the tunnel. |
|
And things worked fine with 1109. What's the easiest way to keep up to date with releases? Just scrape the HTML from |
|
@therc |
|
I've created a documentation PR at coreos/docs#915. I also uploaded some notes and scripts on automating building custom modules when new kernels are detected: https://gist.github.com/dm0-/0db058ba3a85d55aca99c93a642b8a20 |
|
That would be nice if there was an official container image for each official version of CoreOS Linux that would include the kernel headers of the correct version and some compilers (I need gcc, llvm and clang). That would allow to compile kernel modules for a specific CoreOS Linux version using a free CI service such as CircleCI or SemaphoreCI, without having to deploy a CoreOS Linux VM in the CI pipeline (which is usually not possible in free CI services). I would use that for building eBPF/kprobes programs that depend on a specific kernel headers version. My workflow would be something like this script in CircleCI: curl https://coreos.com/releases/releases.json | parse_versions.sh > versions
for VERSION in $(cat versions) ; do
docker pull quay.io/coreos/coreos-kernel-headers:${VERSION}
docker run -v $PWD:/src \
quay.io/coreos/coreos-kernel-headers:${VERSION} \
gcc -o /src/foo-${VERSION} /src/foo.c
doneThen it would generate one compiled object for each CoreOS Linux version and I could publish all of them. |
Thursday Oct 16, 2014 at 13:29 GMT
Originally opened as https://github.com/coreos/coreos-overlay/issues/924
I would need a facility to add custom kernel modules to CoreOS.
I need to access an SDI-Card (professional video interface standard) from my CoreOS machines. I am running a TV playout software in a docker container, but I need the kernel module to output the video data.
CoreOS is perfect for my infrastructure needs, except for the missing facility to compile/load the needed kernel module for my Blackmagic SDI card.
I can imagine, that there is a general need for such a facility. Imagine people wanting to run a CUDA cluster on CoreOS. They would need to compile and install Nvidia modules for this as well.
The text was updated successfully, but these errors were encountered: