git-revision for released Kernel 3.10.25+

[deltaphi]

Using a sequence of dist-upgrade and rpi-update, my Pi running an Raspberry-Foundation Raspian image has arrived at booting Kernel 3.10.25+. I need to build additional kernel modules (namely spi-config and a bunch of CAN-Networking modules). However, I cannot find sources that will let me compile a matching kernel.
I have successfully built a Kernel 3.10.24+ that I believe would match the kernel part of the latest raspbian image by extracting the git hash from the firmware. However this is not the Kernel my Pi is running after the dist-upgrade.

Therefore I would like to know, what is the git hash/revision, the current linux-image-3.10.25+ package for raspbian was built from?

Steps To reproduce:

• Setup a cross compilation environment (Kubuntu 13.10 VM, git-clone raspberry/tools/)
• dist-upgrade Pi to 3.10.25+ (alternate running dist-upgrade and rpi-upgrade until nothing changes anymore)
• apt-get install linux-source-3.10
• Copy and extract the resulting /usr/src/linux-source-3.10.tar.xz to the cross compilation environment
• Copy and extract /proc/config.gz to linux-source-3.10/.config in cross compilation environment
• make oldconfig
• make
• make modules_install

Expected Output:
Modules for Kernel version 3.10.25+.

Actual Output:
Modules for Kernel version 3.10.25 (note the lacking "+").

[popcornmix]

What does "uname -a" report?

[deltaphi]

pi@raspberrypi ~ $ uname -a
Linux raspberrypi 3.10.25+ #622 PREEMPT Fri Jan 3 18:41:00 GMT 2014 armv6l GNU/Linux

In case it is of any concern:

pi@raspberrypi ~ $ cat /etc/issue
Raspbian GNU/Linux 7 \n \l

[licaon-kter]

zcat /proc/config.gz > ~/config.rpi-622 and work from that?

[deltaphi]

That is what I have tried so far - see my steps to reproduce. That will give me Kernel 3.10.25 instead of 3.10.25+, so I'm thinking that I am using the wrong source tree.

[popcornmix]

linux-source-3.10 is not the pi specific kernel source, but a generic debian kernel.
Try a git clone from here:
https://github.com/raspberrypi/linux

(http://elinux.org/RPi_Kernel_Compilation may be useful).

I believe the "+" is appended when you build from a git tree that is not tagged.
I think you don't get the "+" as your source didn't come from git.

You can find the firmware repo hash that matches your uname string from here:
https://github.com/raspberrypi/firmware/blob/3a4ba0c3964f8f0ecc2dd46246a83258d9e4046d/extra/uname_string

and then pick the corresponding githash:
https://github.com/raspberrypi/firmware/blob/3a4ba0c3964f8f0ecc2dd46246a83258d9e4046d/extra/git_hash

which contains:

16eb921a96db3dc3555a65b53b99c15753e6b770

which is here in linux source repo:
16eb921

[deltaphi]

That appears to be the correct kernel version for my Pi, thank you very much!

I have since figured out that I got this kernel not via dist-upgrade (which is why there is no point in looking for a source package) but via rpi-upgrade. So perhaps telling everyone in every tutorial to first run an rpi-upgrade is not the wisest idea, but oh well...

To help anyone else with the same problem who does not have the very excellent "popcornmix search engine" handy, this is (roughly) how to find the correct git revision - hopefully phrased in a way that helps both experts and newcomers:

• Run the following, preferably on your build machine: git clone --depth=20 https://github.com/raspberrypi/firmware.git
  (Depth of 20chosen by sharply calibrated gut feeling. Increase the depth if you need a really old kernel or have a lot of time on your hands)
• On you Pi: uname -a
  gives an output like the following:
  Linux raspberrypi 3.10.25+ fiq_enable=0: Unable to handle kernel NULL pointer dereference #622 PREEMPT Fri Jan 3 18:41:00 GMT 2014 armv6l GNU/Linux
• Extract what resembles the following substring:
  #622 PREEMPT Fri Jan 3 18:41:00 GMT 2014
• Head back to your clone of the firmware repository. Run git log --patch | less
• Search for the substring (may take a while, especially if you increased the depth. Searching in less is done by pressing '/'). You will likely find two matches. The one where the substring was added to the repository (the one with a '+' at the beginning of the line).
• Look around the area of the match. There will also be an edit to the file "extras/git_hash". The addition in that file is the git hash for the kernel, in my case the '16eb921a96db3dc3555a65b53b99c15753e6b770' that popcornmix already mentioned.
• Run git clone --depth=100 https://github.com/raspberrypi/linux/
  (again, increase depth if you have time and/or desire for dusty kernels; EDIT: Or if you are from the future and there have been many commits to the repository since the writing of this post.)
• In the linux-checkout: git checkout the-hash-that-you-got-from-above, i.e., git checkout 16eb921a96db3dc3555a65b53b99c15753e6b770

Et voila, Kernel Sources.

If anyone knows a better way of searching through all commits of a git repository, let me (and especially stackoverflow) know.

Edit: Now with proper markup.

[tz1]

Following the above to the letter, and doing the #486 gives me that same long 16...770 hash. Giving me:

$ git checkout 16eb921
fatal: reference is not a tree: 16eb921

It worked when I changed it to --depth=200. If someone else reads this, they may have to increase the depth further.

I don't think the provision of the GPL that insists sources corresponding to binaries meant that they should be buried, hidden, and hard to find among obscure files, hashes, and git commands. Yet it is nearly impossible to rebuild a released kernel image and binaries from sources - because it is nearly impossible to find the right sources in the massive archive. There are git features which could make finding the corresponding source easier. Please consider doing something along those lines.

[popcornmix]

@tz1 please explain how you would like it to be done.

[tuffnerdstuff]

@popcornmix Why don't you use tags? Or would it be possible to provide a linux headers package available via apt-get like other linux distros?

That would make my life so much easier. I was trying to compile a kernel module for one week now ... without luck. I immediately succeeded with arch arm (provides linux headers), but I prefer debian for embedded/server stuff.

[kenrestivo]

@popcornmix: I would like a linux-source-3.10.25+ package which corresponds exactly to the binary that is running. This is the Debian way. It also would be a lot more GPL-ish. Indeed, requiring hours of detective work in order to obtain the source certainly goes against the spirit, possibly even the letter of the GPL. If you want help generating such a package I'd be happy to.

[kenrestivo]

Though, it seems I was able to cheat by doing as advised here, thusly:

sudo ln -s /lib/modules/3.10-3-rpi/build /lib/modules/3.10.25+/build

So my particular itch looks like it is now scratched. Still think a proper source package or at least headers package would be more Debian-ish though, and would be willing to help if needed.

[popcornmix]

@kenrestivo
Sure, if you provide a script that produces a suitable kernel headers package, I could keep that up to date.
I don't know if a kernel source package makes sense when the code is on github. I could perhaps tag the Linux source tree on each firmware build to make finding the matching source easier. Suggestions of exactly what would be useful are welcome.

[tz1]

I have various things to speed up finding such in my pikernomatic project.
I have the hashes and scripts that fetch the right hash (that is what needs to be tagged), and some docs explaining how to find the "right" hash.

The project does more as it also fetches the tools for a pc build and does some patching for CAN, PiTFT, and a few other things, enabled or disabled in a setup script.

They might provide some base for what you want to do, or simply be helpful.

[tz1]

That project is at https://github.com/tz1/pikernomatic

[kenrestivo]

That's useful, thanks. Seems like you did a lot of the heavy lifting already.

[kenrestivo]

@popcornmix: a simple git tag (i.e. v3.10.25+) would be awesome, straightforward, easy to do, and probably the clearest, thanks.

[tz1]

There is a 'tag' function. It would help if there were a tag for each binary release, with either the date or version or anything - there are only a handful of binary releases, when hexxeh or the raspi foundation or someone else produces a binary, having a tag - the version number is probably sufficient, but something like '3.10.33+ cam support" would work. Or even just "binary release" since the date or other metadata is avilable. I don't know how or if the binary releases are coordinated with this tree. They seem to show up, but I found where they try to put the hash in their tree which links to this one.

[asb]

@kenrestivo the machinery to generate a the Debian packages is at github.com/asb/firmware. I would happily accept any patches to improve the situation.

[popcornmix]

How about tagging linux tree with:

git tag -a "`date +%Y%m%d_%H%M%S`" -m "`cat ~/firmware/extra/uname_string`"

With uname_string from https://github.com/raspberrypi/firmware/blob/master/extra/uname_string

[notro]

I see that git_hash and Module.symvers is now present in https://github.com/Hexxeh/rpi-firmware
Add .config as well, and then copy those three somewhere now that rpi-update deletes the downloaded files by default.

With that in place, make a kernel source install script like this: https://github.com/notro/rpi-firmware/blob/master/extra/install_kernel_source

Now it's very easy to build kernel modules: https://github.com/notro/fbtft-spindle/wiki/Building-kernel-modules

[popcornmix]

Do you need config? Is this not sufficient:
zcat /proc/config.gz

This will avoid deleting git_hash and Module.symver
sudo SKIP_REPODELETE=1 rpi-update

[notro]

Yes, I know of those.
It was just my input on a way to make it easy for everybody to build kernel modules. Not just the few that can dig up the bits and pieces that are needed.

[tz1]

The long-form tag would work fine. I might otherwise want something shorter, but it is better to have too much rather than too little.

The /proc/config.gz works, but has to be obtained on the pi itself. I cross compile since I don't want to wait hours. It would be useful to have the .config to avoid the step, but it hasn't been a problem.

Some things (like the PiTFT and other fbtft) need the kernel build since the backlight setting changes and that affects the tree. Also increasing the number of interrupts.

[popcornmix]

We always build the kernel using this config:
https://github.com/raspberrypi/linux/blob/rpi-3.12.y/arch/arm/configs/bcmrpi_defconfig
so that can be used as well.

[kenrestivo]

Great. For a "script" I'd like to use make-kpkg which is already in Debian (or a thin wrapper around it to fetch the right tag), and with which I'm already familiar, if everyone's OK with that. I hope to have some time tonight to try some builds and see how it goes.

[kenrestivo]

OK, it seems like based on the suggestions above, people are more interested in hand-rolling scripts instead of using the Debian tools. So, in general, before I spend too much time on this, does anyone but me care about making this follow Debian conventions and such as much as possible?

[deltaphi]

I would be interested in a solution "the debian way". It definitely removes the overhead of learning how to use yet another set of tools. Plus, having dpkg packets should ease the pain of juggling multiple kernel versions.

[popcornmix]

I think either a script or a debian package will be more useful than doing it manually.
Doing it the debian way would be preferable, so please continue with your investigation.

If you can explain how to run make-kpkg to produce the deb files, we can update that with firmware builds, and get it into @asb's repo so it's available to raspbian users.

[kenrestivo]

@deltaphi: Great. I built a kernel last night using make-kpkg. I'll have to do some more work tweezing the --revision and --append-to-version parameters to get the version naming right, but so far so good. Also I'll need to get it to include the ungzipped kernel in the package, and add some kind of post-install script that'll put it in /boot/kernel.img. Also, I built it using the Raspberry toolchain at https://github.com/raspberrypi/tools . Not sure if that's the most Debian-ish one to use, or if crosstools-ng would be better, etc.

[kenrestivo]

@popcornmix: Will do. As a first pass, I did this, but CAUTION it is not right yet, still have work to do:

     git clone  https://github.com/raspberrypi/linux/
     cd linux/
     git checkout 16eb921a96db3dc3555a65b53b99c15753e6b770
     git checkout  -b v3.10.25+
     export ARCH=arm
     export DEB_HOST_ARCH=armhf
     export CONCURRENCY_LEVEL=6
     export  EXTRAVERSION=-raspbian-20140107
     export CCPREFIX=/mnt/externalsata/raspberry/tools/arm-bcm2708/arm-bcm2708-linux-gnueabi/bin/arm-bcm2708-linux-gnueabi-
     cd /mnt/externalsata/raspberry/kernel/
     wget -O .config  http://where-can-i-get-the-3.10.25-release-config-not-the-3.XXX-config
     export PATH=$PATH:/mnt/externalsata/raspberry/tools/arm-bcm2708/arm-bcm2708-linux-gnueabi/bin
     fakeroot make-kpkg --arch arm --cross-compile arm-bcm2708-linux-gnueabi-  --append-to-version=-release --revision 1 kernel_image kernel_headers kernel_source

Is there a place I can reliably grab the release config (i.e. the 3.10.25+, not an old version, in the case of the 2014-01-07 release) off of the internet without having to scp it from a running Pi? As noted above, I also need to add in the uncompressed kernel to the package, and a post-install script (part of the Debian package) to install it in /boot. I also want to try fakeroot-ng as an alternative to eliminate some spurious error messages.

[notro]

I have made a Linux kernel source installer: http://www.raspberrypi.org/forums/viewtopic.php?f=29&t=76261

[bullshit]

@kenrestivo i created a gist https://gist.github.com/bullshit/8001ce5eb8f03f056199/
@notro 👍