IRQ #54 disabling on Edison 64-bit kernel

[amcduffee]

I have built a 64-bit 4.9 kernel from your 'eds' (8624493) branch and shortly after booting all the way to a command prompt I get an IRQ disabled stack trace.

dmesg.txt
edison64_config.txt
proc_interrupts.txt

Let me know what other details I should gather.

[andy-shev]

Yeah, I know about the issue. If you are okay not to use DMA for UART, you may try to avoid that. Requires a bit of a hacking code (8250_mid.c: comment out dma_dev = ... in tng_setup() function).

[VladimirAkopyan]

Is making a 64 bit kernel relatively straight forward? I've been using Edison for a while, but never had to compile a kernel. We happen to need a 64 bit version.
About this - we do use UART, would it work fine without DMA? Should we expect a fix any time soon?

[andy-shev]

@VladimirAkopyan

1. Yes, you may take my branch and compile as usual. No magic or hidden stuff, just make x86_64_defconfig && make -j8. Be aware, you have to update U-Boot to boot it directly.
2. It will work fine as far as you okay with CPU performance degradation (since each block between 1 ..FIFO_size / 2 character(s) will require to be served through normal interrupt).
3. My work is unofficial, no expectations at all, sorry. Of course I would much appreciate any help regarding to hunt the issue root cause.

[andy-shev]

It seems I found a root cause. But let me test this better and I will submit patches to my branch soon for getting more tests from you.

[andy-shev]

So, I just have updated my branch. Will very appreciate broad testing!

[VladimirAkopyan]

Cool! thanks. I've never built a kernel before, but i'll look into it and see how far I can get

[htot]

I would like to try. But I need to know how toggle gpio mmapped on the Edison Arduino board, then remove mraa from my code. Any hints?

[amcduffee]

@andy-shev I don't see anymore IRQ torvalds#54 traces in my initial checks of the new 64-bit kernel, but have only looked briefly. I will do a more lengthy test next week.

[andy-shev]

@htot, from this I barely understand what is the issue you try to resolve? Better if you provide some piece of code. And either https://communities.intel.com/community/tech/edison or https://stackoverflow.com/questions/tagged/intel-edison would be more suitable for such discussion.

[andy-shev]

@amcduffee, will wait for you comprehensive tests, thanks!

[htot]

@andy-shev https://github.com/htot/hs_uart I need to toggle to 2 gpio pins to a get trigger for the oscilloscope. I am using mmapped to prevent large delays (reduces from 50us to 2 us, relevant when transmitting at 2Mb/s). So need to replace all mraa stuff to be able to test with your kernel. I am particularly interested to test if transmission has the same transmission interchar gap every 45 msec as the standard 3.10.14 kernel.

[andy-shev]

@htot, got it! So, you just need to use sysfs directly or, if it somehow ready, newly provided character device with associated IOCTLS. Here are the links to get a clue: a) sysfs interface, b) GPIO user space tools for character device (state of progress unknown)

[htot]

@andy-shev Thanks, so you say I could also dig through the mraa source code and use that as an example from here https://github.com/intel-iot-devkit/mraa/blob/master/src/x86/intel_edison_fab_c.c? That has the lines at the top:
// This is an absolute path to a resource file found within sysfs.
// Might not always be correct. First thing to check if mmap stops
// working. Check the device for 0x1199 and Intel Vendor (0x8086)
#define MMAP_PATH "/sys/devices/pci0000:00/0000:00:0c.0/resource0"

[andy-shev]

@htot, zOMG, I didn't realize that mraa is SO ugly! Mentioned device is GPIO controller. sysfs priovides standard ABI to communicate with (including all GPIO expanders that found on Edison/Arduino). OTOH pin controller stuff is not available in the user space. I would expect we can do the proper way to use it whenever we will switch to ACPI as resource provider on Edison (U-Boot).

[htot]

@andy-shev Right now the quick-and-dirty way would be good enough for my testing :-)

[amcduffee]

@andy-shev This issue appears fixed to me. Before the holiday weekend I spent about 20 minutes generating activity on the console without any IRQ 54 stack traces appearing. I also left the Edison module up over the weekend and after ~5 days of up time the dmesg log is still clean.

[andy-shev]

@amcduffee, cool! So, based on your test I close this issue, thanks!

P.S. Nevertheless, please try to run several times something like
dmesg > /dev/ttyS1 (before doing that, set the higher baud rate on the port stty -F /dev/ttyS1 921600).

[amcduffee]

I guess maybe I should have explained how I generated activity. I was using Minicom to run various commands (dmesg included) that generate a decent amount of text output. Since I was using Minicom the stdout, of the commands, was going over ttyS2.

Anyhow, I have now also checked the following speeds: 115200, 230400, 921600 and 2000000. All of them are working as expected. Also, just for fun I ran the following command at 2000000:
dd bs=1M count=16 if=/dev/zero | hexdump -Cv

It completed after ~7 minutes of zeros flying by in Minicom. No IRQ 54 stack traces on dmesg.

[VladimirAkopyan]

Amazin!

[andy-shev]

@amcduffee, yes, thanks! But please check with /dev/ttyS1. It's indeed important to me.