Android ARM support

[docbrown]

I would like to use LKL in an Android app, so I added elf32-littlearm as a supported POSIX output format in the Makefiles and attempted to build with a standalone arm-linux-androideabi toolchain, but I get this:

arm-linux-androideabi-gcc -Iinclude -Wall -g -O2 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -fno-strict-aliasing -fPIC -c -o lib/net.o lib/net.c
In file included from /home/david/Android/toolchains/arm-linux-androideabi-4.9-21/sysroot/usr/include/linux/signal.h:21:0,
                 from /home/david/Android/toolchains/arm-linux-androideabi-4.9-21/sysroot/usr/include/signal.h:48,
                 from /home/david/Android/toolchains/arm-linux-androideabi-4.9-21/sysroot/usr/include/sys/select.h:35,
                 from /home/david/Android/toolchains/arm-linux-androideabi-4.9-21/sysroot/usr/include/unistd.h:34,
                 from include/lkl/linux/shm.h:6,
                 from include/lkl/asm/syscalls.h:81,
                 from include/lkl.h:6,
                 from include/lkl_host.h:5,
                 from lib/net.c:5:
include/lkl/asm-generic/signal.h:103:22: error: expected ':', ',', ';', '}' or '__attribute__' before '.' token
  __lkl__sighandler_t sa_handler;
                      ^
<builtin>: recipe for target 'lib/net.o' failed
make: *** [lib/net.o] Error 1

I'm not a kernel developer at all, so I don't really know where to go from here.

Hi David,

Thanks for reporting the issue. I'll try to debug this in the next couple of days but can you please share the changes you made to the build system?

Thanks,
Tavi

[docbrown]

Here's a diff of my changes:

diff --git a/arch/lkl/Makefile b/arch/lkl/Makefile
index cd4ae2a..9fe481a 100644
--- a/arch/lkl/Makefile
+++ b/arch/lkl/Makefile
@@ -2,7 +2,7 @@ include arch/lkl/auto.conf

 KBUILD_CFLAGS += -fno-builtin

-ifneq (,$(filter $(OUTPUT_FORMAT),elf64-x86-64 elf64-x86-64-freebsd))
+ifneq (,$(filter $(OUTPUT_FORMAT),elf32-littlearm elf64-x86-64 elf64-x86-64-freebsd))
 KBUILD_CFLAGS += -fPIC
 else ifeq ($(OUTPUT_FORMAT),pe-i386)
 prefix=_
diff --git a/tools/lkl/Makefile b/tools/lkl/Makefile
index e3bd8b2..93caf54 100644
--- a/tools/lkl/Makefile
+++ b/tools/lkl/Makefile
@@ -16,7 +16,7 @@ TEST_TARGETS=test valgrind gdb
 lib_source = $(filter-out %-host.c,$(wildcard lib/*.c))
 source = $(wildcard tests/*.c)
 lib = lib/liblkl.a
-ifneq (,$(filter $(OUTPUT_FORMAT),elf64-x86-64 elf32-i386 elf64-x86-64-freebsd))
+ifneq (,$(filter $(OUTPUT_FORMAT),elf32-littlearm elf64-x86-64 elf32-i386 elf64-x86-64-freebsd))
 source += $(wildcard *.c)
 lib_source += lib/posix-host.c
 CFLAGS += -fPIC

It's nothing more than adding elf32-littlearm as a supported output format for the POSIX host library. I knew it surely wouldn't be that simple! 😅

[pscollins]

It looks like this typedef is not getting compiled (tools/lkl/include/lkl/asm-generic/signal-defs.h after a build):

#ifndef __ASSEMBLY__
typedef void __lkl__signalfn_t(int);
typedef __lkl__signalfn_t *__lkl__sighandler_t;

typedef void __lkl__restorefn_t(void);
typedef __lkl__restorefn_t *__lkl__sigrestore_t;

#define LKL_SIG_DFL ((__lkl__sighandler_t)0)    /* default signal handling */
#define LKL_SIG_IGN ((__lkl__sighandler_t)1)    /* ignore signal */
#define LKL_SIG_ERR ((__lkl__sighandler_t)-1)   /* error return from signal */
#endif

No idea why the ARM toolchain is defining __ASSEMBLY__ in different places, though.

[xjia1]

I've seen similar errors when integrating LKL with some existing app. #include <lkl.h> after things like #include <signal.h> triggered that error.

The problem is that host includes do things like

#define sa_handler _u._sa_handler

and sa_handler is a member in a Linux/LKL structure and gets replaces by the above macro.

The way I fixed these was to #undef them before including LKL headers, but in this case <sys/signal.h> is pulled from <stdint.h> which is pulled from lkl/linux/virtio_ring.h.

I am going to try and define these in arch/lkl/include/uapi/syscalls.h to force a _lkl prefix by the install headers script in a couple of hours. This willl allow us to get rid of the undefs in the application code and make things cleaner overall.

I just pushed a few changes that makes LKL compile with the arm-linux-androideabi- toolchain. However, when trying the test/boot on Android emulator (KitKat) I get a seg fault. I will continue to look into this.

[thehajime]

I will try to add a circle ci instance for the particular target.

[phhusson]

It also segfaults here using 5.1 NDK/5.1 device.
I tried building it with -fno-PIC -fno-PIE but it doesn't help.
It looks like a relocation error: the stack trace shows access to 0x701c8c, which sounds too low to be real.
This address is in .debug_info section.
If I try to dump assembly of lkl.o, the debug informations are totally wrong. It is quoting code from exit section, while it is in init section.
Android >=5.0's linker enforces to have PIE executables, so I had to make the binary static to be able to execute the binary.

As for virtio_ring.h, it seems fixed by my PR which has been merged.

Edit: For my usecase, I can just use a standard ARM cross-compiler and staticly link, so I think I'll stick to that.

[pscollins]

@phhusson Where in the initialization process does it segfault? I know @stfairy dealt with a linker bug in gold at one point, conceivably it's related.

[phhusson]

#0  0x001cefe4 in vsnprintf (buf=0x3e0 <exit_nls_cp866+8> <Address 0x3e0 out of bounds>, size=0, fmt=0xffffffff <Address 0xffffffff out of bounds>, args=...) at lib/vsprintf.c:1877
#1  0x001cf88c in vscnprintf (buf=0x4853cc <textbuf.31283> "", size=992, fmt=0x701c8c <Address 0x701c8c out of bounds>, args=...) at lib/vsprintf.c:2027
#2  0x000366fc in vprintk_emit (facility=0, level=-1, dict=0x0, dictlen=4143971644, fmt=0x701c8c <Address 0x701c8c out of bounds>, args=...) at kernel/printk/printk.c:1721
#3  0x00036d1c in vprintk_default (fmt=0x4853cc <textbuf.31283> "", args=...) at kernel/printk/printk.c:1860
#4  0x002e7424 in printk (fmt=0x701c8c <Address 0x701c8c out of bounds>) at kernel/printk/printk.c:1910
#5  0x00316450 in start_kernel () at init/main.c:527
#6  0x003173dc in lkl_run_kernel (arg=0x701c8c) at arch/lkl/kernel/setup.c:45
#7  0x002f0014 in __pthread_start (arg=0xf7019080, arg@entry=<error reading variable: value has been optimized out>) at bionic/libc/bionic/pthread_create.cpp:141
#8  0x002f4f92 in __start_thread (fn=<optimized out>, arg=<optimized out>) at bionic/libc/bionic/clone.cpp:41
#9  0x00000000 in ?? ()

Looks like it is the first place where it tries to access to rodata. I'll try adding code earlier accessing to data

[pscollins]

Yes, this is the same bug, it's a problem in the linker. As a workaround you might want to try commenting out that line (IIRC it's printing a startup message), or switching to a different linker. @stfairy can probably say more.

[phhusson]

Yes it's printing a startup message, but it most likely means there are other problems.
Though adding a static int v=0; static int w=1; const int x=1; and accessing them didn't seem to crash.

I added -fuse-ld=bfd, and it didn't change a thing?!? (the file size is the same and I still have got the same crash)
I built with V=1 and tried to build with --threads, and I'm really sure I was linking with bfd.

FIY, my changes are, in tools/lkl/Makefile:
CFLAGS += -fuse-ld=bfd
LDFLAGS="-fuse-ld=bfd" make -C ... install

[xjia1]

I have a fix at hand but I still don't fully understand it. My best guess is that gold doesn't work well with linker scripts containing multiple input sections for an output section.

The fix is binutils-2.24/gold/object.cc around L2310: os->output_address() returns a virtual address, while what we actually want is an offset starting from the .rodata section. So we need to subtract that, making the code to be:

          else if (!lv_in->is_section_symbol())
            {
              // This is not a section symbol.  We can determine
              // the final value now.
              lv_out->set_output_value(
                  os->output_address(this, shndx, lv_in->input_value()) - os->address());
            }

[thehajime]

now you will see a (the?) seg fault in circleci.

https://circleci.com/gh/lkl/linux/328

I again disabled 3rd instance for anrdoid build and test until we will be ready.