buildtest uses wrong build directory

[cladmi]

Description

When building using buildtest the build is not using the BOARD bin/$(BOARD) directory but the default BOARD defined in the application Makefile BOARD ?= native.

This also has consequences on dist/tools/compile_test/compile_test.py which is using buildtest.

EDIT: This only happens when setting variables on the command line as then the MAKEOVERRIDES from here are done even if these variables are not set.

RIOT/Makefile.include

Line 69 in c7894d2

MAKEOVERRIDES += $(foreach v,$(__DIRECTORY_VARIABLES),$(v)=$($(v)))

I think a solution is to only override variables which were set from the command line (by checking origin) before changing there value.

A better fix would be to never override but require variables to have been set absolute if set from the command line, (and maybe also environment) but it changes the behavior.

Steps to reproduce the issue

I am using #9741 to make it more visible during compilation but can be done without.

When building for iotlab-m3 the output is saved in bin/native.

$ git clean -xdff examples/hello-world

$ BOARDS=iotlab-m3 make -C examples/hello-world/ buildtest 'BUILDTEST_MAKE_REDIRECT='
...
Building application "hello-world" for "iotlab-m3" with MCU "stm32f1".

   text    data     bss     dec     hex filename
   8480     140    2740   11360    2c60 /home/harter/work/git/RIOT/examples/hello-world/bin/native/hello-world.elf

$ find examples/hello-world/bin/
examples/hello-world/bin/
examples/hello-world/bin/native
examples/hello-world/bin/native/hello-world.hex
examples/hello-world/bin/native/hello-world.elf
examples/hello-world/bin/native/hello-world.map

When running some compilations with buildtest I also sometime had strange issues with packages with native but was correct when build alone.
I think it is related to an issue with the package clean which was shown with this path collision.

Expected results

Building in bin/iotlab-m3

Actual results

Build in bin/native.

Versions

Ubuntu 16.04

[cladmi]

More information on this one, the issue is only present when providing variable configuration from the command line.

[cladmi]

What I should address is that directories are overridden only if necessary in this line:

RIOT/Makefile.include

Line 69 in c7894d2

MAKEOVERRIDES += $(foreach v,$(__DIRECTORY_VARIABLES),$(v)=$($(v)))

To not have different behavior when setting a variable on the command line.

[cladmi]

@MrKevinWeiss this is the tracking issue of #9831