IAR build failure when building mbed-os and examples

[0xc0170]

Description

• Type: Bug
• Priority: Major

Bug

This command fails to compile (with traceback) (here is the job (internal link) http://mbed-ci-master-2.austin.arm.com:8081/job/examples_build_matrix/783/target=NUCLEO_F767ZI,toolchain=IAR ):

06:22:52 [DEBUG] Compile: C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm --cpu Cortex-M7 --fpu VFPv5 -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M7 -DARM_MATH_CM7 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F767ZI\IAR\.includes_5a547c6d82611f64c4e2715b8a0e2ca1.txt -o .\BUILD\NUCLEO_F767ZI\IAR\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.o .\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.S
06:22:52 [DEBUG] Return: 3
06:22:52 Traceback (most recent call last):
06:22:52   File "C:\mj\workspace\bm_wrap\1063\examples\mbed-os-example-mesh-minimal\mbed-os\tools\make.py", line 280, in <module>
06:22:52     toolchain))
06:22:52   File "C:\mj\workspace\bm_wrap\1063\examples\mbed-os-example-mesh-minimal\mbed-os\tools\build_api.py", line 538, in build_project
06:22:52     objects = toolchain.compile_sources(resources, resources.inc_dirs)
06:22:52   File "C:\mj\workspace\bm_wrap\1063\examples\mbed-os-example-mesh-minimal\mbed-os\tools\toolchains\__init__.py", line 923, in compile_sources
06:22:52     return self.compile_queue(queue, objects)
06:22:52   File "C:\mj\workspace\bm_wrap\1063\examples\mbed-os-example-mesh-minimal\mbed-os\tools\toolchains\__init__.py", line 986, in compile_queue
06:22:52     raise ToolException(err)
06:22:52 ToolException
06:22:52 [mbed] Working path "C:\mj\workspace\bm_wrap\1063\examples\mbed-os-example-mesh-minimal" (program)

If you compare this command to other examples, the same command, for comparison:

Compile: C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm --cpu Cortex-M7 --fpu VFPv5 -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M7 -DARM_MATH_CM7 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F767ZI\IAR\.includes_0f02e5fe9f3d74c786cba96e00bda7d5.txt -o .\BUILD\NUCLEO_F767ZI\IAR\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.o .\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.S
06:23:32 [DEBUG] Return: 0

For nucleo f429zi, it is different assembly file (here is the job (internal link) http://mbed-ci-master-2.austin.arm.com:8081/job/pr_pipeline/886/)

17:10:17 Compile [  0.2%]: cmain.S
17:10:17 [DEBUG] Compile: C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm --cpu Cortex-M4F -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M4 -DUSB_STM_HAL -DARM_MATH_CM4 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F429ZI\IAR\.includes_c45dbc904e060289cb07543bf5b1e267.txt -o .\BUILD\NUCLEO_F429ZI\IAR\mbed-os\cmsis\TOOLCHAIN_IAR\cmain.o .\mbed-os\cmsis\TOOLCHAIN_IAR\cmain.S
17:10:17 [DEBUG] Return: 3
17:10:17 Traceback (most recent call last):
17:10:17   File "C:\mj\workspace\bm_wrap\953\examples\mbed-os-example-mesh-minimal\mbed-os\tools\make.py", line 280, in <module>
17:10:17     toolchain))
17:10:17   File "C:\mj\workspace\bm_wrap\953\examples\mbed-os-example-mesh-minimal\mbed-os\tools\build_api.py", line 536, in build_project
17:10:17     objects = toolchain.compile_sources(resources, resources.inc_dirs)
17:10:17   File "C:\mj\workspace\bm_wrap\953\examples\mbed-os-example-mesh-minimal\mbed-os\tools\toolchains\__init__.py", line 909, in compile_sources
17:10:17     return self.compile_queue(queue, objects)
17:10:17   File "C:\mj\workspace\bm_wrap\953\examples\mbed-os-example-mesh-minimal\mbed-os\tools\toolchains\__init__.py", line 972, in compile_queue
17:10:17     raise ToolException(err)
17:10:17 ToolException
17:10:17 [mbed] Working path "C:\mj\workspace\bm_wrap\953\examples\mbed-os-example-mesh-minimal" (program)

The same command (for sockets example):

17:10:20 Compile [  0.3%]: cmain.S
17:10:20 [DEBUG] Compile: C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm --cpu Cortex-M4F -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M4 -DUSB_STM_HAL -DARM_MATH_CM4 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F429ZI\IAR\.includes_7085507b6b69db011e3a2f64169c8a9d.txt -o .\BUILD\NUCLEO_F429ZI\IAR\mbed-os\cmsis\TOOLCHAIN_IAR\cmain.o .\mbed-os\cmsis\TOOLCHAIN_IAR\cmain.S
17:10:20 [DEBUG] Return: 0

I tried to reproduce locally, cant, was able to build 20x at least. The same IAR (Compared the version numbers for linker, assembler via these logs). Any suggestions how to reproduce this one?

Note, it happens randomly, not always in the CI.

Mine setup (IAR versions):

    IAR Assembler V7.80.4.12462/W32 for ARM
    Copyright 1999-2017 IAR Systems AB.

cc @theotherjimmy @studavekar

[0xc0170]

cc @bcostm @adustm @LMESTM @jeromecoutant - you noticed these failures, but were not able to reproduce neither?

[jeromecoutant]

Hi Martin

I couldn't reproduce with these commands:

mbed import mbed-os-example-mesh-minimal
cd mbed-os-example-mesh-minimal/mbed-os
mbed compile -m NUCLEO_F429ZI -t IAR -v

Same IAR version:
[DEBUG] Output: IAR ELF Linker V7.80.4.12462/W32 for ARM

Compile [ 0.2%]: cmain.S
[DEBUG] Compile: C:/Program Files (x86)/IAR Systems/Embedded Workbench 7.5/arm\bin\iasmarm --cpu Cortex-M4F -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M4 -DUSB_STM_HAL -DARM_MATH_CM4 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F429ZI\IAR.includes_305dc107280526cdc975f1b97d3fea97.txt -o .\BUILD\NUCLEO_F429ZI\IAR\mbed-os\cmsis\TOOLCHAIN_IAR\cmain.o .\mbed-os\cmsis\TOOLCHAIN_IAR\cmain.S
[DEBUG] Return: 0

[adustm]

@jeromecoutant It looks like it is not easily reproducable.
Did you notice such failures in our non regression test bench ? I can't remember of such an issue.

[LMESTM]

@jeromecoutant @adustm @0xc0170 ... I have never encountered this issue.

[adbridge]

@jeromecoutant @adustm @LMESTM do you guys have any way of putting those devices into a stress test that continually repeats the above steps 100s of times in a row to see if you hit the issue ?

[LMESTM]

@adbridge not sure to understand which devices you're referring to.
It's a compilation issue that has only been seen so far on the mbnd ci test setup ... Would make more sense to investigate on this setup don't you think ?

[adbridge]

@LMESTM I meant running 100s of consecutive IAR compilations for the 2 devices mentioned and seeing if you can reproduce yourselves. We are investigating further in our CI as well.

[LMESTM]

@adbridge I couldn't find the "return 3" error code definition in IAR documentation, but I suspect a problem of access to the file rather than a problem of compilation itself (there is no compilation error here ...). Not clear to me how doing 100s of compilation here would help, @0xc0170 already did 20 without reproducing. Is it possible to restart the ci job and check if the problem happens again ?

[studavekar]

We are able to reproduce this failure now in CI and also locally however the procedure is bit cumbersome, We will try to have an script to reproduce the failure

Windows :

Failing case:

(.env) C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal>"C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm" --cpu Cortex-M7 --fpu VFPv5 -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M7 -DARM_MATH_CM7 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F767ZI\IAR\.includes_db60bf2c33b9563ba7951b63c163c4e6.txt -o .\BUILD\NUCLEO_F767ZI\IAR\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\startup_stm32f767xx.o .\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.S

Passing case :

(.env) C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal>"C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm" --cpu Cortex-M7 --fpu VFPv5 -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M7 -DARM_MATH_CM7 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F767ZI\IAR\.includes_db60bf2c33b9563ba7951b63c163c4e6.txt -o startup_stm32f767xx.o .\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.S

We have additionally did experiments which points it may be bug in IAR itself on handling file system paths.

(.env) C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal>"C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm" --cpu Cortex-M7 --fpu VFPv5 -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M7 -DARM_MATH_CM7 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F767ZI\IAR\.includes_db60bf2c33b9563ba7951b63c163c4e6.txt -o .\BUILD\NUCLEO_F767ZI\IAR\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN\startup_stm32f767xx.o .\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.S

Could not open file: 'C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal\BUILD\NUCLEO_F767ZI\IAR\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN\startup_stm32f767xx.o'

(.env) C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal>"C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm" --cpu Cortex-M7 --fpu VFPv5 -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M7 -DARM_MATH_CM7 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F767ZI\IAR\.includes_db60bf2c33b9563ba7951b63c163c4e6.txt -o .\BUILD\NUCLEO_F767ZI\IAR\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\startup_stm32f767xx.o .\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.S

    IAR Assembler V7.80.4.12462/W32 for ARM
    Copyright 1999-2017 IAR Systems AB.

Errors:   0
Warnings: 0
Bytes: 988

(.env) C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal>"C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm" --cpu Cortex-M7 --fpu VFPv5 -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M7 -DARM_MATH_CM7 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F767ZI\IAR\.includes_db60bf2c33b9563ba7951b63c163c4e6.txt -o .\BUILD\NUCLEO_F767ZI\IAR\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.o .\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.S

    IAR Assembler V7.80.4.12462/W32 for ARM
    Copyright 1999-2017 IAR Systems AB.

Errors:   0
Warnings: 0
Bytes: 988

               * * *   I N T E R N A L   E R R O R   * * *

In function: Assembler
Diagnostic: non-initialized parameter
Line: 798   P0: 0    P1: 0

Dear User,
please send us a copy of the source file(s) that were assembled when
the internal error occurred.  Even better is if you could try to
pinpoint the problem with a small sample program.
In addition to the source file(s) we would like to receive information on:
  1. Error message (as above)
  2. Software version (see sign-on message)
  3. Command-line including any .xcl file used
  4. Hardware (memory etc.) and operating system version
  5. Where and when you bought the software
Please report this error to your distributor or the nearest IAR
Systems office. For further information please see the Software
Performance Report that can be found in your manual.


                           T H A N K   Y O U !

(.env) C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal>echo %errorlevel%
3

(.env) C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal>