esp8266 precompiled bootloaders don't support partitions past 1MB

[iosabi]

Description

The esp8266 cpu ships with some bootloader code pre-compiled from the vendor SDK. I couldn't find what revision and options where used to compile these bootloaders, they have the following bug:

If the partition table contains any partition that starts or ends at the 1MB boundary or after that you get the following error even if using a chip with 4MB flash:

I (69) boot: SPI Flash Size : 4MB
E (72) flash_parts: partition 3 invalid - offset 0x100000 size 0x2000 exceeds flash chip size 0x100000
E (81) boot: Failed to verify partition table
E (85) boot: load partition table error!
user code done

Note that the code detects the flash at 4MB but then complains about the partition being larger than 1MB.

Steps to reproduce the issue

Create a custom partition table with the vendor recommended OTA scheme, for example:

$ cat mypartitions.csv
# Name,   Type, SubType, Offset,   Size
nvs,      data, nvs,     0x9000,   0x6000
phy_init, data, phy,     0xf000,   0x1000
ota_0,    app,  ota_0,   0x010000, 0xf0000
otadata,  data, ota,     0x100000, 0x2000
ota_1,    app,  ota_1,   0x110000, 0xf0000

Then compile and run on a esp12 board the following example program:

USEMODULE=esp_log_startup make -C tests/shell BOARD=esp8266-esp-12x PARTITION_TABLE_CSV=`pwd`/mypartitions.csv -j flash

Expected results

The board should boot all the way to the shell.

Actual results

The ROM bootloader loads the flash bootloader at 0x0000 and prints messages in the UART at 74880 bps while doing that; and then the flash bootloader switches to 115200 and prints the following error:

I (69) boot: SPI Flash Size : 4MB
E (72) flash_parts: partition 3 invalid - offset 0x100000 size 0x2000 exceeds flash chip size 0x100000
E (81) boot: Failed to verify partition table
E (85) boot: load partition table error!

[iosabi]

I was able to compile a new bootloader from scratch from the vendor repo and pass it as BOOTLOADER_BIN (unfortunately that needs a relative path in esptool.inc.mk, but ok, doable) and boot with this partition table. We are using in RIOT a now aging version of the SDK from @gschorcht fork.

Could we make the SDK dependency a PKG and do the symbol manipulation needed in the PKG Makefile instead? Is there any problem with that approach?

I can also just update the bootloader binaries, but if I can build the bootloader from source I'd rather do that than ship a binary.

[gschorcht]

Could we make the SDK dependency a PKG and do the symbol manipulation needed in the PKG Makefile instead? Is there any problem with that approach?

Indeed, we already had the idea to define the SDK as a package as it is done for the Geko SDK for EFM/EFR/EZR32 MCUs. However, at that time the packages were downloaded for each application separately to the application's bin directory. This means that when compiling all test applications, for example, one would have had a lot of copies of the SDK, each 112 MB in size. Therefore, we provided a preconfigured SDK for download. Now that packages are downloaded into a separate directory, the SDK could be implemented as a package with according modifications via the Makefile.

However, having the SDK as a package does not solve the problem of the aging SDK, because the RIOT port of the ESP8266 requires much more changes than renaming some symbols. Some parts of the SDK source code have to be extracted and need to be changed. Also, a FreeRTOS adaptation layer has to be realized to link RIOT with the functions in the SDK that assume to work on a FreeRTOS. Therefore, the RIOT port is quite tricky.

Another problem is that the interface how FreeRTOS functions are used by the binary libraries of the SDK has been completly changed from version 3.1 to version 3.3. Before version 3.3, there was an OS interface that defines an abstract version of used FreeRTOS functions that could be linked to functions implemented by RIOT. Since version 3.3 the SDK links directly to FreeRTOS functions. I had to learn this last when I upgraded the RIOT to the ESP8266_RTOS_SDK version 3.3. I had to change a lot. The upgrade was almost finished and I had only one remaining problem with the hardware timer when WiFi was enabled. But then Corona came and I had absolutely no time anymore to continue working on this upgrade.

[iosabi]

@gschorcht thanks for your explanation and even more for the work you did to bring up ESP here.

I looked at the release/v3.1-for-riot-os branch in your fork and it is only 2 commits ahead of gschorcht/RIOT-Xtensa-ESP8266-RTOS-SDK@913a06a which is the first parent in espressif's 3.1 branch. In those two commits all I see is the binary .a files of three libraries being edited. I looked at the diff between the symbols and made a list of the symbol renames you did. Basically reverse engineering what you did to be able to replicate the same change. From what you said my understanding is that there's nothing blocking today a change to stop using your fork and instead create a SDK pkg/ to pull espressif at that specific SHA and apply the same symbol renaming you did, given that the SDK repo will be pulled only once (to $RIOTBASE/build) and even if we need to make a copy of all the vendor's components .a per app they are only 2.5MB.

I think that the advantage of doing this is that it would be documented what changes were made to the vendor SDK so anybody else can come and fix stuff there. There's a bug elsewhere about documenting how were the toolchain packages created which is a similar problem. You are now busy so I think that enabling other people to help would be a great thing to have. For example, espressif have added more commits in their release/v3.1 branch after the 913a06a9ac3b2f18009e8fee8f092ca9ffeccd38 you used as a parent so even if upgrading to 3.3 is complicated, if we had the documentation of what changes were made or even better the tooling to make those automatically we would be able to upgrade to the latest 3.1* release more easily and maybe fix some of these issues (they have released tags 3.1.1 and 3.1.2 since).

I'll prepare a commit to create a pkg/esp8266_sdk following the Gecko SDK example and rename the symbols in their binary libraries in that pkg at build time. My idea is to write a python script to do this rename based on a simple rename table in python. It would be great to be able to use pyelftools but I think I can manage with just the toolchain. Does this sound good? Do you have any advice about this? how did you do the symbol renaming?

[gschorcht]

I'll prepare a commit to create a pkg/esp8266_sdk following the Gecko SDK example and rename the symbols in their binary libraries in that pkg at build time. My idea is to write a python script to do this rename based on a simple rename table in python. It would be great to be able to use pyelftools but I think I can manage with just the toolchain. Does this sound good? Do you have any advice about this? how did you do the symbol renaming?

Great 👍 Many thanks for taking over some developments for ESP platforms. Due to other priorities I can't contribute myself at the moment.

Below, you will find the changes I made to prepare the ESP8266_RTOS_SDK. I think a small shell script should do the job.

I could also share my last state of the RIOT port for the ESP8266_RTOS_SDK v3.3. It might be worth to finish this work because the resulting source code is becoming more and more compatible with ESP-IDF which in turn could further reduce the code duplication for the ESP platforms.

Also, the RIOT port doesn't support ESP32-S2. ESP32-S2 requires at least ESP-IDF version 4.1. It might also be worth to upgrade the RIOT port to this version because ESP32 NodeMCUs are provided more and more with ESP32s.

The following is what I did to prepare the ESP8266_RTOS_SDK.

git clone https://github.com/espressif/ESP8266_RTOS_SDK.git
cd ESP8266_RTOS_SDK/
git checkout release/v3.1
[git checkout -q 08c234ecb5a3941a4e2eb8d6e3221e6642c56fdc]
cd components/esp8266/lib
xtensa-esp8266-elf-objcopy --redefine-sym rtc_init=esp_rtc_init libcore.a
xtensa-esp8266-elf-objcopy --redefine-sym pbkdf2_sha1=wpa_pbkdf2_sha1 libcore.a
xtensa-esp8266-elf-objcopy --redefine-sym aes_wrap=wpa_aes_wrap libwpa.a
xtensa-esp8266-elf-objcopy --redefine-sym aes_unwrap=wpa_aes_unwrap libwpa.a
xtensa-esp8266-elf-objcopy --redefine-sym hmac_md5=wpa_hmac_md5 libwpa.a
xtensa-esp8266-elf-objcopy --redefine-sym sha1_prf=wpa_sha1_prf libwpa.a
xtensa-esp8266-elf-objcopy --redefine-sym rc4_skip=wpa_rc4_skip libwpa.a
xtensa-esp8266-elf-objcopy --redefine-sym pbkdf2_sha1=wpa_pbkdf2_sha1 libwpa.a
xtensa-esp8266-elf-objcopy --redefine-sym hmac_sha1=wpa_hmac_sha1 libwpa.a
xtensa-esp8266-elf-objcopy --redefine-sym hmac_sha1_vector=wpa_hmac_sha1_vector libwpa.a
xtensa-esp8266-elf-objcopy --redefine-sym aes_encrypt_init=wpa_aes_encrypt_init libespnow.a
xtensa-esp8266-elf-objcopy --redefine-sym aes_encrypt=wpa_aes_encrypt libespnow.a
xtensa-esp8266-elf-objcopy --redefine-sym aes_encrypt_deinit=wpa_aes_encrypt_deinit libespnow.a
xtensa-esp8266-elf-objcopy --redefine-sym aes_decrypt_init=wpa_aes_decrypt_init libespnow.a
xtensa-esp8266-elf-objcopy --redefine-sym aes_decrypt=wpa_aes_decrypt libespnow.a
xtensa-esp8266-elf-objcopy --redefine-sym aes_decrypt_deinit=wpa_aes_decrypt_deinit libespnow.a

xtensa-esp8266-elf-objcopy --redefine-sym printf=core_printf libcore.a
xtensa-esp8266-elf-objcopy --redefine-sym printf=espnow_printf libespnow.a
xtensa-esp8266-elf-objcopy --redefine-sym printf=net80211_printf libnet80211.a
xtensa-esp8266-elf-objcopy --redefine-sym printf=phy_printf libphy.a
xtensa-esp8266-elf-objcopy --redefine-sym printf=pp_printf libpp.a
xtensa-esp8266-elf-objcopy --redefine-sym printf=sc_printf libsmartconfig.a
xtensa-esp8266-elf-objcopy --redefine-sym printf=ssc_printf libssc.a
xtensa-esp8266-elf-objcopy --redefine-sym printf=wpa_printf libwpa.a
xtensa-esp8266-elf-objcopy --redefine-sym printf=wps_printf libwps.a
xtensa-esp8266-elf-objcopy --redefine-sym ets_printf=core_ets_printf libcore.a
xtensa-esp8266-elf-objcopy --redefine-sym ets_printf=espnow_ets_printf libespnow.a
xtensa-esp8266-elf-objcopy --redefine-sym ets_printf=pp_ets_printf libpp.a
xtensa-esp8266-elf-objcopy --redefine-sym ets_printf=wpa_ets_printf libwpa.a

For the downloadable toolchain, I have additionally done the following to reduce the download size:

git clone https://github.com/gschorcht/RIOT-Xtensa-ESP8266-RTOS-SDK.git ESP8266_RTOS_SDK
cd ESP8266_RTOS_SDK/
git checkout -q c0174eff7278eb5beea66ce1f65b7af57432d2a9

rm -rf docs examples Kconfig make README.md tools .git*
cd components
rm -rf app_update aws_iot bootloader cjson coap espos esptool_py esp-tls \
       freertos jsmn libsodium log mdns mqtt newlib partition_table \
       pthread smartconfig_ack spiffs ssl tcpip_adapter vfs
find -type f -name '*.[csS]' -exec rm {} \;
find -type f -name '*.cpp' -exec rm {} \;

[iosabi]

I pushed this branch: https://github.com/iosabi/RIOT/tree/esp8266_bldr which builds the bootloader from source from the SDK code and it fixes this particular issue. That branch depends on the SDK PR #16425 so I'll wait until that on is merged.