ESP-IDF release v4.2 upgrade and support for ESP32-S2 #2893

shubhamkulkarni97 · 2020-12-16T06:11:29Z

Description

Update ESP-IDF submodule to stable v4.2 release.
Changes in Amazon FreeRTOS port for compatibility with ESP-IDF v4.2.
Move port files and extra components to a common directory shared by all boards.
Add support for ESP32-S2 SOC in Amazon FreeRTOS.

Note: ESP-IDF v4.2 release requires updated toolchain (xtensa-esp32-elf-gcc 8.2.0) and is not compatible with older toolchain used in Amazon FreeRTOS(xtensa-esp32-elf-gcc 5.2.0).

Please find Getting Started Guide for ESP32-S2

This PR is dependent on FreeRTOS/FreeRTOS-Kernel#231

Checklist:

I have tested my changes. No regression in existing tests.
My code is Linted.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

yngki · 2020-12-17T21:51:21Z

Hey shubhamkulkarni97,

Can you confirm that this PR is ready for review? (There is an option at the bottom of the PR to turn it out of the draft stage).

shubhamkulkarni97 · 2020-12-18T14:39:38Z

@yngki This PR is ready for review. Thanks.

mingyue86010 · 2020-12-21T22:59:49Z

/bot run checks

mingyue86010 · 2020-12-22T00:28:57Z

Hi @shubhamkulkarni97, can you rebase your branch to the top of master?

shubhamkulkarni97 · 2020-12-22T06:11:11Z

Hi @mingyue86010, I have rebased this branch on top of latest master.

mingyue86010 · 2020-12-22T20:03:03Z

Hi @mingyue86010, I have rebased this branch on top of latest master.

Thank you @shubhamkulkarni97 ! I verified this PR locally it looks no problem and is able to build. The blocker on the build check is caused by the toolchain setup in our CI (some python dependencies need to be updated). I'm working on this with our CI team. Once it is done then I will git this PR and the others relearted PRs meaged.

mingyue86010 · 2020-12-22T21:11:57Z

Hi @shubhamkulkarni97,

Can you resolve the confilct? I think it just need to include the change from #2897 .

Thanks!
Ming

.gitmodules

shubhamkulkarni97 · 2020-12-23T06:57:50Z

Hi @mingyue86010,

I have resolved the merge conflict.

Thanks,
Shubham

mingyue86010 · 2020-12-23T18:27:32Z

/bot run checks

mingyue86010 · 2020-12-23T18:27:56Z

/bot run checks

This reverts ESP32 specific changes in commit 200d2ed

…ild system Remove linker flags from toolchain specific files iot_wifi.c: Use esp_netif APIs Fix warnings in WiFi port layer Change smartconfig implementation extras.c: Add stack overflow hook in freertos component Remove secure_sockets layer in ports directory and use AFR secure_socket layer iot_pkcs11_pal.c: Change ESP_LOGx to ESP_EARLY_LOGx in initialize_nvs_partition

…source code in app_update component

Add mbedtls specific config options in sdkconfig.defaults Disable ESP_NETIF_TCPIP_ADAPTER_COMPATIBLE_LAYER in sdkconfig to fix IP issue with WiFi reconnection Sync freertos component Kconfig with IDF Update linker fragment for freertos component Update sdkconfig.defaults to disable GCC8 warnings and remove corresponding CFLAGS from Makefile

all boards. Remove mbedTLS port files and use ESP-IDF provided files. Changes in CMakeLists for mbedTLS to support ESP32-S2.

library

iot_board_gpio.h: Use pins which are common for ESP32 and ESP32-S2

shubhamkulkarni97 · 2020-12-24T13:36:42Z

Rebased to latest master to fix merge conflict

pavanmr94 · 2020-12-30T02:54:20Z

/bot run checks

octopusbunch · 2021-01-09T05:03:43Z

Hi,

I am currently trying to work with this commit and I'm running into some issues...

I'm using an ESP32-D0WD-V3 chip on a custom board connected to a ESP-Prog board for programming. I'm developing on a Windows 10 PC and I've installed esp-idf tools v2.3.

In my project, I am using amazon-freertos as a submodule and when I tried to build with cmake -DVENDOR=espressif -DBOARD=esp32_plus_ecc608a_devkitc -DCOMPILER=xtensa-esp32 -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE=./amazon-freertos/tools/cmake/toolchains/xtensa-esp32.cmake -GNinja -S . -B build

I was getting the error :

CMake Error at amazon-freertos/vendors/espressif/boards/esp32/CMakeLists.txt:416 (add_executable):
  add_executable cannot create target "my-app" because another target
  with the same name already exists.  The existing target is an executable
  created in source directory "C:/<proj_directory>".  See documentation
  for policy CMP0002 for more details.
Call Stack (most recent call first):
  amazon-freertos/CMakeLists.txt:70 (include)

It appears the changes in vendors\espressif\boards\esp32\CMakeLists.txt broke the build. By reverting this file to commit cc5530d I was able to continue on in the build, but I ran into a different error.

Question 1:
After running .\amazon-freertos\vendors\espressif\esp-idf\tools\idf.py -p COM4 -b 921600 erase_flash flash monitor there is a error in the esptool script output below. Any ideas on how to fix this building and flash issue? Let me know if you need any more information.

Executing action: flash
Running ninja in directory <proj_directory>\build
Executing "ninja flash"...
[0/2] Re-checking globbed directories...
[4/9] Generating binary image from built executable
FAILED: .bin_timestamp
cmd.exe /C "cd /D <proj_directory>\build && python <proj_directory>/amazon-freertos/vendors/espressif/esp-idf/components/esptool_py/esptool/esptool.py --chip esp32 elf2image --flash_mode dio --flash_freq 40m --flash_size 4MB --elf-sha256-offset 0xb0 -o <proj_directory>/build/.bin / && "C:\Program Files\CMake\bin\cmake.exe" -E echo "Generated <proj_directory>/build/.bin" && "C:\Program Files\CMake\bin\cmake.exe" -E md5sum <proj_directory>/build/.bin > <proj_directory>/build/.bin_timestamp"
esptool.py v3.0
Traceback (most recent call last):
  File "<proj_directory>/amazon-freertos/vendors/espressif/esp-idf/components/esptool_py/esptool/esptool.py", line 3969, in <module>
    _main()
  File "<proj_directory>/amazon-freertos/vendors/espressif/esp-idf/components/esptool_py/esptool/esptool.py", line 3962, in _main
    main()
  File "<proj_directory>/amazon-freertos/vendors/espressif/esp-idf/components/esptool_py/esptool/esptool.py", line 3631, in main
    operation_func(args)
  File "<proj_directory>/amazon-freertos/vendors/espressif/esp-idf/components/esptool_py/esptool/esptool.py", line 3079, in elf2image
    e = ELFFile(args.input)
  File "<proj_directory>/amazon-freertos/vendors/espressif/esp-idf/components/esptool_py/esptool/esptool.py", line 2542, in __init__
    with open(self.name, 'rb') as f:
PermissionError: [Errno 13] Permission denied: '/'

Question 2:
Another issue, I've downloaded the latest xtensa-esp32-elf-gcc8_4_0-esp-2020r3-win64 compiler tools but the version is 8.4. Is there any way to get the proper xtensa-esp32-elf-gcc 8.2 version you reference?

xtensa-esp32-elf-gcc --version
xtensa-esp32-elf-gcc (crosstool-NG esp-2020r3) 8.4.0

aggarg · 2021-01-11T03:30:06Z

@octopusbunch - I moved your issue here: #2931

Thanks.

mingyue86010 · 2021-01-13T02:07:43Z

Hi @shubhamkulkarni97,

After this PR get merged. It breaks the AWS OCW matadata generation by some errors of "links to target "idf::bt" but the target was not found". And the same error to "idf::newlib".

You can reproduce the error by running the cmake command with metadata mode from the amazon-freertos root directory, such as:
$ cmake configure -DVENDOR=espressif -DBOARD=esp32_wrover_kit -Bbuild-esp -S. -DAFR_METADATA_MODE=1

It looks the errors are intruduced by this two lines:

The components of "idf::newlib" and "idf::bt" are added with IDF 4.2 PR. I think you might more familar with ESP build system, so can you help check with the error above?

Thank you!

Ming

shubhamkulkarni97 · 2021-01-13T06:43:56Z

Hi @mingyue86010,

Thanks for reporting this issue, I'll check for the issue and keep you updated.

shubhamkulkarni97 · 2021-01-13T11:20:06Z

Hi @mingyue86010,
I have a couple of questions:

What is the usecase of AFR metadata mode and what artifacts are generated by enabling this mode? Is there any documentation about using metadata mode?
Can you help to add a test case which tests AFR metadata mode in IoT Device Tester? It will help ensure that metadata mode is not broken by changes in future.

Thanks,
Shubham

shubhamkulkarni97 · 2021-01-13T13:15:24Z

@mingyue86010,

Can you try applying attached patch and check if it fixes the issue with AFR metadata mode?

0001-esp32-esp32s2-Update-CMakeLists.txt-to-exclude-linki.patch.zip

mingyue86010 · 2021-01-13T18:46:34Z

Hi @shubhamkulkarni97,

Thanks Shubham. The generated metadata is used by our internal services to deliver a customized project from FreeRTOS Console. You are right that there is no documentation or a way to verify that the generated metadata is correct. We will try to add these.

Thank you for the patch. I made a similar fix as your patch yesterday, but without the following change.

target_link_libraries(
     AFR::secure_sockets::mcu_port
     INTERFACE
-        AFR::tls
-        AFR::wifi
-        idf::newlib
+        ${secure_sockets_libraries}
 )

It can get the metadata generated. I will apply your changes later. After my fix there is still some build issues of the "customized projects", which is caused by missing files from the customized project folder. Our internal services "picks" the files from the repo into the customized porject zip file according to the generated metadata information. So the problem may caused by some missing path in the cmake. I will further take a look into this.

Thank you for your help!

Regards,
Ming

shubhamkulkarni97 marked this pull request as ready for review December 18, 2020 14:39

shubhamkulkarni97 requested a review from a team as a code owner December 18, 2020 14:39

mingyue86010 mentioned this pull request Dec 22, 2020

Idf v4.2 and esp32s2 support #2904

Closed

2 tasks

shubhamkulkarni97 mentioned this pull request Dec 22, 2020

Is it possible to enable Secure Boot and Flash Encryption for ESP32 using AWS FreeRTOS project? #2877

Closed

shubhamkulkarni97 force-pushed the feature/idf_v4.2_and_esp32s2_support branch from 9d98204 to 6c98840 Compare December 22, 2020 06:09

mingyue86010 reviewed Dec 22, 2020

View reviewed changes

.gitmodules Outdated Show resolved Hide resolved

shubhamkulkarni97 force-pushed the feature/idf_v4.2_and_esp32s2_support branch from 6c98840 to f2610b8 Compare December 23, 2020 06:54

shubhamkulkarni97 added 9 commits December 24, 2020 19:05

Revert "Kernel update (aws#2543)"

036d2dc

This reverts ESP32 specific changes in commit 200d2ed

aws_esp_ota_ops: Remove OTA source code from ports directory and use …

7d8751e

…source code in app_update component

Enable task watchdog and interrupt watchdog timers for aws_demos

3136da5

Move port and extra components directory to a common location shared by

cc5530d

all boards. Remove mbedTLS port files and use ESP-IDF provided files. Changes in CMakeLists for mbedTLS to support ESP32-S2.

Add support for ESP32-S2

7a5923b

Updates in CMakeLists.txt to fix build issues when AFR is build as a

8fed5f3

library

Fix common_io specific issues

03f32dd

iot_board_gpio.h: Use pins which are common for ESP32 and ESP32-S2

shubhamkulkarni97 force-pushed the feature/idf_v4.2_and_esp32s2_support branch from f2610b8 to 03f32dd Compare December 24, 2020 13:36

n9wxu approved these changes Dec 30, 2020

View reviewed changes

mingyue86010 merged commit 18cb6df into aws:master Dec 30, 2020

shubhamkulkarni97 deleted the feature/idf_v4.2_and_esp32s2_support branch December 31, 2020 15:58

aggarg mentioned this pull request Jan 11, 2021

Compile and Flash issues with ESP-IDF 4.2 #2931

Closed

aggarw13 mentioned this pull request Jan 28, 2021

Related Amazon-freertos #2959

Closed

aggarw13 mentioned this pull request Feb 4, 2021

Add updates about test runner and ESP-IDF in CHANGELOG #2970

Merged

shubhamkulkarni97 restored the feature/idf_v4.2_and_esp32s2_support branch February 11, 2021 04:47

aggarw13 mentioned this pull request May 21, 2021

How to change the latest release branch of Amazon freertos to use esp-idf v4.2 instead of v3.3 #3163

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ESP-IDF release v4.2 upgrade and support for ESP32-S2 #2893

ESP-IDF release v4.2 upgrade and support for ESP32-S2 #2893

shubhamkulkarni97 commented Dec 16, 2020

yngki commented Dec 17, 2020

shubhamkulkarni97 commented Dec 18, 2020

mingyue86010 commented Dec 21, 2020

mingyue86010 commented Dec 22, 2020

shubhamkulkarni97 commented Dec 22, 2020

mingyue86010 commented Dec 22, 2020 •

edited

Loading

mingyue86010 commented Dec 22, 2020

shubhamkulkarni97 commented Dec 23, 2020

mingyue86010 commented Dec 23, 2020

mingyue86010 commented Dec 23, 2020

shubhamkulkarni97 commented Dec 24, 2020

pavanmr94 commented Dec 30, 2020

octopusbunch commented Jan 9, 2021

aggarg commented Jan 11, 2021

mingyue86010 commented Jan 13, 2021

shubhamkulkarni97 commented Jan 13, 2021

shubhamkulkarni97 commented Jan 13, 2021

shubhamkulkarni97 commented Jan 13, 2021

mingyue86010 commented Jan 13, 2021 •

edited

Loading

ESP-IDF release v4.2 upgrade and support for ESP32-S2 #2893

ESP-IDF release v4.2 upgrade and support for ESP32-S2 #2893

Conversation

shubhamkulkarni97 commented Dec 16, 2020

Description

Checklist:

yngki commented Dec 17, 2020

shubhamkulkarni97 commented Dec 18, 2020

mingyue86010 commented Dec 21, 2020

mingyue86010 commented Dec 22, 2020

shubhamkulkarni97 commented Dec 22, 2020

mingyue86010 commented Dec 22, 2020 • edited Loading

mingyue86010 commented Dec 22, 2020

shubhamkulkarni97 commented Dec 23, 2020

mingyue86010 commented Dec 23, 2020

mingyue86010 commented Dec 23, 2020

shubhamkulkarni97 commented Dec 24, 2020

pavanmr94 commented Dec 30, 2020

octopusbunch commented Jan 9, 2021

aggarg commented Jan 11, 2021

mingyue86010 commented Jan 13, 2021

shubhamkulkarni97 commented Jan 13, 2021

shubhamkulkarni97 commented Jan 13, 2021

shubhamkulkarni97 commented Jan 13, 2021

mingyue86010 commented Jan 13, 2021 • edited Loading

mingyue86010 commented Dec 22, 2020 •

edited

Loading

mingyue86010 commented Jan 13, 2021 •

edited

Loading