Update ESP32 series to IDF 5.1.3

[AdrianSoundy]

Description

This PR updates code to the IDF 5.1.3 version and adds the new ESP devices ESP32-C6 and ESP32-H2.
As part of the C6 and H2 addition the OpenThread was added to support the new radio. The openThread requires
IPV6 networking support so all the socket code was updated and tested to support IPV6 if enabled with LWIP_IPV6 option.

The V5.X IDF updates the API of a number of peripherals and the original API have been deprecated.
The other peripherals should have the API updated as some give a warning on boot. These can be done separately to this PR. I will raise issues for them. Devices RMT, ADC, DAC

Outstanding Issues.

@josesimoes do you have time to have a look at these problems.

• For some reason can't debug ESP32 from VS it always gives a [Event dispatch failed:Object reference not set to an instance of an object.] but other devices including C3,C6 & H2 work fine. Not sure what's happening here.

• With IDF 5.X a number of components are not included in the IDF and loaded separately using the espressif component manager from the component register. The TinyUsb component as used by ESP32-S2 & ESP32-S3 is one of these. I tried to get this working but for now commented out in Binutils.ESP32.cmake. So these targets will compile but won't work via USB.

List of changes

• Add support for OpenTHread with ipv6 stack
• Add ESP32-C6 and ESP32-H2 support ( ESP32_C6_THREAD & ESP32_H2_THREAD ), The devices can also support zigbee, add later
• Add RMT to C3 preset
• Don't default the SDK_CONFIG_FILE parameter for ESP32-C3 in CMakePresets.json otherewise picks up wrong one.
• Add RMT to ESP32-S3
• IPV6 support in socket stack ( managed Network Interface changes to be done later once changes have worked out ), System.net updated
• Add Support for OpenThread config variables
• ESP32-C6 Wi-Fi 6 support, tested uses AX mode if available and CONFIG_SOC_WIFI_HE_SUPPORT defined.
• Wi-Fi/ Openthread coexistence (seems to start up and connect OK)
• Add support for new API for PCNT (GpioPulseCounter)
• Add hack to stop DT_DIR already defined warnings in build
• Add ESP32_WESP32 and ESP32_WT32_ETH01 to nightly builds as missing.
• Add ESP32_ETHERNET_KIT_1.2, ESP32_C6 and ESP32_H2 to builds.
• Change sdkconfig files to minimal files so we only define the differences from standard instead of listing everything. In binutils.ESP32.cmake now append changes to sdkconfig.default instead of trying to change existing, makes it far simpler.
• Added a nanoFramework.Networking.Thread for control of openThread from C#, enabling this enables openthread. Currently assembly doesn't exist. WIP
• Sort out header includes for IDF 5.1.1, lots of changes
• Changes to support MBEDTLS 2.x -> 3.x version change, TLS1.0/1.1/1.2 deprecated in V3. Add conditional compile so common code works with MBEDTLS Version 2 or 3 now. Version 3 not tested yet.
• Save the original IDF ffconfig.h to ffconfig.h.sav and restore if not using Fatfs. This resolves issues when changing local builds and ffconfig.h missing.
• Update Ethernet support for IDF 5.1.1
• INTERNAL_RAM_LEAVE_FREE_FOR_ALLOCATION for ble configs was fixed at 30K now uses ESP32_RESERVED_RAM_FOR_IDF_ALLOCATION from CMakePresets.json, CMakePresets.json updated to reflect this.
• Various drivers updated for adjusted API naming with legacy API
• Compiler errors with BuildPathsArray() in io_filesystem_System_IO_Directory due to GCC update. Changed & tested
• Add I2C slave to esp32-c6 and esp32-h2 builds after rebase.
• Made RMT device default ON for esp32-C3 targets.

Motivation and Context

Update to IDF 5.1.1 was so we could add the esp32 C6 & H2 devices.

• Closes Upgrade to IDF 5.1 to allow support for building for ESP32 C6/H2 Home#1345

How Has This Been Tested?

A lot of things have been tested.

• IPV6
• ESP32-C6
• ESP32-H2
• OpenThread will connect to mesh network using default parameters and IPV6 communication is working, but is a Work in progress.
• GpioPulseCounter after change to new API
• GPIO
• RMT

But further testing will be required for

• Mbedtls as we have updated to Mbedtls v3.X in IDF 5.1.1 from 2.X. Other targets are still using v2.X

Types of changes

• Improvement (non-breaking change that improves a feature, code or algorithm)
• Bug fix (non-breaking change which fixes an issue with code or algorithm)
• New feature (non-breaking change which adds functionality to code)
• Breaking change (fix or feature that would cause existing functionality to change)
• Config and build (change in the configuration and build system, has no impact on code or features)
• Dev Containers (changes related with Dev Containers, has no impact on code or features)
• Dependencies (update dependencies and changes associated, has no impact on code or features)
• Documentation (changes or updates in the documentation, has no impact on code or features)

Checklist

• My code follows the code style of this project (only if there are changes in source code).
• My changes require an update to the documentation (there are changes that require the docs website to be updated).
• I have updated the documentation accordingly (the changes require an update on the docs in this repo).
• I have read the CONTRIBUTING document.
• I have tested everything locally and all new and existing tests passed (only if there are changes in source code).

[AdrianSoundy]

Seems to be some build issues, i'll sort them out.

[AdrianSoundy]

@josesimoes
I had a look at current build error but can't see what's wrong. It not finding the compiler etc in path.
If you have time can you have a look.

[AdrianSoundy]

Found that the Component manager is not supported by the IDF as a Library project.
The recommended work around is to download the components(tinyusb) directly from Component Registry using link on component web page.

Added code to binutils.esp32.cmake to download and install tinyusb & esp_tinyusb to IDF components directory and sort other requirements to get them to compile and link.

Now the S2 & S3 targets are working with Tinyusb

The only thing left now is to finish openthread work.

[josesimoes]

@AdrianSoundy that sound great!! Just looked at the CMake and it looks OK. Well done!

[josesimoes]

@AdrianSoundy again looking at the code, it seems that several files not related at all with the PR are now showing diferences! 😯
Suggest that you rebase the branch to get the updates from upstream and then you'll be left with only the changes here.

[AdrianSoundy]

Thats basically what I did.
I still need to do some further testing since rebasing with upstream changes.
Had a lot of conflicts to sort out.

[Ellerbach]

That's a very heavy lifting! Can't wait to see all this available! Very hard to review the PR as there are also changes brought form existing merge. You may want to update the main branch/rebase on it to bring clarity.
And I would say, if it builds and some of the devices works, then, all should work :-) and as always, if we have problems, we will know it as soon as the firmware will be available!

[networkfusion]

Current reason for the M5Core2 smoke test failure is:

[AdrianSoundy]

@networkfusion
Thanks for looking into to that. I'll commit a fix for m5core2 problem.

[AdrianSoundy]

@josesimoes @Ellerbach
This is back to a working state with changes from main merged.
Could this be a good time to rebase & merge ?

[Ellerbach]

Could this be a good time to rebase & merge ?

From my perspective yes!

I let @josesimoes do the magic rebase!

[josesimoes]

@AdrianSoundy sure thing! I'll take care of that.

[josesimoes]

Last merge and review done! All good.