Upgrade to arduino-esp32 v3.0.x (esp-idf v5.1)

[mskvortsov]

In preparation for the upcoming espressif/arduino-esp32 3.0.0 release.
At this point, this is more to see what's going wrong at compilation time.

Update 1: basic things work as tested on tlora-t3s3-v1 (esp32s3), tlora-v2-1-1_6 (esp32) and a custom esp32c3.

[code8buster]

Did they deprecate the old ADC calibration API like expected? Or does all the old stuff still sort of work? Worthwhile to check real battery readings from this branch on esp32s3 / esp32 platforms i.e. a Heltec v3 and a Lilygo t3 2.1-1.6
https://github.com/orgs/meshtastic/projects/20?pane=issue&itemId=51214157

[mskvortsov]

Did they deprecate the old ADC calibration API like expected?

Yes, there are two ADC APIs in IDF5.1 at the moment: a now deprecated legacy API coming from IDF4 and a new improved one that this pull request uses.

Or does all the old stuff still sort of work? Worthwhile to check real battery readings

Thanks for pointing this out, indeed I've found a bug around ADC channel selection. By the way, I've also realized that adc_oneshot_io_to_channel() allows inferring both unit and channel at once from just BATTERY_PIN, so ADC_CHANNEL definitions become redundant.

And here comes some experimental data for a simple discharge test. Two devices, tlora-t3s3-v1 using esp32s3 - first two graphs, and tlora-v2-1-1_6 using esp32-pico-d4 - last two graphs. Both devices have INA219 modules connected and uplink their telemetry data to an MQTT broker. LoRa transmission is off, WiFi is on, sample rate is 30s (downsampled to 60s by a plotting script). The esp32s3 device has a 18650 battery and esp32 has a smaller one 16300.

The plotting script https://gist.github.com/mskvortsov/5bd8045e812d2355ad62f9637556dc76

Overall, I see no big difference across the discharge curves on this branch and the master one. Except maybe for a mystery in the tail of the tlora-v2 graph - voltage deviation on idf5 is increasing instead of decreasing as on idf4.

[mskvortsov]

I may be missing something, but the ADC2/WiFi workaround seems useless (both on the master branch and here).

On a tlora-v2-1-1_6 esp32 device, after reassigning an ADC1-wired BATTERY_PIN to some other unused ADC2-wired pin, adc2_get_raw() always returns ESP_ERR_TIMEOUT.

firmware/src/Power.cpp

Lines 303 to 312 in 62b310a

	int32_t adc_buf = 0;
	for (int i = 0; i < BATTERY_SENSE_SAMPLES; i++) {
	// ADC2 wifi bug workaround, see
	// https://github.com/espressif/arduino-esp32/issues/102
	WRITE_PERI_REG(SENS_SAR_READ_CTRL2_REG, RTC_reg_b);
	SET_PERI_REG_MASK(SENS_SAR_READ_CTRL2_REG, SENS_SAR2_DATA_INV);
	adc2_get_raw(adc_channel, ADC_WIDTH_BIT_12, &adc_buf);
	raw += adc_buf;
	raw_c++;
	}

That leads to constantly zero battery voltage readings:

DEBUG | 13:51:10 141 [Power] Battery: usbPower=0, isCharging=0, batMv=0, batPct=0

SDK documentation clearly states that

If Wi-Fi is started via esp_wifi_start(), this function will always fail with ESP_ERR_TIMEOUT.
https://docs.espressif.com/projects/esp-idf/en/v4.4.7/esp32/api-reference/peripherals/adc.html

There's a possibility that only my board is affected, and the workaround actually helps in some other cases, but I don't have any device with BAT_MEASURE_ADC_UNIT declared to make a crosscheck.

[code8buster]

Something from Heltec would probably use that workaround. The old v2/2.1 uses ADC unit 2
Their new boards starting from v3 and anything later also use unit 2 iirc but they're all ESP32-S3 so IDK that the code really applies.

Thanks for this! Wish I had more time to review it. Well done

[mskvortsov]

Bumping arduino-esp32 from 3.0.1 to 3.0.3 hasn't gone well for original esp32 targets. IRAM overflows again.

region `iram0_0_seg' overflowed by 856 bytes

The last one I fixed was much smaller in size, like 120 bytes IIRC, and I had to put quite an effort to carefully carve out unused bits without rebuilding esp-idf. It looks like the time has come to stop that nonsense hacking and consider a custom esp-idf build.

SDK has quite a few options for controlling IRAM usage: https://docs.espressif.com/projects/esp-idf/en/release-v5.1/esp32/api-guides/performance/ram-usage.html#optimizing-iram-usage

And esp32-arduino-lib-builder offers an easy way for rebuilding the SDK libraries and using them in platformio by adding a line to esp32.ini

platform_packages =
  platformio/framework-arduinoespressif32-libs@file:///.../esp32-arduino-lib-builder/out/tools/esp32-arduino-libs

[jp-bennett]

@mskvortsov Have you seen #4324 to addresses the iram issue? You seem to have a much better handle on how the toolchain works, so interested in your thoughts. Either way, I think we're going to have to pursue the approach of building our own custom arduino-esp32 libs to squeeze out more iram space.

[markbirss]

M5Stack (IAMLIUBO) have shown interest in support for a new ESP32-C6 SX1262 tft device they are busy finalising that is also dependant on the upgrade

Here is a discord link

https://discord.com/channels/867578229534359593/919642584480112750/1265637433911804035

[imliubo]

Either way, I think we're going to have to pursue the approach of building our own custom arduino-esp32 libs to squeeze out more iram space.
Yes, I think this is indeed.

Last two weeks, I working on support our some new products(esp32-c6), but Arduino ESP32 Core V3 change a lot, and I can't solve the NimBLE-Arduino not compatible problem, so I change the whole project to IDF environment, use Arduino ESP32 as an IDF component.
After a lot of work, I have luck to get the project running on esp32-c6. This isn't the best solution, I just make it work.