ESP32 WiFi scanning sketch binary is 86Kb larger, consumes 55Kb more RAM in 2.0.0 vs 1.0.6

[avillacis]

Hardware:

Board: ESP32 Dev Module
Core Installation version: 1.0.6 , 2.0.0
IDE name: Arduino IDE
Flash Frequency: 80Mhz
PSRAM enabled: no
Upload Speed: 115200
Computer OS: Fedora 34 x86_64

Description:

When testing many of my Arduino-ESP32 project with 2.0.0 versus 1.0.6, I notice that both required binary space and RAM use are consistently larger in 2.0.0 versus 1.0.6. This introduces the risk that projects that previously worked comfortably within allocated app partition and RAM space under 1.0.6, will now require extra partition space, risking overflows, and/or crashes because of insufficient ram. Nothing has been changed except the version of SDK (1.0.6 versus 2.0.0).

Sketch: (leave the backquotes for code formatting)

#include "WiFi.h"

#if CONFIG_IDF_TARGET_ESP32
//#define WIFI_PIN  GPIO_NUM_4
#define WIFI_PIN  GPIO_NUM_27
#elif CONFIG_IDF_TARGET_ESP32S2
#define WIFI_PIN GPIO_NUM_14
#endif

void reportMemoryUse(const char * label)
{
  Serial.printf("MEMORY USE AT %s: %u total, %u free, %u max.alloc\r\n",
    label, ESP.getHeapSize(), ESP.getFreeHeap(), ESP.getMaxAllocHeap());
}

void setup()
{
    Serial.begin(115200);

#ifdef WIFI_PIN
    pinMode(WIFI_PIN, OUTPUT);
    digitalWrite(WIFI_PIN, LOW);
#endif

    reportMemoryUse("setup() before WiFi setup");

    // Set WiFi to station mode and disconnect from an AP if it was previously connected
    WiFi.mode(WIFI_STA);
    WiFi.disconnect();
    delay(100);

    Serial.println("Setup done");
    reportMemoryUse("setup() after WiFi setup");
}

void loop()
{
    Serial.println("scan start");

    // WiFi.scanNetworks will return the number of networks found
#ifdef WIFI_PIN
    digitalWrite(WIFI_PIN, HIGH);
#endif
    int n = WiFi.scanNetworks();
#ifdef WIFI_PIN
    digitalWrite(WIFI_PIN, LOW);
#endif
    Serial.println("scan done");
    if (n == 0) {
        Serial.println("no networks found");
    } else {
        Serial.print(n);
        Serial.println(" networks found");
        for (int i = 0; i < n; ++i) {
            // Print SSID and RSSI for each network found
            Serial.print(i + 1);
            Serial.print(": ");
            Serial.print(WiFi.SSID(i));
            Serial.print(" (");
            Serial.print(WiFi.RSSI(i));
            Serial.print(")");
            Serial.println((WiFi.encryptionType(i) == WIFI_AUTH_OPEN)?" ":"*");
            delay(10);
        }
    }
    Serial.println("");
    reportMemoryUse("after WiFi scan");

    // Wait a bit before scanning again
    delay(3000);
}

Debug Messages:

Under 1.0.6, this sketch compiles into a 638608-byte binary, and shows the following output:

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:1216
ho 0 tail 12 room 4
load:0x40078000,len:10944
load:0x40080400,len:6388
entry 0x400806b4
MEMORY USE AT setup() before WiFi setup: 360808 total, 334340 free, 113792 max.alloc
Setup done
MEMORY USE AT setup() after WiFi setup: 359568 total, 283652 free, 113792 max.alloc
scan start
scan done
12 networks found
(redacted)

MEMORY USE AT after WiFi scan: 359484 total, 282476 free, 113792 max.alloc
scan start
scan done
17 networks found
(redacted)

MEMORY USE AT after WiFi scan: 359484 total, 282076 free, 113792 max.alloc

Under 2.0.0, the exact same sketch compiles into a 724912-byte binary (86304 bytes larger), and shows the following output:

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1240
load:0x40078000,len:13012
load:0x40080400,len:3648
entry 0x400805f8
MEMORY USE AT setup() before WiFi setup: 366171 total, 308455 free, 65524 max.alloc
Setup done
MEMORY USE AT setup() after WiFi setup: 364203 total, 228051 free, 65524 max.alloc
scan start
scan done
14 networks found
(redacted)

MEMORY USE AT after WiFi scan: 364179 total, 226875 free, 65524 max.alloc
scan start
scan done
13 networks found
(redacted)

MEMORY USE AT after WiFi scan: 364179 total, 226955 free, 65524 max.alloc

The sketch now sees 226955 bytes of free RAM (versus 282076 bytes under 1.0.6), an unexplained 55121-byte increase.

This is worrying because some of our projects using ESP32 boards (with no PSRAM) are already having trouble running OTA updates via their web interface under 2.0.0, as apparently a 4096-byte malloc() inside the Updater object now fails where it previously succeded with 1.0.6.

[avillacis]

In this comment it was mentioned that "the OS has become larger to support more features". However this is unofficial and somewhat vage, and the example sketch here does not use Bluetooth BLE at all.

[avillacis]

The platform.txt for 1.0.6 specifies -Os as a build flag. The equivalent file for 2.0.0 specifies -O2 instead.

On a sample project I maintain, the binary size increased from 1040032 bytes under 1.0.6 to 1228240 bytes under 2.0.0. I tried hacking the platform.txt file that defines the compile flags for the user project, replacing -O2 with -Os. With this, the final binary shrank to 1184544 bytes which is not ideal, but shows that the binary size can be optimized back. This, however, has no effect on RAM usage whatsoever.

[cjcr]

I can confirm the same problem with 2.0.0, I go back to 1.0.6 that works well in this meaning.

• Cannot OTA.
• Big binary size
• Low Heap memory

This is a big down side for this release :-(

[me-no-dev]

Arduino has no fault here. Memory usage and features come from ESP-IDF. There are reasons why they decided to change memory usage of different components. In any case, you can file an issue there.

[avillacis]

At least in the issue of binary size, I disagree. The platform.txt is a file that exists only for Arduino IDE compatibility, and its contents should therefore be managed by the arduino-esp32 project. I grepped the master branch of the esp-idf repository here in github, and no mention of platform.txt was found. As I commented earlier, changing the compile flags from -O2 to -Os (the 1.0.6 default) already yielded measurable binary size reductions. Why was the compiler flag changed in the first place? I suspect that if the entire library shipped with 2.0.0 were recompiled with -Os , most if not all of the binary file size increase would disappear.

[igrr]

Continuing the comparison from #5474 (comment), this time with IDF's example/wifi/scan:

IDF release/v3.3, default sdkconfig: 572000 bytes
IDF release/v3.3, sdkconfig from arduino-esp32 v1.0.6: 610736 bytes
IDF v4.4-dev, default sdkconfig: 620336 bytes (+48kB compared to IDF v3.3 default config)
IDF v4.4-dev, sdkconfig from arduino 2.0.0: 663296 bytes (+52kB compared to IDF v3.3 + config from arduino-esp32 v1.0.6; or +42kB compared to the default sdkconfig for the same IDF version. 25kB of these 42kB come from enabling Bluetooth functionality in sdkconfig.)

The OP mentions that the Arduino scan example became 86kB larger in 2.0.0 compared to 1.0.6. If we assume that 48kB out of these 86 are due to code bloat in IDF, there is still more than 30kB to be accounted for. Given the 52kB difference between the default sdkconfig and the one used in arduino-esp32 v2.0.0, there might be some room for improvement.

As mentioned in the linked issue, idf_size.py (or idf.py size --components) can be used to analyze per-component sizes and even compare them between two builds.

[SuGlider]

@avillacis, @cjcr, @fdivitto
I'm working on this issue and I was able to get the following results using the same sketch you have posted here as an example, with a customized version of Arduino Core 2.0.0:

a) bin size of 632405 bytes
b) Heap:

SDK: v4.4-dev-3401-gb86fe0c66c
MEMORY USE AT setup() before WiFi setup: 332796 total, 308416 free, 65524 max.alloc - max.alloc internal 65524 
Setup done
MEMORY USE AT setup() after WiFi setup: 331164 total, 275492 free, 65524 max.alloc - max.alloc internal 65524
scan start
scan done
21 networks found

MEMORY USE AT after WiFi scan: 331140 total, 273756 free, 65524 max.alloc - max.alloc internal 65524

I created a repository with a new Arduino Core 2.0.0 that can be tested.
https://github.com/espressif/arduino-esp32/tree/mem-optimized

In order to install it as a separated board for testing, please follow the instructions from
https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html
The only difference here would be to clone mem-optimized branch instead.

git clone -b mem-optimized https://github.com/espressif/arduino-esp32 esp32

In case it works fine for a number of users, we will commit it for the next ESP32 Arduino release.
Please test it with your project and let me know.

[Jason2866]

@SuGlider Thx for this! Is the result achieved by changed settings in sdkconfig? If yes which changes you have done? I am interested since we do build the Arduino libs ourselves (changed sdkconfig settings needed) and would like to integrate and test your changed setup. If not which changes have you made?

[SuGlider]

@Jason2866

Yes, result is achieved mainly by changes in sdkconfig in Arduino Lib Builder.

There is also a change to platform.txt to use -Os (optimize for size) and a change I. WiFI library to tweak wifi configs before calling wifi_init().

The branch is for making it easy for testing.

I'll post a PR with all those changes as soon as I have a number of successful user reports back and then I'll send its information here to this issue.

[Jason2866]

@SuGlider Thx for your answer. Compared your choosen sdkconfig with ours. The are no major differences. Since we do not use Arduino IDE Platform.txt is not relevant. Using Option -Os too.
So i will take the changed wifi stuff from this PR.

[SuGlider]

@Jason2866
Please wait for the PR in order to also look to the changes done in the sdkconfig.

I'll post the PR here next week.

[Jason2866]

@SuGlider Thx, a lot! Will wait for the PR. Btw. Arduino Lib builder does fail at the moment because LittleFS is updated and cmake stops.

[Jason2866]

@SuGlider With your latest commit it works. :-) More free mem and flash. Great work!

[toml2]

@SuGlider, I downloaded your changes and it seems to work a charm on Wifi. Around 50k more. SerialBluetooth seems to use around a 100k itself as well

[SuGlider]

We are about to officially release Arduino Core 2.0.1 RC1 that will incorporate these changes.

Thus, if everything goes well, the final release Arduino V2.0.1 will solve this and some other important issues.