Wifi stops working properly when code size grows, with not much code

[yanshay]

I'm trying to get embassy+wifi+embedded-tls working together on a the most simple program on ESP32 (the dhcp example, only trying to add tls to it).
I managed to get it to compile, but once I add any code that requires embedded-tls (even if execution doesn't get to that code) the program stops connecting to wifi. It has nothing to do with the code, because that code gets to run only after connecting, it just has to do with the required code being added to the binary.

This is the log output from the working state (so w/o embedded-tls):

Chip type:         esp32 (revision v1.0)
Crystal frequency: 40MHz
Flash size:        4MB
Features:          WiFi, BT, Dual Core, 240MHz, Coding Scheme None
MAC address:       40:91:51:9a:11:60
App/part. size:    472,208/4,128,768 bytes, 11.44%
[00:00:00] [========================================]      17/17      0x1000                                                           [00:00:00] [========================================]       1/1       0x8000                                                           [00:00:06] [========================================]     308/308     0x10000                                                          [2024-08-11T22:45:05Z INFO ] Flashing has completed!
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

���������������������Ғ�Ҫ�j

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:7104
0x3fff0030 - _ZN8esp_wifi9HEAP_DATA17h785362a51058b9edE
    at ??:??
load:0x40078000,len:15576
0x40078000 - __bswapsi2
    at ??:??
load:0x40080400,len:4
0x40080400 - _init_end
    at ??:??
ho 8 tail 4 room 4
load:0x40080404,len:3876
0x40080404 - _init_end
    at ??:??
entry 0x4008064c
0x4008064c - core::fmt::Arguments::new_v1
    at /Users/yanivshaya/.rustup/toolchains/esp/lib/rustlib/src/rust/library/core/src/fmt/mod.rs:354
I (30) boot: ESP-IDF v5.1-beta1-378-gea5e0ff298-dirt 2nd stage bootloader
I (30) boot: compile time Jun  7 2023 07:48:23
I (32) boot: Multicore bootloader
I (36) boot: chip revision: v1.0
I (40) boot.esp32: SPI Speed      : 40MHz
I (45) boot.esp32: SPI Mode       : DIO
I (49) boot.esp32: SPI Flash Size : 4MB
I (54) boot: Enabling RNG early entropy source...
I (59) boot: Partition Table:
I (63) boot: ## Label            Usage          Type ST Offset   Length
I (70) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (78) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (85) boot:  2 factory          factory app      00 00 00010000 003f0000
I (93) boot: End of partition table
I (97) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=11cech ( 72940) map
I (132) esp_image: segment 1: paddr=00021d14 vaddr=3ffb0000 size=00e0ch (  3596) load
I (133) esp_image: segment 2: paddr=00022b28 vaddr=3ffc3a0c size=00190h (   400) load
I (138) esp_image: segment 3: paddr=00022cc0 vaddr=40080000 size=0ca48h ( 51784) load
I (167) esp_image: segment 4: paddr=0002f710 vaddr=00000000 size=00908h (  2312)
I (168) esp_image: segment 5: paddr=00030020 vaddr=400d0020 size=53444h (341060) map
I (302) boot: Loaded app from partition at offset 0x10000
I (302) boot: Disabling RNG early entropy source...
INFO - esp-wifi configuration Config { rx_queue_size: 5, tx_queue_size: 3, static_rx_buf_num: 10, dynamic_rx_buf_num: 32, static_tx_buf_num: 0, dynamic_tx_buf_num: 32, ampdu_rx_enable: 0, ampdu_tx_enable: 0, amsdu_tx_enable: 0, rx_ba_win: 6, max_burst_size: 1, country_code: "CN", country_code_operating_class: 0, mtu: 1492, heap_size: 65536, tick_rate_hz: 100, listen_interval: 3, beacon_timeout: 6, ap_beacon_timeout: 300, failure_retry_cnt: 1, scan_method: 0 }
Waiting for link
WARN - esp_wifi_internal_tx 12290
start connection task
Device capabilities: Ok(EnumSet(Client))
Starting wifi
Wifi started!
About to connect...
Waiting for link
Waiting for link
Waiting for link
Wifi connected!
Waiting for link
Waiting to get IP address...
Waiting to get IP address...
Waiting to get IP address...
Waiting to get IP address...
Got IP: 192.168.10.96/24

This is the not connecting log:

Chip type:         esp32 (revision v1.0)
Crystal frequency: 40MHz
Flash size:        4MB
Features:          WiFi, BT, Dual Core, 240MHz, Coding Scheme None
MAC address:       40:91:51:9a:11:60
App/part. size:    561,344/4,128,768 bytes, 13.60%
[00:00:00] [========================================]      17/17      0x1000                                                           [00:00:00] [========================================]       1/1       0x8000                                                           [00:00:07] [========================================]     360/360     0x10000                                                          [2024-08-11T22:58:51Z INFO ] Flashing has completed!
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

����������������������Ғ�Ҫ�j

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:7104
0x3fff0030 - _ZN8esp_wifi9HEAP_DATA17h785362a51058b9edE
    at ??:??
load:0x40078000,len:15576
0x40078000 - __bswapsi2
    at ??:??
load:0x40080400,len:4
0x40080400 - _init_end
    at ??:??
ho 8 tail 4 room 4
load:0x40080404,len:3876
0x40080404 - _init_end
    at ??:??
entry 0x4008064c
0x4008064c - core::fmt::Arguments::new_v1
    at /Users/yanivshaya/.rustup/toolchains/esp/lib/rustlib/src/rust/library/core/src/fmt/mod.rs:354
I (30) boot: ESP-IDF v5.1-beta1-378-gea5e0ff298-dirt 2nd stage bootloader
I (30) boot: compile time Jun  7 2023 07:48:23
I (32) boot: Multicore bootloader
I (36) boot: chip revision: v1.0
I (40) boot.esp32: SPI Speed      : 40MHz
I (45) boot.esp32: SPI Mode       : DIO
I (49) boot.esp32: SPI Flash Size : 4MB
I (54) boot: Enabling RNG early entropy source...
I (59) boot: Partition Table:
I (63) boot: ## Label            Usage          Type ST Offset   Length
I (70) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (78) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (85) boot:  2 factory          factory app      00 00 00010000 003f0000
I (93) boot: End of partition table
I (97) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=14b1ch ( 84764) map
I (136) esp_image: segment 1: paddr=00024b44 vaddr=3ffb0000 size=00e0ch (  3596) load
I (137) esp_image: segment 2: paddr=00025958 vaddr=3ffc3a0c size=00190h (   400) load
I (142) esp_image: segment 3: paddr=00025af0 vaddr=40080000 size=0a528h ( 42280) load
I (168) esp_image: segment 4: paddr=00030020 vaddr=400d0020 size=66b4ch (420684) map
I (320) esp_image: segment 5: paddr=00096b74 vaddr=4008a528 size=02520h (  9504) load
I (330) boot: Loaded app from partition at offset 0x10000
I (330) boot: Disabling RNG early entropy source...
INFO - esp-wifi configuration Config { rx_queue_size: 5, tx_queue_size: 3, static_rx_buf_num: 10, dynamic_rx_buf_num: 32, static_tx_buf_num: 0, dynamic_tx_buf_num: 32, ampdu_rx_enable: 0, ampdu_tx_enable: 0, amsdu_tx_enable: 0, rx_ba_win: 6, max_burst_size: 1, country_code: "CN", country_code_operating_class: 0, mtu: 1492, heap_size: 65536, tick_rate_hz: 100, listen_interval: 3, beacon_timeout: 6, ap_beacon_timeout: 300, failure_retry_cnt: 1, scan_method: 0 }
Waiting for link
WARN - esp_wifi_internal_tx 12290
start connection task
Device capabilities: Ok(EnumSet(Client))
Starting wifi
Wifi started!
About to connect...
Waiting for link
Waiting for link
Waiting for link
Waiting for link
Waiting for link
Waiting for link
Waiting for link
Waiting for link
Waiting for link
Waiting for link
Waiting for link
Waiting for link
Waiting for link

[bjoernQ]

I can only guess but most probably there are buffers allocated on the stack which makes the stack grow into the data segment

You could try reducing the size of buffers and allocate them statically. If you can post the code (or a link to a repo) someone can have a look if there is anything obvious to improve the situation

[yanshay]

Thanks for the quick response.
I reduced the buffers, including those required by embedded-tls to in total half of the wifi embassy dhcp original example.
In the example there were two of 4096 bytes each, I reduced it to four of 1024 (so half in total).
Could the stack override the data segment with such a minimal program? If I get it right the stack starts from top and the data segment from the bottom and there are a few hundreds of KB for the entire area? Or is the code also running from that same memory space? (if that's the case then what's the use of the flash).
I also recall from some time ago that esp-wifi used the dram-seg2 area so shouldn't conflict with the program.

Any document that explains well the memory arrangements when using esp-hal?

I uploaded the code to a repo : https://github.com/yanshay/embassy-tls

The code is pretty ugly, sorry about that, just to get it to compile I did unsafe things, but those shouldn't affect execution because it doesn't reach the areas of the code where those ugly things could have an effect. And it works with that unsafe code when removing the tls code that is never executed.

I wonder how there doesn't seem to be code using tls when all web sites require that, aren't people connecting from esp32 (using esp-hal) to secure servers (which is the standard for quite some time, even for web API's)?

[bjoernQ]

I gave your code a try and it works so far.

I had to change a few things to get it completely though the whole code:

• PRINTER_IP/PRINTER_PORT changes to google.com:443 (just to see the handshake will work)
• read_record_buffer needs to be 16640 bytes
• let config = TlsConfig::new().with_server_name("google.com");

Waiting for link
WARN - esp_wifi_internal_tx 12290
start connection task
Device capabilities: Ok(EnumSet(Client))
Starting wifi
Wifi started!
About to connect...
Waiting for link
Waiting for link
Wifi connected!
Waiting for link
Waiting to get IP address...
Got IP: 192.168.137.99/24
connecting...
connected!
connecting...
connected!

Could the stack override the data segment with such a minimal program? If I get it right the stack starts from top and the data segment from the bottom and there are a few hundreds of KB for the entire area?

It depends on how much memory is allocated on the stack - especially if data is copied/moved that might happen. Also, for async the generated code currently can be very bad in terms of stack usage. In general, it's better to prefer borrowing over moves

Or is the code also running from that same memory space? (if that's the case then what's the use of the flash).

Some parts of the code (inside esp-hal and esp-wifi) need to run from RAM and therefore are copied to RAM. All other code runs from flash-memory - flash on ESP32 is XIP (execute-in-place)

I also recall from some time ago that esp-wifi used the dram-seg2 area so shouldn't conflict with the program.

Yes - on the original ESP32 esp-wifi currently allocates the wifi-heap in dram-seg2 but the stack starts at the end of the lower dram-segment (where also the data sections is located)

Any document that explains well the memory arrangements when using esp-hal?

I don't think we explain that in detail in the current documentation. Probably something we should consider

Regarding example code for TLS: we plan to have examples using Rustls and most probably also a simple example using embedded-tls. We are just not there yet

[yanshay]

Thanks
This is strange. The only change you've made that could affect my issue (which is about the connection phase) is changing the buffer size, that could affect memory footprint. The rest are relevant only for later. I tried changing the buffer and it still doesn't work.
How did you use google.com? Did you fill in their IP address? (since the code doesn't have any DNS)

Only differences I can think of:

1. Board - could it be a board issue? It says on my chip ESP32-WROOM-32D. So I tried a different board and I witness the same behavior (and it's a completely different board, not just another copy of the same). Are you using ESP32 or one of the S2/S3 maybe?
2. Maybe the rust compiler version or something about my environment? Maybe you're using something newer/different? This is the most relevant option from what I can think of.
3. I'm building using cargo run --release. Are you doing the same? I tried also w/o --release and it doesn't work either.
4. Something about my network? I can't see how that could be if it connects with a minor difference in the code in no relevant area, but it's a difference.

At least good to know that the embedded-tls stuff is working, now I just need to find how to get it working for me.

[bjoernQ]

Did you fill in their IP address? (since the code doesn't have any DNS)

Yes - I used an IP address

[bjoernQ]

Board

I used two different boards

Chip type:         esp32 (revision v1.0)
Crystal frequency: 40 MHz
Flash size:        4MB
Features:          WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None

and

Chip type:         esp32 (revision v3.0)
Crystal frequency: 40 MHz
Flash size:        4MB
Features:          WiFi, BT, Dual Core, 240MHz, Coding Scheme None

both work fine with your code

Compiler

❯ cargo +esp --version
cargo 1.80.0-nightly (376290515 2024-07-16)

Building / Running

❯ cargo run --release
   Compiling embassy-tls v0.1.0 (C:\projects\review\yanshay\embassy-tls)
warning: unused imports: `rng::Rng` and `timg::TimerGroup`
  --> src/main.rs:13:5
   |
13 |     rng::Rng,
   |     ^^^^^^^^
14 |     system::SystemControl,
15 |     timer::{timg::TimerGroup, ErasedTimer, OneShotTimer, PeriodicTimer},
   |             ^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

warning: `embassy-tls` (bin "embassy-tls") generated 1 warning (run `cargo fix --bin "embassy-tls"` to apply 1 suggestion)
    Finished `release` profile [optimized + debuginfo] target(s) in 10.33s
     Running `espflash flash --monitor target\xtensa-esp32-none-elf\release\embassy-tls`
[2024-08-12T12:38:46Z INFO ] Detected 2 serial ports
[2024-08-12T12:38:46Z INFO ] Ports which match a known common dev board are highlighted
[2024-08-12T12:38:46Z INFO ] Please select a port
✔ Remember this serial port for future use? · no
[2024-08-12T12:38:49Z INFO ] Serial port: 'COM28'
[2024-08-12T12:38:49Z INFO ] Connecting...
[2024-08-12T12:38:50Z INFO ] Using flash stub
[2024-08-12T12:38:51Z WARN ] Setting baud rate higher than 115,200 can cause issues
Chip type:         esp32 (revision v3.0)
Crystal frequency: 40 MHz
Flash size:        4MB
Features:          WiFi, BT, Dual Core, 240MHz, Coding Scheme None
MAC address:       xx:xx:xx:xx:xx:xx
App/part. size:    560,640/4,128,768 bytes, 13.58%
[2024-08-12T12:38:52Z INFO ] Segment at address '0x1000' has not changed, skipping write
[2024-08-12T12:38:52Z INFO ] Segment at address '0x8000' has not changed, skipping write
[00:00:09] [========================================]     360/360     0x10000
[2024-08-12T12:39:02Z INFO ] Flashing has completed!
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

ets Jul 29 2019 12:21:46

rst:0x1 (POWERON_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:7104
0x3fff0030 - esp_wifi::HEAP_DATA
    at ??:??
load:0x40078000,len:15576
load:0x40080400,len:4
ho 8 tail 4 room 4
load:0x40080404,len:3876
entry 0x4008064c
0x4008064c - core::fmt::Arguments::new_v1
    at C:\Users\Bjoern\.rustup\toolchains\esp\lib\rustlib\src\rust\library\core\src\fmt\mod.rs:354
I (31) boot: ESP-IDF v5.1-beta1-378-gea5e0ff298-dirt 2nd stage bootloader
I (31) boot: compile time Jun  7 2023 07:48:23
I (33) boot: Multicore bootloader
I (37) boot: chip revision: v3.0
I (41) boot.esp32: SPI Speed      : 40MHz
I (46) boot.esp32: SPI Mode       : DIO
I (50) boot.esp32: SPI Flash Size : 4MB
I (55) boot: Enabling RNG early entropy source...
I (60) boot: Partition Table:
I (64) boot: ## Label            Usage          Type ST Offset   Length
I (71) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (79) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (86) boot:  2 factory          factory app      00 00 00010000 003f0000
I (94) boot: End of partition table
I (98) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=14a28h ( 84520) map
I (137) esp_image: segment 1: paddr=00024a50 vaddr=3ffb0000 size=00e0ch (  3596) load
I (139) esp_image: segment 2: paddr=00025864 vaddr=3ffc3a0c size=00190h (   400) load
I (143) esp_image: segment 3: paddr=000259fc vaddr=40080000 size=0a61ch ( 42524) load
I (169) esp_image: segment 4: paddr=00030020 vaddr=400d0020 size=66988h (420232) map
I (321) esp_image: segment 5: paddr=000969b0 vaddr=4008a61c size=0242ch (  9260) load
I (331) boot: Loaded app from partition at offset 0x10000
I (331) boot: Disabling RNG early entropy source...
INFO - esp-wifi configuration Config { rx_queue_size: 5, tx_queue_size: 3, static_rx_buf_num: 10, dynamic_rx_buf_num: 32, static_tx_buf_num: 0, dynamic_tx_buf_num: 32, ampdu_rx_enable: 0, ampdu_tx_enable: 0, amsdu_tx_enable: 0, rx_ba_win: 6, max_burst_size: 1, country_code: "CN", country_code_operating_class: 0, mtu: 1492, heap_size: 65536, tick_rate_hz: 100, listen_interval: 3, beacon_timeout: 6, ap_beacon_timeout: 300, failure_retry_cnt: 1, scan_method: 0 }
Waiting for link
WARN - esp_wifi_internal_tx 12290
start connection task
Device capabilities: Ok(EnumSet(Client))
Starting wifi
Wifi started!
About to connect...
Waiting for link
Waiting for link
Wifi connected!
Waiting for link
Waiting to get IP address...
Got IP: 192.168.137.214/24
connecting...
connected!

Network

For me it sometimes takes a bit longer to get an IP from my router - works better using my smartphone or laptop as an access-point but with some patience it also works fine with my router (FRITZ!Box 5590 Fiber)

[yanshay]

I have same compiler version. Upgraded my espflash now to match.

I see a few minor differences in the logs, you can see diffs at: https://www.diffchecker.com/mEk4hyPD/

• There's the boot 0x12 vs. 0x13 on mine (which I don't know what that means), and a few other diffs regarding the board
• Chip revision is 3.1 on mine (but also on my 1.0 it doesn't work)
• Application size is a bit different and that has implications on various addressed I guess. I don't know why there should be any difference since I changed my code (except the IP of google.com, and wifi info, maybe that's the diff) as you wrote. Unless you mistyped the size of the buffer (read_record_buffer needs to be 16640 bytes - maybe you meant 16384?)
• I'm on a Mac and you're on Windows

Seems like a dead end 😞

[bjoernQ]

Here is the exact diff of my changes

❯ git diff
diff --git a/src/main.rs b/src/main.rs
index 7688acf..2736e73 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -60,8 +60,8 @@ macro_rules! mk_static {

 const SSID: &str = env!("SSID");
 const PASSWORD: &str = env!("PASSWORD");
-const PRINTER_IP: Ipv4Address = Ipv4Address::new(192, 168, 10, 78);
-const PRINTER_PORT: u16 = 8883;
+const PRINTER_IP: Ipv4Address = Ipv4Address::new(142,251,36,132);
+const PRINTER_PORT: u16 = 443;

 #[main]
 async fn main(spawner: Spawner) -> ! {
@@ -168,12 +168,12 @@ async fn main(spawner: Spawner) -> ! {
         println!("connected!");

         // TLS
-        let mut read_record_buffer = [0u8; 1024];
+        let mut read_record_buffer = [0u8; 16640];
         let mut write_record_buffer = [0u8; 1024];

         /* Commenting out this section makes this work */

-        let config = TlsConfig::new(); //.with_server_name("example.com");
+        let config = TlsConfig::new().with_server_name("google.com"); //.with_server_name("example.com");
         let mut tls: TlsConnection<TcpSocket, Aes128GcmSha256> =
             TlsConnection::new(socket, &mut read_record_buffer, &mut write_record_buffer);

[yanshay]

Managed to make some progress.

I switched to another SSID on the same wifi router (maybe there's some diff in settings, I'll check that later).
What I see now:
On one board (rev1.0) it now connects and gets IP, another board (rev3.1) connects but doesn't get an IP. Both with the exact same code (with embedded-tls). This is consistent.

Add to that that on the previous SSID with minor code changes it switched between connecting to not connecting, I believe there is some sensitivity in the code, at least when using embassy + esp-wifi, maybe some timing issues, that would cause connection to fail.

I could blame the network if those issues would be inconsistent, but the consistency of success/failure depending on board+software leads me to believe it is a software issue.

Edit: BTW - a few months ago I developed with esp-wifi+embassy quite a lot, a pretty complex application, very network oriented, and I didn't have these issues on the same SSID that now doesn't work, including with one of these boards (also with ESP32-S3). Never had any issues like that. So maybe it's something relatively new.

[yanshay]

I have one board that's working, so tried to connect to the device I need and I can't get it to work.
I now noticed embedded-tls supports only TLS 1.3, so that's probably the reason. I know the device I need to connect to supports 1.2, I guess not 1.3. What a disappointment.

So my only other option is rusttls. But that one seems like way more complex, right?
I saw you are building a provider, which is a large project just to get this working, is that indeed the case?
If that's indeed the case seems like I'll need to dump my project and switch to another language until rusttls is supported.
Will rusttls be supported also for async?
If you want I can beta test that code when it's ready.

[bjoernQ]

Oh yes - embedded-tls can only support TLS 1.3.

Out rusttls provider is currently more in an experimental phase and would also only support a TLS 1.3 cipher - so currently your option would only be esp-mbedtls.

In the end (not yet) our helpers for rusttls will also support wrappers for embedded-io-async

[yanshay]

Thanks, I managed to establish TLS 1.2 connection using esp-mbedtls example apps, both sync and async!!!