dfu-util: Error during download (LIBUSB_ERROR_TIMEOUT) (IDFGH-5227) #6999
Comments
github-actions
bot
changed the title
|
Hi @majodi. Thank you for pointing out a possible issue but I'm having difficulties to reproduce it. I have filled up the partition with an embedded file. I still can flash the large image with DFU. I'm running Linux so maybe the issue is with the driver of macOS. I'll try to find someone who will double check this on macOS.
The options are outlined on this page: https://docs.espressif.com/projects/esp-idf/en/latest/esp32s2/api-guides/usb-otg-console.html. You still can use USB CDC (you can entirely avoid DFU and UART). You can manually put the chip into ROM bootloader mode to do the initial upload. |
Yes, I know, I'm using that now in a cumbersome way. I have to do a DFU with a small fw and no wifi because of this issue: Monitor no access (error) over native USB after esp_wifi_init() (IDFGH-5196) #6969 Then I can use CDC with all embedded files and wifi on. I do loose my CDC port (because of this #6969 issue) but at least I'm up and running. For monitoring I have to use the Saola which has a uart bridge. Not ideal, so was hoping for another tool or something.
Okay, I tested on Linux too. This is what I get (maybe it is something stupid I do that I overlook somehow): Executing action: dfu-flash Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc. Opening DFU capable USB device... |
|
@majodi Could you prepare a simple project which you can share with us where you reproduce the DFU flashing issue? I have modified the blink example. I've added your partition table and added a large text file which is printed in the beginning before turning the LED on/off. I increased the size of the text file until it filled up the partition. I couldn't reproduce the issue in this setup. |
|
@dobairoland I played around with the blink example but could not get it to fail (as with you). So I created a failing project (stripped as much as possible). For this example project "idf.py set-target esp32s2" and then "idf.py dfu" works but then "idf.py dfu-flash" gives the error. The project is here: https://github.com/majodi/issue6999 I'm sure it must be something stupid I overlook. Maybe you can spot it with a fresh mind. Hope I'm not wasting your time... |
|
@majodi Thank you for taking the time and preparing a project. No, you are not wasting my time. I'm happy if I can help. But I'm afraid my answer won't make you happy. I cloned your project and checked out the exact same version of ESP-IDF you are using. I'm on Linux with dfu-util 0.10 and using the Saola board v1.0. I run: idf.py build dfu
idf.py -p /dev/ttyACM0 dfu-flash... and I was able to flash successfully: I can even repeat the process and I'm getting success every time. Are you still using the Saola board? Do you have anything else connected? Can you check your USB cable? |
Yes, and also other custom board (see below)
No external peripherals connected to the board if that is what you mean.
Used 3 different USB-C cables for the custom board (as it has a USB-C connector) and made a new USB cable with dupont pinouts for the Saola. Okay. I'm stuck. This is what I have and have done: I have a Saola board and I have my custon pcb with ESP32-S2-WROVER module on it.
Below you find dfu-util logs for totally different conditions. I could buy a new Saola (maybe different chip version? although the modules on the cust.pcb are recent), I could try Windows10. Or maybe you know other debugging options to check what is going on or what is different with my setup compared to yours. I could make a screencast if you like. Thnx. So procedure: clone issue6999 in file main.c uncomment above --> error is back same on macos/ubuntu/Saola/cust.pcb/usb/usb-c/different cables Two logs from "dfu-util -vvv -d 303a:2 -D /Users/majodi/esp/issue6999/build/dfu.bin" (-vvv for max verbose output) This is the verbose output (on macos, custom pcb, usb-c dfu-util v0.10, esp-idf v4.4-dev-1183-g9d34a1cd4) only the last bit: [ 1.074107] [0021700b] libusb: debug [darwin_handle_transfer_completion] handling transfer completion type control with kernel status 0 This is the verbose output (on ubuntu, Saola, usb dfu-util v0.9, esp-idf v4.4-dev-1254-g639e7ad49 ) so totally different setup only the last bit again: [ 6.060580] [000103e1] libusb: debug [handle_control_completion] handling completion status 0 please let me know if you need more. |
|
@majodi The Ubuntu and Mac are two different machines (computers), right? I've got a bad feeling about USB-C. I cannot try with that. Are you able to try it with good old USB 2.0? For example, through an USB bridge, Raspberry Pi, ... |
|
@dobairoland yes, one is an iMac (tried both the normal USB and USB-C it has) and the other (Ubuntu) is an old Dell Optiplex with normal USB (2.0). Also, the flashing works okay on all setups when leaving off one embedded file. |
|
I will ask someone else to look at this issue because I'm out of ideas. I'm sorry. |
|
Thnx. Just let me know what you need. |
|
Hm, I'm also unsure of what's going on here... It's almost like the device writes to flash but gets hung up there. The decoding and writing of a DFU file is somewhat complicated (an ESP32 DFU file actually is a CPIO archive, for instance) so perhaps you managed to trigger a bug there that causes the code to go out to lunch on an edge case (triggered by e.g. a specific file size or offset). Just to confirm this, can you upload a dfu file that fails? (You should be able to find it in the build/ directory of your project). |
|
Good idea. This (attached) is the dfu that fails to flash. It's this code: https://github.com/majodi/issue6999 and commenting the following lines will reduce the bin's file size to 840 KB and make it flash again. There's no need to edit component.mk and CMakeLists.txt just comment these lines to make it flash: httpd_resp_set_type(req, "application/javascript; charset=UTF-8"); I'm eager to hear if this flashes or not on your side. If not, there must be something going on here with the build process and we narrowed it down. In that case we should compare stuff. |
|
Yep, that DFU replicates the issue on my side (Debian Linux, kernel 5.10.0-6-amd64). Time to start debugging, I guess. |
|
Good news! I will think about what would be good items to compare. Let me know what your thoughts are on candidates to compare. Thnx. |
|
I can flash successfully the attached |
|
That makes it more confusing... perhaps it's still a timing thing? Let me check... |
|
@dobairoland auch! That's crazy. |
|
Okay, I think I found it. The issue is that the ROM will erase the region a partition is in as soon as it receives the first bit of the data that is in the partition. This means that for larger partitions, larger regions of flash will need to be erased - which takes more time. In your specific case, because you embed a fairly large javascript file in the flash, the main partition is large enough to take >5 seconds to erase. Unfortunately, dfu-utils has a hardcoded 5 second timeout, and for you and me, that gets triggered. My guess is that @dobairoland is sitting in a colder room than we are (or has a less worn-out flash chip, or just got lucky) and his chip managed to erase within this timeout. The trivial solution is to get the sources to dfu-util and change the timeout (src/dfu.c, line 33) to something longer. Longer term, we may need to change our dfu image generator to break up partitions that are >1MiB into multiple split partition files so they don't hit the timeout. |
|
That's a known issue, but I didn't dig into it because it was about erasing, but now it makes sense. I will try to find the link. |
|
Ah, just look for "dfu-util timeout erasing" and you'll see multiple cases. Stupid of me to not mention this before (just didn't make the connection as it was about erasing). |
|
That looks all somewhat related indeed; most of these point to some DFU variation that does erasing as a separate (not technically DFU compliant) step, so I can imagine you didn't initially think of this. |
|
@Spritetm @dobairoland Happy to confirm that recompiling dfu-util with a timeout of 20sec did indeed do the job. Thank you. Should I close this issue? |
espressif-bot
added
Status: In Progress
|
@dobairoland With the unmodified dfu-util it didn't flash (timeout), with the modified (20sec) it did flash ok. |
|
@dobairoland Unmodified dfu-util: dfu_1Mi.bin does NOT flash I don't know if you have CONFIG_ESP_CONSOLE_USB_CDC=y, but after reset I don't get a port with this one either (see #6969). But that's another thing. |
|
@majodi Thank you very much for your help. I'll adjust the partition size according to your results. The CDC should be enabled in these binaries because I have used your settings: https://github.com/majodi/issue6999/blob/master/sdkconfig.defaults#L9 |
|
@dobairoland Okay. That means that issue #6969 manifests itself with this bin too, as I don't get a CDC port after flashing. However it does appear when turning off wifi init code. |
|
I can see the CDC port and connect to it after flashing your project wit dfu-util. It shows a warning about the wifi password being too short and aborts in loop. |
|
@dobairoland Okay, maybe not the right place here in this issue to go on about CDC port not appearing ( should be in #6969 perhaps) but this is what I get (repeatable): Your dfu_512k.bin (posted here in dfu.zip) flashed with normal unmodified dfu-util --> no CDC port after flashing/reset code from https://github.com/majodi/issue6999 with only line (main) 41-44 commented (to be able to use unmodified dfu-util) flashed with normal unmodified dfu-util --> CDC port appears after reset code from https://github.com/majodi/issue6999, nothing commented, flashed with modified dfu-util (now have to use modified dfu-util with longer timeout) --> no CDC port after flashing/reset code from https://github.com/majodi/issue6999, only line (main) 97 (wifi) commented, flashed with modified dfu-util (now have to use modified dfu-util with longer timeout) --> CDC port appears after reset |
espressif-bot
added
Status: Done
ROM will erase the region a partition is in as soon as it receives the first bit of the data that is in the partition. For large partitions it takes more than 5 seconds to erase which is a hard-coded limit in dfu-utils. This splits large binaries and adds them by chunks which should avoid timing-out during flashing. Closes #6999
ROM will erase the region a partition is in as soon as it receives the first bit of the data that is in the partition. For large partitions it takes more than 5 seconds to erase which is a hard-coded limit in dfu-utils. This splits large binaries and adds them by chunks which should avoid timing-out during flashing. Closes #6999
The 5 seconds timeout in dfu-util is for USB requests, which always should be replied to immediately. The bootloader should use the DFU mechanism to tell how long dfu-util should wait before issuing the /next/ USB request. The bootloader should definitely not get busy doing something that might take long time before replying to this USB request. The real solution is to make the bootloader DFU compliant. |
Environment
Problem Description
Running idf.py dfu-flash (after creating flash using idf.py dfy) gives the following output:
Executing action: dfu-flash
Running ninja in directory /Users/majodi/esp/ccfw/build
Executing "ninja dfu-flash"...
[0/1] cd /Users/majodi/esp/ccfw/build && /usr/local/Cellar/cmake/3.19.2/bin/cmake -D ESP_DFU_BIN="/Users/majodi/esp/ccfw/build/dfu.bin" -D ESP_DFU_PID="2" -P /Users/majodi/esp/esp-idf/tools/cmake/run_dfu_util.cmake
Command list: dfu-util;-d;303a:2;-D;/Users/majodi/esp/ccfw/build/dfu.bin
dfu-util 0.10
Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2020 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/
Opening DFU capable USB device...
ID 303a:0002
Run-time device DFU version 0110
Claiming USB DFU Runtime Interface...
Determining device status: state = appIDLE, status = 0
Device really in Runtime Mode, send DFU detach request...
dfu-util: Device will detach and reattach...
error detaching
Opening DFU USB Device...
Claiming USB DFU Interface...
Setting Alternate Setting #0 ...
Determining device status: state = dfuIDLE, status = 0
dfuIDLE, continuing
DFU mode device DFU version 0110
Device returned transfer size 64
Copying data from PC to DFU device
Download [ ] 0% 0 bytesdfu-util: Error during download (LIBUSB_ERROR_TIMEOUT)
Done
This error seems to appear only with 'larger' firmwares. In this case the following partition table:
Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, 0x9000, 0x6000,
phy_init, data, phy, 0xf000, 0x1000,
factory, app, factory, 0x10000, 0x120000,
If I remove one of the embedded files (COMPONENT_EMBED_TXTFILES) the firmware fits in the standard partition table and I get no error. Seems it takes too long to download the firmware if larger than normal.
So, I know dfu-util is an 'external' tool but I'm not sure how much the process can be tuned at the (rom?) counterpart side (if any) by Espressif. Or if there is some setting to increase the timout. All I know is that I'm stuck if my custom PCB (later on) doesn't have a uart bridge and I have to rely on flashing using the native usb. Is there another way (or tool) to use perhaps?
The text was updated successfully, but these errors were encountered: