uPyLoader gets stuck during transfer of bigger files on WiFi connection

[profra]

I tested this sample https://github.com/peterhinch/micropython-samples/tree/master/SSD1306 and I wanted to transfer the file freesans20.py (approx. 16 KB) ... via USB is working but via WiFi not .... transmission through WEBREPL is successful

[BetaRavener]

Hi, I'm really sorry for getting onto this so late, it's been crazy last months. I just have few hours spare now so I wanted to check out this issue. Unfortunately, I wasn't able to replicate the problem. Downloaded the mentioned file, but both writing and reading the file works on WiFi and USB without issues.

For WiFi, I have repeated the upload 5 times and it always succeeded. Could you please try again, if the problem persists on your side? I've used the latest release and 1.8.7 firmware. Also, make sure that you use decent power supply. I don't think this is the case but just to be sure.

[cyberlab2018]

Hello BetaRavener, first of all thank you for your time in the creation of upyloader and share it with all of us, I just want to tell you that I have exactly the same results as the PROFRA partner, normally when transferring files of more than 10Kb it is locked upyloader, and when transfer to the pc is relatively slow compared to webrepl, it is worth mentioning that with webrepl the same large or small files are sent and received without problems, I would like to use your program since it has better view and has many other necessary functions! thanks again and I am waiting for a new version!
P.S. I am using micropython 1.9.3

[BetaRavener]

Thanks @cyberlab2018 for reporting on this. Could you share the files that are not working for you so I can try it myself? I trust you that there's a problem but without being able to reproduce it (which I was unable so far), it's almost impossible for me to fix the issue.

[cyberlab2018]

Hello again BetaRavener, thanks for responding so fast! I send you the files with which upyloader is locked, it is worth mentioning that use the task manager to close upyloader.
when opening again the program had to try as 3 times the connection with the device (nodemcu) or reset it.
Once the reconnection was done, the name of the file with which it had been locked appeared in the file list.
when trying to pass it to the pc it is not possible maybe because of not going through the data completely or corrupting it.
files.zip

[BetaRavener]

I'm really scratching my head now, it just working for me. Here's my board setup and on the following pics you can see that I was able to transfer the files without a problem whatsoever. It does take a while to transfer each file and I looked into this. The progress bar goes 0->100% immediately, so the data itself are sent pretty fast, but uPyLoader is then waiting for response from device and this may take few seconds to happen. I have tested with WebREPL and while we can't see progress, the time it takes to send file is roughly the same.

Now, you mentioned that you're using NodeMCU.. i wonder if this could be the difference that's causing the issue but it's unlikely, because looking at pics it's the very same chip and package that I'm using (ESP8266-12E), just with some additional circuitry. I'm really out of ideas here as to what causes the issue.

(Note: The resistor between +3.3V and GND is redundant)

[cyberlab2018]

Hello, betaravener, I understand that if you can not reproduce the problem, it will be very difficult to find the solution.
Take some photos and maybe you can get an idea of ​​the problem, as you appreciate in the photo where I send the file hacienda.dst to 41
fotos.zip
% is blocked!
after closing the program with the task manager I take 3 pictures where I have to try 3 times the connection to be successful, in the first I mark error in the second error of the password and in the third it connects successfully.
I think because the nodemcu stayed half in the transfer, because if I reset the nodemcu connects on the first attempt.
Once the connection is established, you can see in the list that appears the hacienda.dst file but it is not possible to transfer it to the pc, I think it is incomplete or corrupt.
I do the same operation with the file masis.dst and if it happens successfully because when it went from pc to nodemcu it was also successful.
It is worth mentioning that in your images I see that it has a capacitor that by size goes around 2000 or 3000 mf. As an additional filter, I installed it and the results were the same.
without making any connection changes, I send any file through webrepl and all transfers to nodemcu are successful.
Thanks again for your time! Greetings from Guadalajara Mexico!

[BetaRavener]

Hmm that's pretty interesting! We may be getting somewhere. I didn't expect that it would get stuck midway through (e.g. at 41%). You're right, the reason why you can't connect right away is that NodeMCU is probably still in transfer mode and it treats any communication as file content.

I say it's interesting, because it shouldn't get stuck during transfer. I thought your problem was that it never received response from the MCU, but this happens at 100%. So if you get stuck at 41% it's probably getting stuck somewhere in uPyLoader and not in the communication.

I think you would be able to locate place where it gets stuck quite easily. You need to be running uPyLoader from source and know how to debug. Setting breakpoint in _write_file_job function should lead you to place where it gets stuck (debugger should also hang in there).

However, I know that lot of users here don't have much experience with python and debugging large code. If you need help, we could use software like TeamViewer to work on the issue together (it's software that lets me remote control your PC, as long as you allow me, and you really don't have to worry). If you were considering this option, just leave me some contact here (email, skype,..) and I'll give you further details.

[cyberlab2018]

hello betaRavener, I did what you told me I send screen capture when it is blocked when sending the file, I hope it serves you. regards!

[BetaRavener]

Great job! I will try to research what causes this exception and why could this happen.

For now it seems that underlying socket buffer gets full and because socket is non-blocking, writing more data to it causes exception which is not handled correctly. This could happen due to lag in communication during which MCU is not able to receive data and I keep pumping the rest of file anyway (there's no check for this).

The first thing I can try is to handle this exception and try again in few seconds. Once I'll com,it an update, I'll let you know here and you can try if it fixes the problem.

[profra]

Congratulations, you are stars! 👍

[BetaRavener]

@cyberlab2018 Alright, I've updated code to handle that specific exception and it should now wait+retry several times before considering file transfer failed. Even in such case, the program shouldn't hang up but show an error dialog window. I'm not sure in what state will this leave MCU, so maybe you'll still need to restart the board before being able to work with it again.

However, the main goal is to fix the problem in the first place, so please pull the changes and try uploading your files. The progress bar may hang at some % for a little on occasions, but ultimately the file should upload successfully. If not, please report back what went wrong and if anything changed from last attempts.

[cyberlab2018]

Hello BetaRavener, Congratulations works perfectly!

[BetaRavener]

Great :) I'm closing this for now but in case of any other issues feel free to reopen.