Just a thought. Large http.Updates

[Hiddenvision]

Sorry if this is the wrong place for suggestions but delete as required.

To Precede.
I've been using the http Update process (v2.30) for a year or two. Very happy.
But I guess my code has always been smaller than 500k as I never had any failed to update.
Recently updated to V2.4 and Bam, updates fail. (Due to Sketch being too big).
V2.3 Binsize 489k
V2.4 Binsize 538k

Got me thinking.?

Is it possible to:
1: Download the update.Bin from http into FS,
2: Then flash a small Minimal Bin file from FS and reboot,
3: Then finally the minimal boot will simply mounts the FS and flash the main update.bin and reboot again.
4: On reboot delete the update bin if required.

Would that Present any issues? Apart from someone having to write the code.!

OR

Can the Flash allocation be changed to give 1M + 1M + 2MB
rather than the current 512 + 512 + 3MB

I actually use 64Mb memory so lots of space to play with.

Sorry again if this is posted in the wrong place.
Hv.

[Hiddenvision]

Ok So I tried a quick test and looks like Option 1 will work.
Just got to make a simple project that contains enough to Flash from FS
and also include a little rescue up-loader in case the downloaded file failed.

Be nice if Option 2 was possible though.
Any thoughts on that.. anyone.

[d-a-v]

For the generic board and 64MB we currently have at least 16MB(15MB spiffs) and 4M(3MB spiffs) (which you seem to use).
Would you need 16MB(14MB spiffs) / 4M(2MB spiffs) ?
You can try it yourself as per this comment and following.

I'm not an expert myself about OTA constraints, but beeing able to live-update 1MB sketchs would be I guess very welcome.

Please share your findings.

Out of topic, your original concern beeing binary size, you have now two ways of reducing it. Consider tools/board.txt.py --nofloatboards generator option, and Tools/DebugLevel/NoAssert-NDEBUG in menus.

[Hiddenvision]

Hi @d-a-v
Many thanks for the pointers.
My initial findings. Gonna need more coffee.

I am guessing that you refer to editing/recompiling the Source
rather than editing files on the local machine.? or am I being dense.

I am still running 2.3, as my projects when compiled on 2.4 have a few issues.
Least not the OTA size thing but qwerks with some of the other functions I used.
Not worth mentioning until I can document the issues and research solutions.
But too much for me to deal with all in one go right now.

Hv.

[d-a-v]

@Hiddenvision You need latest git master ("nightly" version after 2.4.0) for the mentioned points above. No need to edit source files.

When you have dealt with your other 2.4.0 issues, I guess core-2.4.1 will be available through the boards manager, which will include the mentioned options.

edited

[Hiddenvision]

I had a play with various things.
I got the idea from your previous pointers.
Looking thru various files, I added in the missing menu options in the boards.txt file.
Then I created a few more xxx.ld files.
Reprogramming the devices seems to have changed the flash allocation. Cool.

I have attached the .ld files and also made the edits to the boards.txt.py file
Happy for this to be included in the builds if desired.
Not sure if the numbers are right but I am sure someone with more info will correct if required.

On that subject of numbers what is the largest compiled sketch the ESP will run.
Is it only limited by what's allocated.?
Would seem logical to increase the max sketch size for the larger flash devices.

FLASH_ALLOCATION_TEST_sdk.zip

This was a very trim & Edited Boards.txt file
boards.txt.zip

So if you want to try,
DO NOT TRY THIS ON PRODUCTION FIRMWARE.!!
1: Backup your current boards.txt file,
2: Extract and use the one above, or edit a copy of your existing file, adding in the extra menu options.
3: You may also want to backup the ld folder.
4: Extract the extra .ld files from the other zip.

?? It may work if you put the boards.txt.py file in place but as it is missing from the install I guess not. ??

5: Restart the IDE.

if it all goes pear shaped, read this post backwards to reverse the process.
if your not sure where things are, then stop right there.

Hv.

[Hiddenvision]

Just to update on the findings.

I just updated to 2.4
Realised I forgot to copy the old boards.txt file
So, made all the edits from scratch again,
Programmed device using http update, (Serial Programming is so yesterday)
the headers are "pre-sketch-update" in case they look out of sequence.

x-ESP8266-free-space: 1605632
x-ESP8266-sketch-size: 489648 << Notice the size using 2.3
x-ESP8266-chip-size: 4194304
x-ESP8266-sdk-version: 1.5.3(aec24ac9)
x-ESP8266-mode: sketch
Worked fine.

Re compiled again Started update process again.

x-ESP8266-free-space: 1564672
x-ESP8266-sketch-size: 531536 << Notice the size using 2.4
x-ESP8266-chip-size: 4194304
x-ESP8266-sdk-version: 2.1.0(deb1901)
x-ESP8266-mode: sketch
PERFECT. Complete 100%

And after a reboot.

x-ESP-fw-ver: 1802012220
x-ESP-heap-free: 20384
x-ESP-SD-size: 1953282
x-ESP-SD-free: 1730645
x-ESP-SD-ver: 100
x-Voltage: 3055
x-ESP-free-space: 1564672
x-ESP-sketch-size: 531536 << Notice the size using 2.4
x-ESP-chip-size: 4194304
x-ESP-sdk-version: 2.1.0(deb1901)

just gotta work on that POST ARGS Error I get on some pages with 2.4 and were sorted.
Who's gonna be the first to do a 4 or 8meg update.?

Now just got to find a pretty way to optimise all the F() Strings.

Hv.

[d-a-v]

Happy for this to be included in the builds if desired

Why not making a pull request for these ?

On that subject of numbers what is the largest compiled sketch the ESP will run.

someone corrects me, I think 1MB is the max addressable program space by design.

[Hiddenvision]

Why not make a pull request.?

Ran out of coffee.
Plus I don't know how yet, and not sure if it is all correct, AND, so much more.
I tried to follow the guide to install from Git and was going to do some edits.
1+1 does not always equal 42. so another day.

But hey, I am pretty good at cooking a sponge cake.

I think 1MB is the max addressable program space by design.

Darn, 1MB limit, Who would'a thunk it.
I know it sounds horrible, but I hope your wrong. ;-)

Hv.

[Hiddenvision]

just gotta work on that POST ARGS Error.

Yep 2.4 handles things slightly different.
or is fussier about my code doing things wrong.
Either way , Sorted now.

[d-a-v]

Darn, 1MB limit, Who would'a thunk it.
I know it sounds horrible, but I hope your wrong. ;-)

I am wrong but that's another story.

[Hiddenvision]

This was the required edits for the new version boards.txt.py

That looked rubbish, Zip file below.

[Hiddenvision]

boards.txt.py.zip

[sukretniy]

Hello! Can you a make a pull request with posibility of OTA > 500k, or at least post your findings here?
Thank you.

[Hiddenvision]

hi @sukretniy

I thought I had posted my findings above.
It was never taken up so I sort of forgot all what I discovered now.
Odd as everyone was banging on about wanting it in previous and recent posts.

But you can either edit the boards txt file directly of edit the .py file to build a new text file.
The zip file above should point you in a sensible direction.
If not I can look at my files and see what I changed again.

I can say that my esp devices are working fine giving (1+1) 2MB of Code space and 2MB of FS.
that allows me to write upto 1M programs and also OTA with http.

Hv.

[Hiddenvision]

Yep just checked, all what I know is written above.
In-fact, it's fair to say what's written/included above is MORE than I currently know or remember.

The included zip files dotted about above will have the ldr files that were needed.
With those and the boards.txt.py file edited and run the new sizes should appear in the Arduino menus.

But too long ago for me to remember clearly so just feel your way thru and shout again when you get lost.

Hv.

[Hiddenvision]

From memory the python script above had an error.
Something stupid like a tab out of place, I forget.!

So this zip is from my current working environment.
Contains parts of the tools folder.
Naimly:
Boards.txt.py
The sdk folder containing the ld folder with the extra allocation files.
It maybe out of date now if other things have been added or replaced in the current release.
I have not updated my own environment since way back then.

tools.zip

Hv.

[Hiddenvision]

Looks like support for the 4M2M it is actually in the current release.
#4454
Although the 8M stuff has not been included for some reason.

Hv.

[sukretniy]

Thank you for your help.
Hmm, i thought you managed to do this

1: Download the update.Bin from http into FS,
2: Then flash a small Minimal Bin file from FS and reboot,
3: Then finally the minimal boot will simply mounts the FS and flash the main update.bin and reboot again.
4: On reboot delete the update bin if required.

Can not understand, how memory allocation
1M + 1M + 2MB
resolves issue of updating through OTA fw more then 512K.
From what @igrr said here here implies, that if you already have fw more then 512K you can OTA new firmware only less then free space (and free space is less then 512K)

Now I'm thinking on this

1. Using OTA, update current fw (that uses more than 512K) with very small transitional fw (it's only purpose is to update itself to new full fw)
2. Using OTA, small transitional firmware updates with new fw (with more than 512K)

With this approach my firmwares can be more than 512K and modules can have 1M total.

[devyte]

@Hiddenvision is talking about 4MB flash and 2MB spiffs.
@sukretniy is talking about 2MB flash and 1MB spiffs, and requires a smaller or no spiffs.

[Hiddenvision]

Ahh, 2M device.
Change the IC. they cost less than the time required to think of a solution.!
I guess you could stir up a new allocation of say 768K+768K+whatever is left for FS.

Otherwise, the concept of downloading a small update only bit of code then downloading the larger UPDATE works.
I tried that first before I did the re-allocation,
obviously never looked at it since as all my devices get slapped with 4M or 8M memory.

To do that just take a copy of your existing code and get rid of any code with exception of the stuff needed to do the OTA and perhaps some rescue code for those unforeseen fail events.
Then Have your device get the smaller update file, reboot and that will in turn OTA the real Larger Bin.

Having typed that I realise I never did the saving the BIN to FS and then Flashing from the FS.
I just did the double HTTP update.

I am away right now so will give it some more thought when I get back.

Hv.

[earlephilhower]

After looking over the chain, I can't seem to find an issue here. Closing, but if someone has a specific problem, feel free to reopen.