Compressed binary and SPIFFS image upload via wifi fails with ESP8266HTTPUpdateServer

[jan-gerard]

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: ESP-12-F
• Core Version: 2.6.3
• Development Env: Arduino IDE 1.8.12
• Operating System: Windows

Settings in IDE

• Module: Generic ESP8266 Module
• Flash Mode: dout
• Flash Size: 4MB
• lwip Variant: v2 Lower Memory
• Reset Method: nodemcu
• Flash Frequency: 40Mhz
• CPU Frequency: 80Mhz
• Upload Using: OTA

Problem Description

Compressed binaries (.gz, using gzip) sketch binaries and SPIFFS FS image files are not processed correctly by ESP8266HTTPUpdateServer. If I upload the uncompressed .bin file, it's working fine, but if I compress the .bin file with gzip (or 7zip), it's uploading, but the result is an empty file system on my ESP8266 for SPIFFS images files, and no program update for binaries (the old code keeps running). I have compiled my sketch with a line
#define ATOMIC_FS_UPDATE
included as described in
https://arduino-esp8266.readthedocs.io/en/latest/ota_updates/readme.html#compression Then I first uploaded the program with a wired serial interface (as described in the documentation), and later ran the actual test with a wireless connection.

I use a "Generic EPS8266 module" with setting "Flash size = 4MB (FS:2MB
OTA:~1019KB)".

I created the SPIFFS image .bin file with

%sketchdir%\mkspiffs.exe --size 0x1FA000 --page 256 --block 8192 -d 5
--create %sketchdir%\data %sketchdir%\spiffs.file.bin
%gzip% a -tgzip -mx9 %sketchdir%\spiffs.file.bin.gz
%sketchdir%\spiffs.file.bin

where %sketchdir% is the sketch folder (I copied mkspiffs.exe there for
convenience) and %gzip% points to 7z.exe on my machine. I also tried Linux' gzip.

I based my sketch upon
https://github.com/esp8266/Arduino/tree/master/libraries/ESP8266HTTPUpdateServer/examples/SecureWebUpdater
When I connect to it, and go to the update folder, I see four buttons. I
select the spiffs.file.bin.gz file with the FileSystem button, and press
"Update FileSystem". Then I wait, until the confirmation comes that it
was successful. I reboot, but then the FS is empty. While if I upload
the non-compressed file spiffs.file.bin, the filesystem contains the
desired files correctly. So the SPIFFS image is good, but the compressed version seems corrupt.
Similar for binaries: I can load a new program with the uncompressed .bin file, but compressed, the binary is transferred but not deployed over the original program.

According to
https://arduino-esp8266.readthedocs.io/en/latest/ota_updates/readme.html#compression
this should work when I define ATOMIC_FS_UPDATE. However, I cannot find
an example anywhere of of a sketch that includes this option.

I'm using Arduino 1.8.12 (latest), and ESP8266 2.6.3 (latest).

MCVE Sketch

#include <ESP8266WiFi.h>
#include <DNSServer.h>
#include <ESP8266WebServer.h>
#include <FS.h>
#include <ESP8266mDNS.h>
#include <WiFiUdp.h>
#include <ESP8266HTTPUpdateServer.h>

const byte DNS_PORT = 53;
IPAddress apIP(192, 168, 1, 1);
DNSServer dnsServer;
ESP8266WebServer webServer(80);

ESP8266HTTPUpdateServer httpUpdater;
#define ATOMIC_FS_UPDATE

char *ap_ssid = "xxx";
char *ap_password = "xxx";
const byte softAP_maxConnections = 8;
struct station_info *stat_info;
char *ap_hostname = "xxx.xx";
const char* update_path = "/update";
const char* update_username = "xxx";
const char* update_password = "xxx";

File fsUploadFile;

void setup() {
  Serial.begin(115200);
  setup_FileSystem();
  setup_WiFi();

  if (!MDNS.begin(ap_hostname))
    Serial.println("Error setting up MDNS responder!");
  else
    Serial.println("mDNS responder started");
  httpUpdater.setup(&webServer, update_path, update_username, update_password);
  setup_Webserver();
  webServer.begin();
  MDNS.addService("http", "tcp", 80);
  Serial.printf("HTTPUpdateServer ready! Open http://%s.local%s in your browser and login with username '%s' and password '%s'\n", ap_hostname, update_path, update_username, update_password);
}

void loop() {
  dnsServer.processNextRequest();
  webServer.handleClient();
  MDNS.update();
  delay(10); // ms
  yield();
}

void setup_WiFi() {
  byte softAP_channel = 0;
  bool ap_hidden = false;
  WiFi.mode(WIFI_AP);
  WiFi.softAPConfig(apIP, apIP, IPAddress(255, 255, 255, 0));
  WiFi.softAP(ap_ssid, ap_password, softAP_channel, ap_hidden, softAP_maxConnections);
  Serial.println("Connect to " + (String)ap_ssid + " with '" + (String)ap_password + "'");
  dnsServer.setErrorReplyCode(DNSReplyCode::NoError);
  dnsServer.start(DNS_PORT, "*", apIP);
}

void setup_FileSystem() {
  if (!SPIFFS.begin())
    Serial.println("Failed to mount file system");
  else
    Serial.println("Started spiff.");
  Dir dir = SPIFFS.openDir("/");
  while (dir.next()) {
    Serial.println(dir.fileName());
  }
}

void setup_Webserver() {
  webServer.onNotFound([]() {
    dirList();
  });
}

void dirList() {
  String html =
    "<html><head>\n"
    "</head><body>\n"
    "DIRECTORY LISTING:<br /><table>\n";
  Dir dir = SPIFFS.openDir("/");
  while (dir.next()) {
    html += "<tr><td><a href='." + (String)dir.fileName() + "'>";
    File f = dir.openFile("r");
    html += (String)dir.fileName() + "</a></td><td>(" + (String)f.size() + ")</td></tr>\n";
  }
  html +=
    "</table></body>\n"
    "</html>\n";
  webServer.send(200, "text/html", html);
  webServer.client().stop();
}

.... the rest of the code is not relevant for this topic (I think).

I tried to test everything thoroughly, and include all relevant information. But maybe I am still making a stupid mistake. However, I'm a bit surprised that I cannot find any online examples. @earlephilhower I think you are the expert on this topic since I see a lot of replies and updates on this topic.

[jan-gerard]

@earlephilhower If there is anything I can do to help investigate, please let me know.
Could it be related to my browser? Would that already apply compression before the transfer, so that actually I use compressed upload while I think I'm not, and that a double compressed file (browser compressing a .gz file) cannot be handled by the SecureWebUpdater code?

[jan-gerard]

I noticed using the developer console of Firefox, that the headers of the spiffs image upload (POST action) contain a line:
Accept-Encoding: gzip, deflate
So it might be that although I don't send a compressed file, it is transferred as gzipped data after all. On the other hand, another header says
Content-Length: 2072809
which is the size of the spiffs image file. I don't know if this header line should contain the true transfer size, or the uncompressed size. file: 2,072,576 bytes

[earlephilhower]

I'm not able to reproduce this, sorry.

Not being able to upload a compressed INO executable is a real head scratcher. It doesn't need to have any ATOMIC_FS_UPDATE defined (the binary is always flashed by the eboot, so it's always that way).

For a test, I took git head, built and ran the WebUpdater.ino example, then took an old .bin file (of a different app used to debug #7162), ran gzip BasicOTA.bin and then uploaded the resulting .gz file thru the web interface and it rebooted into BasicOTA, not WebUpdater.

If this is a remote app, you need to ensure you flash one uncompressed version with the new GZIP compatible bootloader or else the one in the flash itself won't be able to decompress things.

What is even odder in your setup is that the flash came back with the same app! If you send in a gzip and it didn't understand it, the old bootloader would just copy the GZ-compressed bits over the app flash and you'd end up in a reboot loop (since it'd be garbage to the CPU!). The fact that you're not getting that leads me to believe that you're not actually uploading anything at all.

Please start with the plain WebUpdater.ino and give that a try to avoid any possible issues brought in by another sketch or changes.

[TD-er]

The fact that you're not getting that leads me to believe that you're not actually uploading anything at all.

.. or the file is not considered as being complete ?
With the normal uploader (not zipped), an incomplete transfer results in a locked unit which has to be manually reset and then it reboots with the old sketch active.

[jan-gerard]

Hi,
First of all, thanks for looking into my issue and investigating.
I proceeded as you mentioned, and uploaded the WebUpdater.ino example, after changing the AP credentials of course. I can upload new sketches and SPIFFS images fine. I added a little bit of code to read the SPIFFS sector and display the files present, so I can verify if SPIFFS uploading is working. And I added a webserver response so I can check the version of the code, to see if it was updated indeed. All uploading via the web interface (./update) working fine for uncompressed files.
However, uploading compressed files did not work.
Then I noticed that the version of EPS8266 in my Arduino installation was the latest, 2,6,3; however the file types in the file ESP8266HTTPUpdateServer-impl.h were only .bin
<input type='file' accept='.bin' name='firmware'>
instead of .bin;.bin.gz
<input type='file' accept='.bin,.bin.gz' name='firmware'>
like in the latest version on github. So I concluded that something was outdated. Updating via the Arduino IDE did not resolve this, but I manually installed the ESP8266 libraries then from https://github.com/esp8266/Arduino. This is version 2.7.0-dev.

Now I can upload a compressed program file! (And the browser recognizes .bin.gz files.) So that part is working.

However, if I gzip any of the SPIFF image files, that fails. If I upload a SPIFFS.bin.gz image, the result is an empty file system (no errors though). I first uploaded an uncompressed SPIFFS image. That's fine, its showing all the files in it. But a compressed file results in an empty file system. I am using version 2.7.0-dev, and also the mkspiffs.exe from that version.
mkspiffs.exe --size 0x1FA000 --page 256 --block 8192 -d 5 --create %sketchdir%\data %sketchdir%\spiffs.file.bin
Which is then gzipped with the same command and binary as the successfully uploaded program-code. And the mkspiffs command is also working correctly, as I can upload the uncompressed SPIFFS image.

What could possibly go wrong? There are no errors or other feedback. I could add some debugging code to the esp8266 library and monitor responses via the serial interface, maybe?

[jan-gerard]

I enabled debugging by adding TRUE in init call.
ESP8266HTTPUpdateServer httpUpdater(true);
When I upload the SPIFFS image uncompressed, I get

13:18:32.343 -> Update: spiffs.file.bin
13:18:32.343 -> sleep disable
13:18:32.343 -> ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Update Success: 2072576
13:19:08.118 -> Rebooting...

And when compressed I get:

13:20:04.451 -> Update: spiffs.file.bin.gz
13:20:04.451 -> sleep disable
13:20:04.519 -> ..Update Success: 3174
13:20:04.587 -> Rebooting...

So far, that seems all right, but then the SPIFFS sector as FS reads it, is empty.
I cannot find out how the SPIFFS upload is further handled, so additional debugging stops for me now. If you have any suggestions what I could do to help troubleshooting, please let me know.

[earlephilhower]

@jan-gerard, can you confirm that GZIP executable binaries(sketch.bin.gz) do not work for you? I saw it earlier, and if that's not working then there's no point in worrying about FSes.

Eboot is very simple, and it doesn't distinguish between FS (when using atomic updates) and executable uploads. It just takes a chunk of raw data and writes/decompresses it somewhere:
https://github.com/esp8266/Arduino/blob/master/bootloaders/eboot/eboot.c

SPIFFS is pretty aggressive about formatting the flash if it can't see what it wants. So it's possible that an incorrect setting in the mkspiffs command would cause it to just format even if it was being uploaded and decompressed properly.

[jan-gerard]

@earlephilhower, on the contrary: I can confirm that GZIP executable binaries (sketch.bin.gz) DO work. Only FS uploads don't when compressed. If I take an image.bin file and upload, its fine. But if I gzip it and then upload, it fails.
I attached the FS image file to this post.
spiffs.file.bin.gz
I used memory setting 4MB (2MB SPIFFS, 1 MB OTA).
The only thing I can imagine is that the decompressor skips unzipping the 'empty' part of the archive (but if there is no information in there, that shouldn't matter), while the raw image file is a full 2MB of emptyness. Something is different anyway.
I guess I should add some more debugging lines to the code, but that is quite some reverse engineering to find out what is the whole flow of the upload process.

[earlephilhower]

I think I see your problem, @jan-gerard ! Your example has #define ATOMIC_FS_UPDATE inside the main sketch, but that does not apply to all files! So the main Updater.cpp is compiled w/o the flag and just immediately overwrites your FS with the raw compressed image.

You need to add a platform.local.txt file and restart the Arduino IDE. In the platform.local.txt you need to add a line

build.extra_flags=-DESP8266 -DATOMIC_FS_UPDATE

which will add the define to all compiled sources.

[jan-gerard]

Hi @earlephilhower, I tried your suggestion. I added the line build.extra_flags=-DESP8266 -DATOMIC_FS_UPDATE to the file C:\Users\xxxx\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.7.0-dev\platform.txt and indeed now all compiler commands contain -DATOMIC_FS_UPDATE when I enable 'show verbose output during compiling'. So I guess it is used correctly.
But unfortunately, until now till no result. I get ERROR[4]: Not Enough Space whatever I try to upload, either a plain SPIFFS image file, or a gzipped version. Something did change in the behavior, but not for the better.
I checked the program uploading via webupdater, and that still works (compressed and uncompressed), and uploading the data folder via the arduino python interface. That all works, and according to the speed with which a 2MB image is uploaded, its compressed as well. So the code on the ESP8266 is working fine, maybe. Only with the mkspiffs.exe versions I can find, it doesn't work. Maybe that is where the problem resides.

[earlephilhower]

It's now working as expected, but there's no space free to fit your updated FS.

You need to use an flash option which leaves space unoccupied. Say, 4MB with 1MB FS leaving 2MB free. Without unallocated flash space, there's nowhere for the uploader to temporarily store the bits.

At this point I suggest you transition to the Gitter channel https://gitter.im/esp8266/Arduino or the web forum https://www.esp8266.com as the Issue tracker here isn't the appropriate spot.

[jan-gerard]

All right, now it works indeed. I had to reduce the 'Flash size' setting in the Arduino IDE to '4MB (FS: 1MB, OTA ~1019kB)'. Then it worked. But with some side remarks:

• I had to change of course '--size 0x1FA000' into '--size 0xFB000' in the call to mkspiffs.exe
• It no longer works to upload an uncompressed SPIFFS image file, only gzippped ones get accepted (which is no problem)
• Now only 1MB is available as SPIFFS size, while with the plain uncompressed upload I could get it to work with 2 MB.

I was under the impression that a compressed FS image transfer would be optimal, in the sense that I could transfer quicker, but apparently when doing a uncompressed FS upload, it gets directly written to the SPIFFS image allocated memory, while a compressed transfer first needs to be stored in a separate location, and is uncompressed and written to the FS area afterwards.

So in the end, the functionality of having compressed FS image transfer does not offer much advantages, except when you only have a small amount of data; but then the compressed transfer is less of an advantage.

It was a nice exercise, but I will stick to the uncompressed scheme, using '4MB (FS: 2MB, OTA ~1019kB)'.

@earlephilhower Thanks for all your help and patience.