Fail to open file for write on SPIFFS

[brianjmurrell]

Basic Infos

Same as in #4291

Sketch

#include <FS.h>

void setup() {

  Serial.begin(115200);

  SPIFFS.begin();

  FSInfo fs_info;
  SPIFFS.info(fs_info);
  printf("SPIFFS: %lu of %lu bytes used.\n",
         fs_info.usedBytes, fs_info.totalBytes);

  SPIFFS.remove("/stats-test.txt");
  File f = SPIFFS.open("/stats-test.txt", "w");
  if (!f) {
    Serial.println("Failed to open /stats-test.txt.  Nothing else is going to work.");
  }
  f.close();

}

loop() {
  File f = SPIFFS.open("/stats-test.txt", "a");
  if (!f) {
    Serial.println("Failed to open /stats-test.txt");
    return;
  }
  if (!f.println(msg)) {
    Serial.println("Failed to write to /stats-test.txt");
    f.close();
    return;
  }
  unsigned long fsize = f.size();
  f.close();

  if (fsize > last_fsize) {
    Serial.println("appending ");
  } else {
    Serial.println("stopping, nothing new written to file");
    
    FSInfo fs_info;
    SPIFFS.info(fs_info);
    printf("SPIFFS: %lu of %lu bytes used.\n",
           fs_info.usedBytes, fs_info.totalBytes);
    
    while (true) {
      delay(1000);
    }
  }
}

Debug Messages

SPIFFS: 2206039 of 2949250 bytes used.
Failed to open /stats-test.txt.  Nothing else is going to work.
Failed to open /stats-test.txt
Failed to open /stats-test.txt
Failed to open /stats-test.txt
Failed to open /stats-test.txt
Failed to open /stats-test.txt
Failed to open /stats-test.txt
Failed to open /stats-test.txt
Failed to open /stats-test.txt
Failed to open /stats-test.txt
Failed to open /stats-test.txt
Failed to open /stats-test.txt
stopping, nothing new written to file
SPIFFS: 2206039 of 2949250 bytes used.

Why did it fail to open so many times? Then it did finally open but the write added nothing new.

There is still lots of space available on the SPIFFS.

[ondabeach]

@brianjmurrell, I think you will find that you have a PUYA branded flash chip on your esp module.

Don't worry, you're not going crazy :)

There is a problem with SPIFFS and these chips that has not been resolved yet.
See these issues:
#4102
#4061
If you search the repository for PUYA and click on issues you'll find more.

I have sourced some replacement flash chips and am swapping them out until the issue is fixed. I tested your sketches on a 'fixed' module and it behaves as expected, but with a module that still has the PUYA chip I get the same results that you're getting.

[brianjmurrell]

How do I determine if I have said PUYA branded flash chip?

My problem doesn't seem to be as bad as the issues you linked. I was able to write the 2206039 byes of files that are on there with SPIFFS. It just seems to have stopped accepting writes at 2206039 bytes out of 2949250 instead of writing up to the max of 2949250 bytes.

[ondabeach]

With a magnifying glass, or you can run this sketch:

#include "FS.h"

void setup() {
Serial.begin(115200);

SPIFFS.begin();
FSInfo fs_info;
SPIFFS.info(fs_info);

float fileTotalKB = (float)fs_info.totalBytes / 1024.0;
float fileUsedKB = (float)fs_info.usedBytes / 1024.0;

float flashChipSize = (float)ESP.getFlashChipSize() / 1024.0 / 1024.0;
float realFlashChipSize = (float)ESP.getFlashChipRealSize() / 1024.0 / 1024.0;
float flashFreq = (float)ESP.getFlashChipSpeed() / 1000.0 / 1000.0;
FlashMode_t ideMode = ESP.getFlashChipMode();

Serial.println("==========================================================");
Serial.println("Firmware: ");
Serial.printf(" Chip Id: %08X\n", ESP.getChipId());
Serial.print(" Core version: "); Serial.println(ESP.getCoreVersion());
Serial.print(" SDK version: "); Serial.println(ESP.getSdkVersion());
Serial.print(" Boot version: "); Serial.println(ESP.getBootVersion());
Serial.print(" Boot mode: "); Serial.println(ESP.getBootMode());
Serial.printf("__________________________\n\n");

Serial.println("Flash chip information: ");
Serial.printf(" Flash chip Id: %08X (for example: Id=001640E0 Manuf=E0, Device=4016 (swap bytes))\n", ESP.getFlashChipId());
Serial.printf(" Sketch thinks Flash RAM is size: "); Serial.print(flashChipSize); Serial.println(" MB");
Serial.print(" Actual size based on chip Id: "); Serial.print(realFlashChipSize); Serial.println(" MB ... given by (2^( "Device" - 1) / 8 / 1024");
Serial.print(" Flash frequency: "); Serial.print(flashFreq); Serial.println(" MHz");
Serial.printf(" Flash write mode: %s\n", (ideMode == FM_QIO ? "QIO" : ideMode == FM_QOUT ? "QOUT" : ideMode == FM_DIO ? "DIO" : ideMode == FM_DOUT ? "DOUT" : "UNKNOWN"));
Serial.printf(" CPU frequency: %u MHz\n\n", ESP.getCpuFreqMHz());
Serial.printf("__________________________\n\n");

Serial.println("File system (SPIFFS): ");
Serial.print(" Total KB: "); Serial.print(fileTotalKB); Serial.println(" KB");
Serial.print(" Used KB: "); Serial.print(fileUsedKB); Serial.println(" KB");
Serial.printf(" Block size: %lu\n", fs_info.blockSize);
Serial.printf(" Page size: %lu\n", fs_info.pageSize);
Serial.printf(" Maximum open files: %lu\n", fs_info.maxOpenFiles);
Serial.printf(" Maximum path length: %lu\n\n", fs_info.maxPathLength);
Serial.printf("__________________________\n\n");

Dir dir = SPIFFS.openDir("/");
Serial.println("SPIFFS directory {/} :");
bool filesExist=false;
while (dir.next()) {
filesExist=true;
Serial.print("File Name "); Serial.println(dir.fileName());
Serial.print("File Length "); Serial.println(dir.fileSize());
}
if (!filesExist) {
Serial.println("No files in SPIFFS");
}

File f = SPIFFS.open("/formatComplete.txt", "w");
f.println("Format Complete");
f.close();
delay(200);

if (!SPIFFS.exists("/formatComplete.txt")) {
Serial.println("Please wait a few seconds for SPIFFS to be formatted");
SPIFFS.format();
delay(3000);
//Serial.println("Spiffs formatted");
}
f = SPIFFS.open("/formatComplete.txt", "w");
f.println("Format Complete");
f.close();
delay(200);

f = SPIFFS.open("/formatComplete.txt", "r");
if (!f) {
Serial.println("bugger... format failed.");
}
else {

String s=f.readStringUntil('\n');
Serial.print("First line of file : ");
Serial.println(s);

Serial.println("SPIFFS is formatted. Moving along...");
f.close();

}

dir = SPIFFS.openDir("/");
Serial.println("SPIFFS directory {/} :");
filesExist=false;
while (dir.next()) {
filesExist=true;
Serial.print("File Name "); Serial.println(dir.fileName());
Serial.print("File Length "); Serial.println(dir.fileSize());
}
if (!filesExist) {
Serial.println("No files in SPIFFS");
}

while (dir.next()) {
filesExist=true;
SPIFFS.remove(dir.fileName());
}
dir = SPIFFS.openDir("/");
Serial.println("SPIFFS directory {/} :");
filesExist=false;
while (dir.next()) {
filesExist=true;
Serial.print("File Name "); Serial.println(dir.fileName());
Serial.print("File Length "); Serial.println(dir.fileSize());
}
if (!filesExist) {
Serial.println("No files in SPIFFS");
}
delay(3000);
}

void loop() {
// open file for writing
File f1 = SPIFFS.open("/f1.txt", "w");
if (!f1) {
Serial.println("file open failed");
}
Serial.println("====== Writing to SPIFFS file =========");
// write 10 strings to file
for (int i=1; i<=10; i++){
f1.print("Millis() : ");
f1.println(millis());
Serial.println(millis());
}
f1.close();

// open file for reading
f1 = SPIFFS.open("/f1.txt", "r");
if (!f1) {
Serial.println("file open failed");
}

Serial.println("====== Reading from SPIFFS file =======");
// read 10 strings from file
for (int i=1; i<=10; i++){
String s=f1.readStringUntil('\n');
Serial.print(i);
Serial.print(":");
Serial.println(s);
}
// wait a few seconds before doing it all over again
delay(2000);

}

[brianjmurrell]

Yeah, found that sketch elsewhere.

Flash chip information: 
 Flash chip Id: 00164020 (for example: Id=001640E0 Manuf=E0, Device=4016 (swap bytes))
 Sketch thinks Flash RAM is size: 4.00 MB
 Actual size based on chip Id: 4.00 MB ... given by (2^( "Device" - 1) / 8 / 1024
 Flash frequency: 40.00 MHz
 Flash write mode: DIO

As for the magnifying glass, I tried that but didn't see anything with PUYA on it.

[ondabeach]

Ok, interesting. The "Flash chip Id" of all the ones I have is 00146085, but you're getting the exact same symptoms.

Here's a picture of a PUYA chipped module:

[brianjmurrell]

My board is a NodeMCU with a "shield" over that part I would guess. I'm not going to remove it. :-)

So looks like I don't have a PUYA given that mine is Manuf E0 and yours is 85.

[ondabeach]

Ah, yes can't say I can blame you for not wanting to remove the shield.

Here's the relevant debug line from my PUYA chipped module:
Flash chip Id: 00146085 (for example: Id=001640E0 Manuf=E0, Device=4016 (swap bytes))
and yours:
Flash chip Id: 00164020 (for example: Id=001640E0 Manuf=E0, Device=4016 (swap bytes))
Your manufacturer code is 20. The E0 is just the example info.

Are you getting the read fails like this:
====== Writing to SPIFFS file =========
4771
4771
4771
4771
4771
4771
4771
4771
4771
4771
====== Reading from SPIFFS file =======
1:
2:
3:
4:
5:
6:
etc.

On a healthy module, the 1:, 2: etc should be followed by the same numbers as the corresponding lines in the write section.

[brianjmurrell]

Oh, yes, of course. s/E0/20/ for mine.

Are you getting the read fails like this:

Generally no. reads and writes generally work -- until I hit 2206039 bytes out of the max. of 2949250 that SPIFFS.info() is giving me.

I'm not convinced that my problem is the same as yours.

[ondabeach]

Ok, if you're reading fine in the lower addresses then no , not the same, sorry didn't catch that. It's almost as if the flash chip is smaller than reported.

Have you tried with 2.3 core? Going the other way have you tried the latest git or just the 2.4 release?

[brianjmurrell]

Have you tried with 2.3 core?

As in https://github.com/esp8266/Arduino/releases/download/2.3.0/esp8266-2.3.0.zip? If yes, then no, I have not.

Going the other way have you tried the latest git or just the 2.4 release?

No, just using the latest GA release, 2.4.0.

[ondabeach]

You can simply downgrade/upgrade between the 2.3 and 2.4 release in boards manager.

[ondabeach]

I've never thought to try, no need or point as far as I'm aware. I only ever simply use Spiffs.begin.

No idea if that could be your problem but certainly worth trying.

[ondabeach]

Good luck.. I'm off to bed :)

[brianjmurrell]

just a related question.

Is it a bad idea to Spiffs.begin() and Spiffs.end() thruout the program
Or is it best to leave it Begun.

To be clear, I (the OP) don't use SPIFFS.end() anywhere so I'm not sure how the question is related.

I've never thought to try, no need or point as far as I'm aware. I only ever simply use Spiffs.begin.

No idea if that could be your problem but certainly worth trying.

Per above, I am not using SPIFFS.end() anywhere nor do I see why I would since my sketch is constantly writing to files.

[earlephilhower]

@TD-er, is this what you were talking about for #6340 ? SPIFFS not automatically clearing freed space and returning it to the free pool, so even though there's space left you can't write?

[TD-er]

It looks a bit like it.

What I know of SPIFFS:

• At least 2 free blocks (8k) are needed for SPIFFS to do its administration.
• A block can only be freed (erased and marked available) if all pages in it are marked to be deleted
• SPIFFS code is not always calling the garbage collection when out of free (erased) blocks (design choice with performance in mind?)
• call to the Garbage Collection with 0 as parameter does not move sectors (like defrag) to make a free block. This is the only call internal in the SPIFFS code to the garbage collection function.
• Garbage collection function does not check all possible blocks to be freed per call to it.

Due to these design choices, the file system may become fragmented and thus not all storage capacity can be used.
Even when the file system is not fragmented, at least 2 free (erased) blocks must remain available.
So at least 16k must be free, but in practice it can be as much as 32 times as much when of all blocks eligible to be erased, 1 page is still not marked for deletion.

I think if you also add a call to the gc() function with the parameter and call it defrag, you may be able to clear a lot of space, but it will take quite some time and I doubt if it can do anything if there is no free block left.
Not sure if it can be called on a mounted file system.

[bmidgley]

It might not be obvious to everyone that the Tools->Flash Size arduino menu could get reset to FS:none. It might not be the same symptom since I think SPIFFS.begin() returns null when there is no fs space.

[earlephilhower]

SPIFFS is no longer maintained by the author. There are some weird corner cases when the filesystem is under heavy use involving GC/etc. We have added the ::gc() method which can be called as desired (because it may have a long runtime). The suggestion for issues like this is to mode to LittleFS which is presently under active support by the ARMbed folks.