Question: Can I open two files from SPIFFS at same time?

[drschlaumeier]

Hallo,
I'm trying to optimize my code since I need to save RAM/Stack. See #4115
I came to the idea that I could move some stuff to SPIFFS. I want to read a file line by line and at the same time save some of that lines modified into another/new file on SPIFFS. So I have two files open, one for reading and one for writing.

However, If I try to open the second file for writing the ESP crashes.
Exception 3: LoadStoreError: Processor internal physical address or data error during load or store

Is it NOT allowed to have two files open the same time? So I need to search somewhere else why my code crashes?

Thanx in advance DRS

Here some code:

const char REPLACEFILE[] PROGMEM = "/somefilename.txt"; 
bool ReplaceStringInFile(const String& fname, const String& sreplace, const String& sdata){
    Serial.print(F("--> Entering ReplaceStringInFile: ")); Serial.println(fname);
    
    //open original file 
    File forg = SPIFFS.open(fname, "r");
    if (!forg) {
      Serial.print(F("  --> Error: Open File for read - ")); Serial.println(fname);
      return false;
    } 
    //open new temp file
    Serial.print(F("  --> Create tmp File: ")); Serial.println(REPLACEFILE);
    File fnew = SPIFFS.open(REPLACEFILE, "w"); //<--- CRASH IS HERE !!!!
    if (!fnew) {
      Serial.print(F("  --> Error: Open File for write - ")); Serial.println(REPLACEFILE);
      forg.close();
      return false;
    }
    
    //read line by line
    Serial.println(F("  --> Loop through file: "));
    String line;
    int i = 0;
    while(forg.available()) {
      Serial.print(String(i++));Serial.print(F(", ")); //print some line # for debugging
      line = forg.readStringUntil('\n');
      if(line.indexOf(sreplace) > 0){ //we could save this if and only call replace
        Serial.println(F("  --> Found replace string... "));
        line.replace(sreplace, sdata);      
      }
      fnew.print(line);
    }
    Serial.println("");
    forg.close();
    fnew.flush();
    fnew.close();
    
    //delete forg & rename fnew to forg
    SPIFFS.remove(fname);
    SPIFFS.rename(REPLACEFILE, fname);
    return true;
}

[ondabeach]

@drschlaumeier You can have multiple open files. Run this sketch to see how many open files your flash chip supports.

//Good tec data here: https://nurdspace.nl/ESP8266
//ESP class here: https://github.com/esp8266/Arduino/blob/master/cores/esp8266/Esp.cpp

#include "FS.h"

void setup() {
Serial.begin(9600);

SPIFFS.begin();
FSInfo fs_info;
SPIFFS.info(fs_info);

float fileTotalKB = (float)fs_info.totalBytes / 1024.0;
float fileUsedKB = (float)fs_info.usedBytes / 1024.0;

float flashChipSize = (float)ESP.getFlashChipSize() / 1024.0 / 1024.0;
float realFlashChipSize = (float)ESP.getFlashChipRealSize() / 1024.0 / 1024.0;
float flashFreq = (float)ESP.getFlashChipSpeed() / 1000.0 / 1000.0;
FlashMode_t ideMode = ESP.getFlashChipMode();

Serial.println("==========================================================");
Serial.println("Firmware: ");
Serial.printf(" Chip Id: %08X\n", ESP.getChipId());
Serial.print(" Core version: "); Serial.println(ESP.getCoreVersion());
Serial.print(" SDK version: "); Serial.println(ESP.getSdkVersion());
Serial.print(" Boot version: "); Serial.println(ESP.getBootVersion());
Serial.print(" Boot mode: "); Serial.println(ESP.getBootMode());

Serial.printf("__________________________\n\n");

Serial.println("Flash chip information: ");
Serial.printf(" Flash chip Id: %08X (for example: Id=001640E0 Manuf=E0, Device=4016 (swap bytes))\n", ESP.getFlashChipId());
Serial.printf(" Sketch thinks Flash RAM is size: "); Serial.print(flashChipSize); Serial.println(" MB");
Serial.print(" Actual size based on chip Id: "); Serial.print(realFlashChipSize); Serial.println(" MB ... given by (2^( "Device" - 1) / 8 / 1024");
Serial.print(" Flash frequency: "); Serial.print(flashFreq); Serial.println(" MHz");
Serial.printf(" Flash write mode: %s\n", (ideMode == FM_QIO ? "QIO" : ideMode == FM_QOUT ? "QOUT" : ideMode == FM_DIO ? "DIO" : ideMode == FM_DOUT ? "DOUT" : "UNKNOWN"));
Serial.printf(" CPU frequency: %u MHz\n\n", ESP.getCpuFreqMHz());
Serial.printf("__________________________\n\n");

Serial.println("File system (SPIFFS): ");
Serial.print(" Total KB: "); Serial.print(fileTotalKB); Serial.println(" KB");
Serial.print(" Used KB: "); Serial.print(fileUsedKB); Serial.println(" KB");
Serial.printf(" Block size: %lu\n", fs_info.blockSize);
Serial.printf(" Page size: %lu\n", fs_info.pageSize);
Serial.printf(" Maximum open files: %lu\n", fs_info.maxOpenFiles);
Serial.printf(" Maximum path length: %lu\n\n", fs_info.maxPathLength);
Serial.printf("__________________________\n\n");

Dir dir = SPIFFS.openDir("/");
Serial.println("SPIFFS directory {/} :");
while (dir.next()) {
Serial.print("File Name "); Serial.println(dir.fileName());
Serial.print("File Length "); Serial.println(dir.fileSize());
Serial.print("====== First 10 lines of file ("); Serial.print(dir.fileName()); Serial.println(") =======");
// open file for reading
File f = SPIFFS.open("/Test.txt", "r");
if (!f) {
Serial.println("file open failed");
}
else{
// read 10 strings from file
for (int i=1; i<=10; i++){
String s=f.readStringUntil('\n');
Serial.print(i);
Serial.print(" : ");
Serial.println(s);
}
}
Serial.println("==========================================================");
}

}

void loop() {
delay(3000);
}

[devyte]

@drschlaumeier
I think the problem is the filename string in PROGMEM.
What happens if instead of this:
File fnew = SPIFFS.open(REPLACEFILE, "w"); //<--- CRASH IS HERE !!!!

you do this:
File fnew = SPIFFS.open(String(REPLACEFILE), "w"); //<--- CRASH IS HERE !!!!
or this:
File fnew = SPIFFS.open("/somefilename.txt", "w"); //<--- CRASH IS HERE !!!!
?

[jonhp]

@ondabeach
I think line 24 has a syntax error. the embedded double-quotes around "Device" need to be escaped?

[devyte]

@ondabeach also, I believe that loop opens one file at a time during the iteration.
It's a good sketch for board info and configuration diagnostics, though. Maybe it'd make sense to add as an example.

[ondabeach]

Thanks @jonhp, I hadn't noticed that. I just went in to add the escapes and they were already there... I could swear I didn't fix it already so maybe my IDE has auto syntax correct :) Or maybe I had more glasses of vino that I thought ;)

Sorry @devyte, I was only answering OP's question of whether it was possible or not, and giving a way to check his chip's specs. I wasn't offering to rewrite his code for him. Is that considered poor form??

I agree, that sketch or something similar would be a good addition to examples. Especially since there's a flood of esp-01 (and probably other) modules with a new brand/family of flash chips( PUYA) hitting the market that aren't supported by the current(any?) version of SPIFFS. Having that sketch in examples would be a good way to get peeps to check their flash chip.

[devyte]

@ondabeach not at all, I'm just saying that OPs question was about multiple files opened simultaneously, while you said:

Run this sketch to see how many open files your flash chip supports.

Your example seems to open all files, but only one is open at any given moment. Unless I'm missing something? It could be modified to open multiple files to test, though.

But like I said above, I think OPs problem is elsewhere.

[ondabeach]

All good @devyte. The sketch tells you how many files can be open simultaneously using fs_info.maxOpenFiles, I guess it would've been more helpful to actually demonstrate doing so... but hey, call me lazy ;)

[devyte]

@ondabeach oh I did in fact miss that. But in older core versions there were other reasons that you couldn't have multiple files opened at the same time related to implementation details. I was thinking of that, and of of a test I did ages ago to check precisely if you could have multiple files opened simultaneously.
Anyways, will you be making a PR?

[ondabeach]

@drschlaumeier I might add a bit more to it first.

[drschlaumeier]

Hallo all, thanks a lot for all the input.

@devyte , was right: It was the PROGMEM.
I did File fnew = SPIFFS.open(FPSTR(REPLACEFILE,) "w"); and its working now. Slowly I unterstand when to use F() or PROGMEM or FPSTR or const String& .... when putting all the strings into flash and avoid wasting RAM.

@ondabeach , thanks for the sketch. fs_info.maxOpenFiles says max 5 open files. Interresting. I did not know the limit. It helped to further optimize the code.

It took some days to rewrite some code to save RAM/Stack but now I have more than 5k free and all works very stable. I also had to move some code from esp into html - javascript (e.g. dynamic generation of form data with jquery) to save space and improve the performance. The limits of esp82xx are reached very fast :-(.
I think I will move to esp32 for my next project. It has more power and RAM and costs nearly the same.

DRS

[pieman64]

@drschlaumeier the ESP32 comes with a whole host of problems. I will be sticking with ESP8266 for now.