Can't remove/reinsert SD card in SPI mode, esp32 just hangs

[josedpedroso]

Hardware:

Board: ESP32-cam
Core Installation version: esp32 boards 1.0.4
IDE name: Arduino 1.8.12
Flash Frequency: 80Mhz
PSRAM enabled: no
Upload Speed: 115200
Computer OS: Windows 7 x64

Description:

One part of my current project involves making a web-based file explorer for the SD card in an esp32-cam board. So, I decided to start with the SDWebServer example. It works fine, as long as I don't remove the card while the sketch is running. However, I'd like for that to be possible. Looking at sd_diskio.cpp, I can see that there are timeouts, but for some reason they're not working.

Sketch:

/*
  SDWebServer - Example WebServer with SD Card backend for esp8266

  Copyright (c) 2015 Hristo Gochkov. All rights reserved.
  This file is part of the WebServer library for Arduino environment.

  This library is free software; you can redistribute it and... etc etc

*/
#include <WiFi.h>
#include <WiFiClient.h>
#include <WebServer.h>
#include <ESPmDNS.h>
#include <SPI.h>
#include <SD.h>

#define SDCARD_SCK 14
#define SDCARD_MISO 2
#define SDCARD_MOSI 15
#define SDCARD_SS 13

#define DBG_OUTPUT_PORT Serial

const char* ssid = "********";
const char* password = "********";
const char* host = "esp32sd";

WebServer server(80);

static bool hasSD = false;
File uploadFile;


void returnOK() {
  server.send(200, "text/plain", "");
}

void returnFail(String msg) {
  server.send(500, "text/plain", msg + "\r\n");
}

bool loadFromSdCard(String path) {
  String dataType = "text/plain";
  if (path.endsWith("/")) {
    path += "index.htm";
  }

  if (path.endsWith(".src")) {
    path = path.substring(0, path.lastIndexOf("."));
  } else if (path.endsWith(".htm")) {
    dataType = "text/html";
  } else if (path.endsWith(".css")) {
    dataType = "text/css";
  } else if (path.endsWith(".js")) {
    dataType = "application/javascript";
  } else if (path.endsWith(".png")) {
    dataType = "image/png";
  } else if (path.endsWith(".gif")) {
    dataType = "image/gif";
  } else if (path.endsWith(".jpg")) {
    dataType = "image/jpeg";
  } else if (path.endsWith(".ico")) {
    dataType = "image/x-icon";
  } else if (path.endsWith(".xml")) {
    dataType = "text/xml";
  } else if (path.endsWith(".pdf")) {
    dataType = "application/pdf";
  } else if (path.endsWith(".zip")) {
    dataType = "application/zip";
  }

  File dataFile = SD.open(path.c_str());
  if (dataFile.isDirectory()) {
    path += "/index.htm";
    dataType = "text/html";
    dataFile = SD.open(path.c_str());
  }

  if (!dataFile) {
    return false;
  }

  if (server.hasArg("download")) {
    dataType = "application/octet-stream";
  }

  if (server.streamFile(dataFile, dataType) != dataFile.size()) {
    DBG_OUTPUT_PORT.println("Sent less data than expected!");
  }

  dataFile.close();
  return true;
}

void handleFileUpload() {
  if (server.uri() != "/edit") {
    return;
  }
  HTTPUpload& upload = server.upload();
  if (upload.status == UPLOAD_FILE_START) {
    if (SD.exists((char *)upload.filename.c_str())) {
      SD.remove((char *)upload.filename.c_str());
    }
    uploadFile = SD.open(upload.filename.c_str(), FILE_WRITE);
    DBG_OUTPUT_PORT.print("Upload: START, filename: "); DBG_OUTPUT_PORT.println(upload.filename);
  } else if (upload.status == UPLOAD_FILE_WRITE) {
    if (uploadFile) {
      uploadFile.write(upload.buf, upload.currentSize);
    }
    DBG_OUTPUT_PORT.print("Upload: WRITE, Bytes: "); DBG_OUTPUT_PORT.println(upload.currentSize);
  } else if (upload.status == UPLOAD_FILE_END) {
    if (uploadFile) {
      uploadFile.close();
    }
    DBG_OUTPUT_PORT.print("Upload: END, Size: "); DBG_OUTPUT_PORT.println(upload.totalSize);
  }
}

void deleteRecursive(String path) {
  File file = SD.open((char *)path.c_str());
  if (!file.isDirectory()) {
    file.close();
    SD.remove((char *)path.c_str());
    return;
  }

  file.rewindDirectory();
  while (true) {
    File entry = file.openNextFile();
    if (!entry) {
      break;
    }
    String entryPath = path + "/" + entry.name();
    if (entry.isDirectory()) {
      entry.close();
      deleteRecursive(entryPath);
    } else {
      entry.close();
      SD.remove((char *)entryPath.c_str());
    }
    yield();
  }

  SD.rmdir((char *)path.c_str());
  file.close();
}

void handleDelete() {
  if (server.args() == 0) {
    return returnFail("BAD ARGS");
  }
  String path = server.arg(0);
  if (path == "/" || !SD.exists((char *)path.c_str())) {
    returnFail("BAD PATH");
    return;
  }
  deleteRecursive(path);
  returnOK();
}

void handleCreate() {
  if (server.args() == 0) {
    return returnFail("BAD ARGS");
  }
  String path = server.arg(0);
  if (path == "/" || SD.exists((char *)path.c_str())) {
    returnFail("BAD PATH");
    return;
  }

  if (path.indexOf('.') > 0) {
    File file = SD.open((char *)path.c_str(), FILE_WRITE);
    if (file) {
      file.write(0);
      file.close();
    }
  } else {
    SD.mkdir((char *)path.c_str());
  }
  returnOK();
}

void printDirectory() {
  if (!server.hasArg("dir")) {
    return returnFail("BAD ARGS");
  }
  String path = server.arg("dir");
  if (path != "/" && !SD.exists((char *)path.c_str())) {
    return returnFail("BAD PATH");
  }
  File dir = SD.open((char *)path.c_str());
  path = String();
  if (!dir.isDirectory()) {
    dir.close();
    return returnFail("NOT DIR");
  }
  dir.rewindDirectory();
  server.setContentLength(CONTENT_LENGTH_UNKNOWN);
  server.send(200, "text/json", "");
  WiFiClient client = server.client();

  server.sendContent("[");
  for (int cnt = 0; true; ++cnt) {
    File entry = dir.openNextFile();
    if (!entry) {
      break;
    }

    String output;
    if (cnt > 0) {
      output = ',';
    }

    output += "{\"type\":\"";
    output += (entry.isDirectory()) ? "dir" : "file";
    output += "\",\"name\":\"";
    output += entry.name();
    output += "\"";
    output += "}";
    server.sendContent(output);
    entry.close();
  }
  server.sendContent("]");
  dir.close();
}

void handleNotFound() {
  if (hasSD && loadFromSdCard(server.uri())) {
    return;
  }
  String message = "SDCARD Not Detected\n\n";
  message += "URI: ";
  message += server.uri();
  message += "\nMethod: ";
  message += (server.method() == HTTP_GET) ? "GET" : "POST";
  message += "\nArguments: ";
  message += server.args();
  message += "\n";
  for (uint8_t i = 0; i < server.args(); i++) {
    message += " NAME:" + server.argName(i) + "\n VALUE:" + server.arg(i) + "\n";
  }
  server.send(404, "text/plain", message);
  DBG_OUTPUT_PORT.print(message);
}

void setup(void) {
  DBG_OUTPUT_PORT.begin(115200);
  DBG_OUTPUT_PORT.setDebugOutput(true);
  DBG_OUTPUT_PORT.print("\n");
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  DBG_OUTPUT_PORT.print("Connecting to ");
  DBG_OUTPUT_PORT.println(ssid);

  // Setup SPI for SD card
  SPI.begin(SDCARD_SCK, SDCARD_MISO, SDCARD_MOSI, SDCARD_SS);

  // Setup SD card
  if (SD.begin(SDCARD_SS) && (SD.cardType() != CARD_NONE)) {
    DBG_OUTPUT_PORT.println("SD Card initialized.");
    hasSD = true;
  }

  // Wait for wifi connection
  uint8_t i = 0;
  while (WiFi.status() != WL_CONNECTED && i++ < 20) {//wait 10 seconds
    delay(500);
  }
  if (i == 21) {
    DBG_OUTPUT_PORT.print("Could not connect to");
    DBG_OUTPUT_PORT.println(ssid);
    while (1) {
      delay(500);
    }
  }
  DBG_OUTPUT_PORT.print("Connected! IP address: ");
  DBG_OUTPUT_PORT.println(WiFi.localIP());

  if (MDNS.begin(host)) {
    MDNS.addService("http", "tcp", 80);
    DBG_OUTPUT_PORT.println("MDNS responder started");
    DBG_OUTPUT_PORT.print("You can now connect to http://");
    DBG_OUTPUT_PORT.print(host);
    DBG_OUTPUT_PORT.println(".local");
  }


  server.on("/list", HTTP_GET, printDirectory);
  server.on("/edit", HTTP_DELETE, handleDelete);
  server.on("/edit", HTTP_PUT, handleCreate);
  server.on("/edit", HTTP_POST, []() {
    returnOK();
  }, handleFileUpload);
  server.onNotFound(handleNotFound);

  server.begin();
  DBG_OUTPUT_PORT.println("HTTP server started");
}

void loop(void) {
  server.handleClient();
}

Debug Messages:

04:37:25.671 -> rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
04:37:25.671 -> configsip: 0, SPIWP:0xee
04:37:25.671 -> clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
04:37:25.671 -> mode:DIO, clock div:1
04:37:25.671 -> load:0x3fff0018,len:4
04:37:25.671 -> load:0x3fff001c,len:1216
04:37:25.705 -> ho 0 tail 12 room 4
04:37:25.705 -> load:0x40078000,len:9720
04:37:25.705 -> ho 0 tail 12 room 4
04:37:25.705 -> load:0x40080400,len:6352
04:37:25.705 -> entry 0x400806b8
04:37:25.941 -> ⸮�T⸮U⸮⸮⸮2-hal-cpu.c:178] setCpuFrequencyMhz(): PLL: 320 / 2 = 160 Mhz, APB: 80000000 Hz
04:37:25.975 -> 
04:37:26.076 -> [D][WiFiGeneric.cpp:337] _eventCallback(): Event: 0 - WIFI_READY
04:37:26.076 -> [D][WiFiGeneric.cpp:337] _eventCallback(): Event: 2 - STA_START
04:37:26.076 -> Connecting to ********
04:37:26.211 -> [D][WiFiGeneric.cpp:337] _eventCallback(): Event: 4 - STA_CONNECTED
04:37:26.245 -> [D][WiFiGeneric.cpp:337] _eventCallback(): Event: 7 - STA_GOT_IP
04:37:26.245 -> [D][WiFiGeneric.cpp:381] _eventCallback(): STA IP: 192.168.3.83, MASK: 255.255.255.0, GW: 192.168.3.1
04:37:26.380 -> SD Card initialized.
04:37:26.380 -> Connected! IP address: 192.168.3.83
04:37:26.380 -> MDNS responder started
04:37:26.380 -> You can now connect to http://esp32sd.local
04:37:26.380 -> HTTP server started
04:38:03.913 -> [E][WebServer.cpp:617] _handleRequest(): request handler not found
04:38:03.913 -> [D][WiFiClient.cpp:509] connected(): Disconnected: RES: 0, ERR: 128
04:38:21.328 -> [E][WebServer.cpp:617] _handleRequest(): request handler not found
04:38:21.328 -> [W][sd_diskio.cpp:137] sdCommand(): no token received
04:38:21.430 -> [W][sd_diskio.cpp:137] sdCommand(): no token received
04:38:21.531 -> [W][sd_diskio.cpp:137] sdCommand(): no token received

At 04:38:03, I loaded a page on my browser. Then, I removed the microsd card and, at 04:38:21, hit refresh on my browser, but there was no response. Some time later, I reinserted the same card, and hit stop and refresh again, but there was no response either.

1st workaround I tried:

extern "C" bool sdWait(uint8_t pdrv, int timeout);

class SwappableSD: public fs::SDFS {
	public:
		SwappableSD(FSImplPtr impl): SDFS(impl) {} //call parent constructor SDFS
		
		bool cardResponding() {
			if (_pdrv == 0xFF) {
				return false; //card was never initialized
			}
			
			return sdWait(_pdrv, 1000);
		}
};

#define new_SwappableSD SwappableSD(FSImplPtr(new VFSImpl()))

SwappableSD sdcard = new_SwappableSD;

bool init_sd() {
	if (sdcard.cardResponding()) {
		return true;
	}
	else {
		return sdcard.begin(SDCARD_SS) && (sdcard.cardType() != CARD_NONE);
	}
}

Then, on each request, I called init_sd(), and if it returned false, I just had the request fail. However, this doesn't work because the linker can't find sdWait(), and even if it did find it, I really don't know if this would fix the issue.

2nd workaround:

I just do SD.begin() and check SD.cardType() before serving any request, and afterwards, I call SD.end(). This works, but it's kinda ugly, and will probably still fail if I remove the card while it's reading or writing something.

[dsyleixa]

also different other ESP SD lib functions are still working faulty, e.g. see #3603

[rel1ct]

"Can't remove/reinsert SD card in SPI mode, esp32 just hangs"
Provide a link to the documentation that describes the possibility of these actions.

[josedpedroso]

Looking at sd_diskio.cpp, there are many calls to sdWait(), which means that the code can deal with timeouts, which should then allow the higher level SD library to fail gracefully, ie, calls would return false.

The log shows 3 "no token received" messages just before hanging. It's possible that it's hanging in an endless loop somewhere after that, instead of just returning a failure.

Since this looks more like a bug, rather than an intentional limitation, instead of going on a possible snipe hunt myself, I thought I'd check with the people who wrote, or at least are familiar with this code. That is, not you.

And it's possible, I provided a workaround, but it's inefficient and it's just covering what looks like a bug.

[rel1ct]

It would not occur to one sane person to remove a memory card from a working microcontroller.
If you really need it, then study the libraries and add the necessary functions (or fix the ones that do not work correctly).
No one will do this for you.

[josedpedroso]

I don't really see why it's up to you to decide what should and should not get fixed, or at the very least discussed with the devs here. Go do something useful instead of trolling random issues on github.

[rel1ct]

OK good luck.

[atanisoft]

@josedpedroso I suspect this usecase has not been implemented in the library itself for much the same reason as @rel1ct alludes to, why should the library support removal of the device it is actively using.. there are a lot of overheads to contend with in supporting removal that are not easily solved, one being closing/invalidating open file handles (not all are/can be tracked by the SD class!)

I would suggest if you need support for hot swapping of SD cards is to treat it similar to how Windows does.. provide a user option to gracefully shutdown usages and then inform the user to swap. This will involve SD.end() before removal and SD.begin() after a new SD has been inserted.

[lbernstone]

If you have a HARDWARE card detect pin, then it is possible to set an interrupt on the pin. If you paid $5 for a cheap SD module, you get what you paid for.

[josedpedroso]

there are a lot of overheads to contend with in supporting removal that are not easily solved, one being closing/invalidating open file handles (not all are/can be tracked by the SD class!)

That makes sense.

provide a user option to gracefully shutdown usages and then inform the user to swap.

I considered doing something like that, though for now, my workaround does the trick, and doesn't require user interaction. But it's possible I'll have to do just that.

If you have a HARDWARE card detect pin, then it is possible to set an interrupt on the pin. If you paid $5 for a cheap SD module, you get what you paid for.

The ESP32-cam doesn't have it, it's actually one of the things I checked while troubleshooting this.