ESP.getResetReason() doesn't allow to distinguish between timer wakeup and physical reset during deep sleep

[php4fan]

Consider these three different situations:

1. During program execution, the ESP gets reset by shorting RST to ground.
2. During deep sleep, the ESP gets woken up by externally shorting RST to ground.
3. The ESP has GPIO16 connected to RST and it gets up from deep sleep when the built-in timer goes off.

Expected:
ESP.getResetReason() should return different values in these situations, allowing you to distinguish 2 from 3 as well as from 1.

Observed:
ESP.getResetReason() returns "External System" in case 1, but it returns "Deep-Sleep Wake" both in cases 2 and 3, so they are not distinguishable.

I know that the way automatic wakeup works is that GPIO16 goes low, hence driving RST to ground exactly as you would do by physically shorting it to ground before the timer goes off, but there are plenty of ways you can internally accomplish the result of getting 3 different values for the 3 different sistuations. There is a built-in RTC that you are talking to after all, which keeps running during deep sleep.

[Pablo2048]

Well, your "plenty of ways" are just two IMHO - store this info to the (simulated) EEPROM, or use RTC storage (the really only way to accomplish this without burning out the Flash). I'm also using RTC storage in my application/s so I don't like that somewhere in the core is something, that can sometime overwrite my own content (or block part of RTC memory even when I not need this feature). There is no need to put this technique to the core, but you can do it in your own code the same way as you described. What do you think?

[php4fan]

store this info to the (simulated) EEPROM, or use RTC storage

None of those will help. The second is what getResetReason() already does and it's what allows it to distinguish between case 1 on one hand, and case 2 or 3 on the other, but not between 2 and 3. It writes something to RTC storage just before it goes to sleep, and it resets it when it wakes up: so, when it wakes up (before it resets whatever it is it wrote), if that thing is in the storage, it knows that it has waken up from deep sleep (as opposed to resetting during execution); but it can't know whether it woke up because the timer went off or because RST was externally shorted to ground.

I'm sure the RTC hardware interface provides a way to know whether it fired a wakeup or not.

Some time ago I looked a bit into this and found out the following.
There is a low level function of which I can't remember the name that returns the time of the RTC, that is, since it was last reset. I found out the actual code in the source code of the espressif SDK that actively reset the RTC at startup, so that function actually always return the time since boot, which makes it identical to another function that is supposed to do that.

Let's invent fake names for these functions whose actual names I don't remember, because otherwise this is confusing:

get_system_time() : returns time since program execution began
get_rtc_time(): returns time since the RTC was reset.

Because the SOFTWARE is resetting the RTC at startup, and there's nothing I can do to prevent it from doing that, those two return the same value (perhaps a few microseconds apart).

If only the software wasn't resetting the RTC at startup, I could easily know how logn it's been since I went to deepsleep, and, with a little bit of work (just knowing how long I had set the deepsleep to be) I could know within a small margin of error whether or not this is an automatic wakeup or not (or whether or not I've been asleep as long as the deepsleep was supposed to be).

[devyte]

My understanding was that it's not possible to distinguish the cases, but from your comments maybe that's wrong.
To discuss further, please provide the details you mention:

• code in the sdk that resets the RTC
• where it's called from
• function that tells time since last RTC reset

[php4fan]

My understanding was that it's not possible to distinguish the cases, but from your comments
maybe that's wrong.

To clarify:

• it is impossible for us users due to how the software is implemented.
• it is theoretically possible given the hardware. It is possible for the software (i.e. SDK and esp8266-Arduino) to do a better job, not shoot itself in the foot, and distinguish all the cases and/or enable us the users to do so.

However, I don't know exactly where the espressif SDK ends and esp8266-Arduino begins, so I'm not sure whether it's something that you can fix in esp8266-Arduino or it requires a fix by espressif in the SDK itself. Perhaps there might be something in the RTC system memory (or somewhere) that they have "forgot" to reset and that you may use as a workaround?

code in the sdk that resets the RTC

Unfortunately when I found it I didn't take notes.

function that tells time since last RTC reset

system_get_rtc_time()

Because the RTC gets reset at startup, system_get_rtc_time() becomes essentially a useless duplicate of system_get_time() with a small almost-fixed offset.

[devyte]

@php4fan we've discussed this internally. Assuming you're right, and it is in fact possible to differentiate between 2 and 3 via software, we consider the effort likely to be high for the return.
You're the user who needs this. Are you willing to pursue yourself? We can provide guidance and answer questions as we can, but the effort would need to be yours.

[php4fan]

Assuming you're right, and it is in fact possible to differentiate between 2 and 3 via software, we consider the effort likely to be high for the return.

Seriously??

I think you are seriously underestimating the "return". You are not thinking about the use cases.

You're the user who needs this.

That made me laugh hard. THE user. Right.

When googling for a solution I found quite a few instances (even from years ago) of people asking how to accomplish this (of course the answer always being you cannot).
Here's an example:
https://bbs.espressif.com/viewtopic.php?f=6&t=2383

Being able to know whether the board has been reset manually or woken up at the programmed time is a very basic need for any project, that currently is not satisfied. The "return" here is for any user (i.e. developer) to be able to do such a basic obvious thing. To overcome a huge, nonsensical limitation. I can't imagine an effort too big to be worth that.

Here's my example use case:
I have an IoT device that normally "does its thing" once a day. Its thing is to connect to the internet, download data from a given URL, and refresh an image on an e-paper. Because the maximum deep-sleep time is about three hours, it wakes up about every 3 hours, check whether it's time to do its thing (by counting the time it has woken up this way), and either do it or go back to sleep. But the user should also be able to force an update at any time by pressing a button. This button is just a reset button. Because of the issue at hand, the program can't tell whether it has woken up from deepsleep because the timer has gone off, or because the user has pressed the button. So a button press is indistinguishable from a wake-up.

So my current workaround is to instruct my users to push the button twice rather than once. That I can easily distinguish (I leave the how as an exercise to the reader). However this is a shitty user interface, counterintuitive AF. A single button press is something the user is never supposed to do, and if they do, it results in nothing immediately, and in the next update happening at the wrong time (only the next, and again I leave how I manage to trigger the following updates at the right time as an exercise to the reader).

If you replace "downloading data from the internet and refresh an image on an e-paper display" with "doing something", the description of my use case becomes:

"an IoT device that spends most of its time in deep sleep and periorically wakes up to do stuff. But it can also be forced to do that stuff at any given time by pressing a button".

To me, that seems like the description of practically any IoT device (and several kinds of non-IoT devices).

Of course, you can add hardware to accomplish the task. Either by adding an external RTC clock, or by adding circuitry that, at the press of the button, will keep a voltage high for a short time, and having the software read that voltage via an input (that implies that the button must be "decoupled" from the contact between RST and GND, because otherwise, the system wakeup would trigger the same thing that the button does; so, the button needs to trigger that voltage change and drive a switch that closes the contact between RST and GND. Just in case you missed how much circuitry one would need to add).

Having to add extra circuitry just because the software is too lazy to do its job - no, actually, because the software is overzealous and does something that it shouldn't even do, that is, reset the RTC.
It's a ridiculous amount of circuitry you need to add. In my case the limiting factor is that all the GPIOs of my ESP8266 are already taken by the interface with the e-paper display, so I wouldn't even have that one spare pin to read that hypothetical voltage triggered by the button (ignoring all the additional circuitry needed to do that), let alone to communicate with an external RTC.

Now, that's for the "return". Regarding the effort.

Ok so maybe you understand the above and are not underestimating the return, but the effort is indeed great.

Now, if tackled where it should be tackled, that is at the SDK level, it cannot be a great effort at all.
I think I have found where the RTC is reset:
https://github.com/espressif/ESP8266_RTOS_SDK/blob/master/components/esp8266/source/startup.c#L85

I'm not entirely sure, I can't find the source of rtc_init() but I guess it resets the RTC.

Anyway, I can easily be wrong about that line of code, but it's clear that all is needed is to not reset the RTC at startup.

Now, from a esp8266-Arduino perspective. Fixing it from esp8266-Arduino could indeed be difficult. Actually, it could be completely impossible. If the SDK code is resetting everything, I'm afraid there might be nothing you can do at the arduino level.

the effort would need to be yours

I'm afraid I wouldn't even know where to start.
I know nothing about programming an arduino core; I barely even know any C++ at all.
I have a hard enough time programming in Arduino already.

Anyway, here's what I tried and failed, in case it helps. Tried from the "user space" if I can call it that - from an arduino sketch.

1. Using preinit()

No luck there. When preinit() is called, the RTC has already been reset, so it doesn't help

2. This

// This is a .ino sketch
#include "user_interface.h"

class MyTestClass {

  private:
    uint32_t t = 0;
  public:
    MyTestClass() {
      this->t = system_get_rtc_time();
    }
    uint32_t getTime() {
      return this->t;
    }
};

MyTestClass test = MyTestClass();


const int SERIAL_BAUD_RATE = 74880;

uint32_t systime_before_init=0;

void preinit() {
  systime_before_init = test.getTime();
}

void setup() {
   Serial.begin(SERIAL_BAUD_RATE);

   Serial.println("---\n");
   Serial.printf("RTC time before init: %d\n", systime_before_init);
}

I thought this way I might be calling system_get_rtc_time() before whatever code resets the RTC.

And I suspect I actually am. The problem is, it doesn't work, it always returns 0.

I tried adding a delay(5) before the call, to rule out that this was still running after the RTC reset (hence meaning it would be a hopeless useless approach) no more than zero milliseconds after the RTC is reset. That doesn't seem to be the case, as even after adding the 5ms delay, I still get systematically 0.
So I guess I have managed to run the call before the RTC is reset, but probably it's called even before other initialization that is needed in order for system_get_rtc_time() to work as expected.

If I'm right, then that gives me some hope. Maybe we can just talk to the RTC directly and get the time.

But it might also be that I'm missing something and that this approach makes no sense.

[php4fan]

I think I have found where the RTC is reset:

Scratch that, I think I was looking at the wrong SDK. The relevant one is the NONOS, right?

Also, there seems to be no source code at all?

No wonder you were saying it's going to be difficult to fix...

[devyte]

Correct, now you're starting to understand: what you're requesting amounts to at least a partial reverse engineering of the rtc handling code, which is in the closed lib blobs from Espressif. The FreeRTOS sdk can provide some hints, but it means investing a large amount of effort, of the order of several months, and none of the current maintainers are willing to spend it. There is a very long list of much higher priority issues on our plates, and an even longer list of issues waiting after that.
So if you are unable/unwilling to pursue yourself, this should be closed.
As I said, if you do intend to pursue, we can answer questions about what we know.

[Tech-TX]

The RTC is reset by the blob during the boot phase, so the user needs to use RTC RAM to track changes across Deep Sleep. By the time the blob passes control to SYS the details needed to discern between Deep Sleep and external reset are gone. We aren't going to get help from Espressif on Non-OS SDK as they've abandoned it and moved on to IDF.

I haven't looked at the BBS post yet, but I'd guess it's saying the same. I'll look later when I get home.

Delay() doesn't run in preinit, it just returns immediately as the timers aren't set up yet.

[php4fan]

so the user needs to use RTC RAM to track changes across Deep Sleep

If you're talking about writing to RTC RAM and then reading what you (i.e. the user) have written in it, then again, you are talking about distinguishing between 1 and {2,3} (already accomplished by getResetReason()) but not between 2 and 3.

[php4fan]

The RTC is reset by the blob during the boot phase

So if I understand correctly, that is before the constructor of MyTestClass in my example runs, right? (I guess that ugly example of mine can be simplified by just writing instructions at the top of the sketch outside of setup(), but I was under the impression you can't do that)

Then I guess there's nothing at all we (we users, and I'm afraid also you) can do, if we don't get a chance to read from the RTC before it is reset.

[devyte]

Execution of the global object constructors is under our control. See file esp8266_main.cpp functions user_init() at the bottom and init_done() above it for the start sequence, including where preinit() and do_global_ctors() are called.
Of course, the app start happens only after eboot is already done.

[Tech-TX]

so the user needs to use RTC RAM to track changes across Deep Sleep

If you're talking about writing to RTC RAM and then reading what you (i.e. the user) have written in it, then again, you are talking about distinguishing between 1 and {2,3} (already accomplished by getResetReason()) but not between 2 and 3.

Yes, I saw the same issue when I was researching Deep Sleep, and couldn't find a spot early enough to save the RTC count. It was already cleared, thus impossible without a binary patch to the blob to NOP the RTC clear. That could be problematic if the blob does an integrity check.

[Tech-TX]

@php4fan One thing you might look at to see if it helps: FRC2, the free-running 32-bit counter that's used to set the ETS_Timer increments. It's apparently not cleared by the bootloader, so you might be able to read it in the RF_PRE_INIT() stage before it gets re-initialized. No bet on that, as I haven't looked at it yet. I'm pretty sure the ETS_Timers aren't running until somewhere around preinit(). I think the resolution of FRC2 is around 3.2 microseconds. If you grab it before it's cleared, then subtract off the time since boot began and your Deep Sleep interval, and if the remainder is a small number then it probably woke via GPIO16 and not external reset. I don't know how variable the boot time is, so it'll take some experimenting.

edit: No joy, sorry. On a cold start or after external reset, FRC2 shows a count of 222. After Deep Sleep (either RTC or external reset) it shows a count of 218, so it's cleared roughly 700us before RF_PRE_INIT(). FRC1 is no different: it's restarted before RF_PRE_INIT(). I also looked at the RTC counter, and it's count is all over the map in RF_PRE_INIT(). I didn't see anything that you could use there to determine reset cause.