Cannot connect if ESP8266 running already

[vpatron]

Would it be possible to have uPyLoader send a Ctrl-C to make sure an existing running program is stopped before uPyLoader tries to connect?

I have a program automatically run by boot.py and in order to connect, I have to use a serial terminal, hit Ctrl-C, delete my boot.py (import os; os.remove('boot.py'), connect with uPyLoader, restore my boot.py

This is very nice work by the way, and very useful. I did PyQt stuff years ago. If you give me some pointers, maybe I can fix it.

[BetaRavener]

How are you connecting to your ESP? If you're using serial, it should send Ctrl-C to the board when connecting (see here).

I'm probably forgot to do this in WiFi, so I wonder if you're using it or the problem is elsewhere.

[vpatron]

I'm actually connected via serial (onboard USB-to-serial on NodeMCU). I think what's happening is you may be sending the Ctrl-C, but I think there's a step you do with the serial DTR or RTS or something tells ESP chip or MicroPython to reset (from reading your code). If that is the case, then I think it's that reset that restarts MicroPython and makes the Ctrl-C ineffective. You would need to do the serial reset, wait a bit for the >>> prompt, and send a Ctrl-C if it does not get the prompt.

[BetaRavener]

It's interesting lead but that code only executes if you choose to by ticking "Issue Reset" checkbox next to Baud rate. Are you using it? Or do you notice that reset happens when connecting? You could check it by making a seconds counter and looking at values after the board had been running for a while or something like that.

On the side note, I have reviewed your current workflow and have some questions. What happens if you connect with uPyLoader while the program from boot.py is running? It should only have problem listing files (which will result in error), but the connection should be established anyway. You could then use terminal in uPyLoader to break the code and follow up from there.

[vpatron]

I can't remember about the "Issue Reset" I remember playing around with it but setting iddn't make a difference.

When I try to connect with uPyLoader (when it's been set to autorun a program via boot.py), I can never connect. It just times out. So I cannot even control it via the uPyLoader terminal. That's why I have to connect with a serial terminal, hit Ctrl-C, delete boot.py.

I'll try getting the seconds idea and/or maybe do some kind of blinky LED pattern to tell me the state. Will try in a day or two when I get time and report back. Thanks so much for the support!

[BetaRavener]

No problem, let me know what you'll learn. When I find some free time, I'll try to reproduce this behavior and check why it doesn't connect with debugger.

[vpatron]

Ok, when I click on "Connect", I get a blue LED flash, then my code starts running again (I can tell because I have the red LED flashing when my code runs). So basically, connect with "Issue reset" checked or unchecked seems to always reset the NodeMCU.

I tried some cheesy experiments and got it to work. I added a print and wait time to function send_kill() in the connection.py file and it connects fine:

    def send_kill(self):
        self.send_character("\3")
        print('sleeping 0.2 sec')
        time.sleep(0.2)

This function gets called 4 times because I see 4 prints on the console. 0.2 seconds is the fastest. If I set it to 0.1 second sleep, it does not connect. So some kind of timing thing? And why send kill 4 times?

[BetaRavener]

You probably want that sleep to happen before sending the Ctrl-C. I can't find the 4th occurrence but it gets called once after connecting and then before listing files and checking for transfer scripts, as these are individual operations that need to ensure nothing is running. I'll place short sleep in next version before sending that first kill on connection, something around 0.5s just to be sure (it won't be terribly noticeable).

I still wonder why your MCU gets reset on every connection. This is not normal behavior as far as I can tell, If it bothers you, as a first thing try powering it from external power supply through pins.

[vpatron]

Ah, right. If it's getting reset, then I'd want the delay before it sends Ctrl-C to give it time to finish booting. I'll also try an Adafruit Feather and also try external power as you suggested when I get some time. Thanks for the support!

[vpatron]

Ok, I think I figured out why it needed the delay. The program I am running traps the KeyboardInterrupt and closes stuff before exiting, so maybe that's why it needs extra time after sending the Ctrl-C?

Also, I think my boards are weird because they reset when I first run uPyLoader, resets when I click on Connect, and resets when I click on Disconnect. The really strange thing is when I connected two boards to USB ports, simply running your program resets both boards! Clicking Connect resets only the board being connected to. But clicking Disconnect resets both boards again! Very bizarre behavior. I tried opening and closing the serial port using a serial terminal (GtkTerm) and this reset does not happen.

I also tried using a Feather Huzzah with uPyLoader and it does not do any of these resets and it does not need the extra delays and works perfectly.

For the record, the problem boards are "HiLetGo" brand NodeMCU boards purchased on Oct 2016 from Amazon.com. I noticed the boards there now are marked "New Version".

I think it might be good to still add the extra delay after sending the first Ctrl-C.

[BetaRavener]

Thanks for the report. If the real cause is that you're trapping KeyboardInterupt, fixed delay won't help. So I'll probably add an option to settings that allows you to set custom delay. The thing about boards is surely weird, it seems as a problem in handling UART protocol.

I'm using plain ESP-12E, which you can get from eBay for under $2 with an adapter (small white PCB that you'll surely find when you search for esp-12e). You'll have to have some skill with soldering iron to solder chip onto adapter, or like me get a cheap toaster oven and some soldering paste, which is useful for SMD :) This way, you'll have plain chip with nothing else, which is nice if you're planing to power it from batteries and such. You'll also need an USB->UART converter (this one works nice https://goo.gl/GXrac8) and some power supply because that converter can't supply enough current. Put it all on breadboard and you're good to go, with no surprises later on.

[vpatron]

Thanks BetaRavener. I'm thinking a smart detection might be better: send Ctrl-C, wait for ">>>" up to 0.5 sec, and repeat maybe 5 times before giving up and reporting error.

Wow, those bare modules are cheap! If I do a board for a project I'll use those bare modules. For my project right now, I ordered the Sonoff TH16. It is a very good price and already comes in a nice enclosure, easy to mount, easy to wire the AC and wire the temp sensor, and easy to reprogram. It is for an attic fan and garage fan. I want the fan to only run if it is hotter than 30C, but only if outside temp is 25C or lower otherwise it makes no sense to run the fan.

Your uPyLoader (and Geany IDE) makes it super easy to develop in MicroPython for ESP8266!

I ordered 5 pcs of the USB-UART. Such a good price. Thanks!

[BetaRavener]

Yeah, you're right, I forgot that I'm able to check if I got prompt 👍 You can expect it pushed into source somewhere mid-May, I'm currently working on my diploma thesis day and night.. But it's nice to hear some positive feedback, really appreciate it.

And thanks for bringing that company to my attention. Those modules are quite cheap compared to what I've seen so far and I like that they got certification (I already got some smoking wall chargers from china, so unless it has certification, I'm not plugging it in 😄 ). The UI and enclosure are also neat. Anyway, sounds like great project, I like those kind of home improvements. Good luck!

[vpatron]

Hi, BetaRavener, it's me, Vince, again. I've been trying out the wifi connection and it is great! Except it doesn't break if a program is already running. It might be because I have a webrepl password. uPyLoader prompts me for a password, and then says "Connected". But then it errors with "Could Not List Files".

I have to go into the Terminal screen and send Ctrl-C, and then I see the message that program was stopped. I see "import os;os.listdir()" so it looks like uPyLoader sent that but forgot to send the Ctrl-C in the beginning.

By the way, this is on the Sonoff TH16 relay board. I got it working! The trick is to use the parameter "--file_mode dout" when programming with esptool.py

[BetaRavener]

Yeah you're right, WiFi is missing the Ctrl-C. Because it's really trivial, I just added it in right now. You can try it out if the problem got fixed.

[vpatron]

Thanks, but not quite working. I see that it now sends "import os;os.listdir()" when I open the Terminal dialog, and the ESP responds with a list of filenames, but these names do not appear in the main dialog box (list under "Remote MCU" text is blank).

I tried this with 3 different ESP boards (NodeMCU, Sonoff TH16, and LinkNode R4).

[BetaRavener]

Can you copy the full output of the terminal?

[vpatron]

In wifi connection mode, the "Remote (MCU)" is blank instead of showing the list of files. In the Terminal screen, I get:

import os;os.listdir()
['webpage.py', '__upload.py', '__download.py', 'fan_io.py', 'config.json', 'boot.py', 'simple_wlan.py', 'simple_rest.py', 'webrepl_cfg.py']
>>> 

I reverted back to previous commit and it works fine on ESP8266 Huzzah Feather. Errors on my old NodeMCU.

It works fine on both if I just add a 0.5 sec delay after the send_kill in wifi_connection.py. A 0.1 second delay does not help.

        self._auto_read_enabled = False
        self.send_kill()
        time.sleep(0.3)
        self.read_junk()
        self.ws.write("import os;os.listdir()\r\n")

[BetaRavener]

Hm, those pesky delays :) I think it's because your program takes a while to shutdown and therefore self.read_junk() won't catch it's output, which is left in the terminal. Then I execute code for listing files and read to next prompt, but there's one leftover prompt after your program ended, so it parses the junk and leaves list of files in the terminal (which is then read by auto-reader and displayed).

What I probably need to do is to check, if the parsed string contains echo of the command (that import os;os.listdir()) so I can be sure, that this is the output I should be parsing.

[vpatron]

What can I say, I like trapping Ctrl-C and shutting my code down gracefully, heh heh. Well, you know, that smart send_kill algorithm would fix all these issues :) Just saying.

Maybe I'll try fixing it. I'm a HW guy who writes code so I'm learning a lot poking around with your code. I need to use class inheritance more.

[vpatron]

Hello,

Ok, I made changes that make stopping an autorun program very reliable and have been using it for
several weeks with great success.

This might be related to the "Wifi connected but no files listed" problem also.

The big problem was that you keep resetting the device unnecessarily using the DTR and DSR lines
whenever a serial connection is made. Also, I'm not sure if your original logic was correct but below code works on NodeMCU. I left the old code in but commented out.

First, if reset is not called for, then don't change the DTR and DSR lines. Next, I use the correct reset
logic for NodeMCU is dtr=False, rts=True. The correct idle state for NodeMCU is dtr=False and rts=False.

In serial_connection.py:

        try:
            self._serial = serial.Serial(None, self._baud_rate, timeout=0, write_timeout=0.2)
            #self._serial.dtr = False
            #self._serial.rts = False
            self._serial.port = port
            self._serial.open()
            if reset:
                self._serial.dtr = False    # Set DTR high voltage
                self._serial.rts = True     # Set DSR low voltage
                #self._serial.rts = True
                time.sleep(0.1)
                #self._serial.rts = False
                self._serial.dtr = False
                self._serial.rts = False
                time.sleep(1.0)
                self.send_kill()            # Extra kill to stop autorun programs

The next code below fixes the annoying problem of all NodeMCU's on all serial ports are reset even if you requested a WiFi connection.

In connection_scanner.py:

        for port in ports:
            try:
                # Use correct baud rate when testing port. In Linux, another program can
                # listen in to same port for debugging. Keeping same baud rate preserves
                # all the messages that MicroPython sends during WebREPL.
                s = serial.Serial(baudrate=baud_rate)
                # Do not clear DTR and RTS. Doing so will reset *all* NodeMCU attached even if
                # user wants to use WiFi instead of serial.
                #s.dtr = False
                #s.rts = False
                s.port = port
                s.open()
                s.close()
                result.append(port)
            except (OSError, serial.SerialException):
                pass

These changes work beautifully on NodeMCU. I have not tried it on other ESP8266 boards but
should be fine. I'll try on Feather Huzzah, bare ESP8266, Sonoff switches, etc later when I get time.

The fix is in the code I forked from yours: https://github.com/vpatron/uPyLoader

[BetaRavener]

Great job, thanks for the effort. Truth is, establishing serial connection was one of first things I made when creating uPyLoader and haven't revisited it since then. At the time, I wasn't very sure about dtr and rts signals, but since it worked (at least most of the time), I didn't have incentive to change it.

I will review the changes, try it with bare ESP8266 and if I don't spot anything troubling, will merge them into my repository.

You're right that if these signals reset all devices on line, it could affect WiFi. However, the problem with listing files also happens when there's no UART connected, so there's still an issue probably with parsing the output now that there are some new delays.

[BetaRavener]

So, I have reviewed the code and need to have some discussion. First of all, why shouldn't I change the DTR & RTS in first place? In both places (unless reset == True) they are set to False, which as you say is an idle state. What if the lines were previously in different state, like reset? And how does setting them to False can cause reset?

The problem is, I can't connect to ESP when removing those lines. I have checked with multimeter and with debugger that when python Serial is opened, both lines are set to True, so they go low. I don't believe this is what we want.

Maybe NodeMCU is wired differently? I think I read somewhere that it's equipped to handle cases when both lines are pulled down (exactly the case when you comment out those lines in my code). But on bare module, where RTS goes to REST pin and DTR goes to GPIO0, this doesn't work and you need to explicitly set the lines.

[BetaRavener]

I have just pushed new code that tries to fight with already running scripts.

• If reset was issued, it will first wait for board to reboot and if it doesn't detects prompt in 10 seconds, it will send a kill command that should end any startup script. If script doesn't end itself in 5 seconds, connection is aborted and error displayed (previously, the uPyLoader would hang).
• Without reset it just sends kill in case there's a running program. The behavior is then same as above - waits 5 seconds and if still busy, ends with error.

I haven't changed the RTS/DTR behavior yet. First we need to know exactly what levels each device expects for normal operation / reset / programming. I believe that current state is correct for bare modules. However, the additional circuitry on Feather / NodeMCU might be interfering.

[vpatron]

Yes, I agree, we need to check DTR/RTS behavior for different ESP8266 boards and also Windows and Linux Python defaults when an open command is sent. I'll try to get some time tonight when I get home and put some test wires on various boards.

[vpatron]

Hi BetaRavener, I finally had the chance to wire DTR and RTS to an oscilloscope.

Ok, you're right. The better idle state should be DTR and RTS = True. But because of the 2 transistor circuit on the board, DTR, RTS both false is also idle.

Port open and Closed

State	/DTR	/RTS
Port Closed	3.3 Volts	3.3 Volts
Port Open	0 Volts	0 Volts

SW state of DTR and RTS

ser.dtr=	ser.rts=	/MCU_Reset	GPIO0, Flash	Comments
False	False	High (not reset)	High (button released)	State when port is closed
True	False	High (not reset)	Low (button pressed)	Flash mode, Button Pressed
False	True	Low (MCU reset)	High (button released)	MCU Reset
True	True	High (not reset)	High (button released)	State when port is opened

[BetaRavener]

Hi, thanks for trying this out. What device did you measure this on? Good job with the table, however, the last row seems a bit strange. I'd expect when rts == True the Reset line to be Low (so, in reset state), same as in third row.

I believe you measured this on NodeMCU. I couldn't find original discussion where I seen that thing that when NodeMCU's serial communication circuit handles the state when both lines go low, but found a better thing - schematics. You can also see there truth table for Auto Program Circuit which is the same as you observed.

However, on bare modules, you don't have this circuit and if you left both of the lines True (which is default when you open port), the device would be in permanent Reset (and / or flash mode). That's why leaving the lines as they were caused me problems. I think that this state (both lines True -> Low) is invalid and should be avoided. It's just that NodeMCU is equipped to handle it anyway, because some programs that actually don't set their RTS and DTR lines correctly would cause resets.

The final thing for discussion is - why commenting out the lines helped in your case. For example, you reported that boards get reset when listing ports. You should first try the new code and see if the problem is still there and if yes, we'll need to diagnose it.

[vpatron]

Hi, BR, I measured this on a NodeMCU. Yes, I was looking at the schematic you're showing.

In the logic table above, the first 3 rows should be true for all ESP8266 boards.

However, the last row is only true for boards that implement the 2 transistor circuit on DTR/RTS (transistors VT1 and VT2 in the schematic). This includes NodeMCU, WeMos D1, Adafruit Huzzah and others.

The bare ESP8266 board does not have this, so the last row in the table would hold the MCU into reset. It should look like this:

Only for bare ESP8266 boards:

ser.dtr=	ser.rts=	/MCU_Reset	GPIO0, Flash	Comments
True	True	Low (reset)	Low (button pressed)	State when port is opened

I agree that this state should be avoided because it is different depending on the board used so will cause different operation. However, it sort of cannot be avoided because as soon as you open the serial port, this is the state that Python puts the DTR and RTS lines at, dtr=True, rts=True.

So as soon as you open the serial port, you can then run dtr=False and rts=False. There would be a tiny glitch due to the time delay between open and setting dtr/rts.

You should probably do it in this order:
ser.open(...)
rts=False
dtr=False

Reason is that for boards with the 2 transistors, if there's a delay between setting RTS and DTR, it will be recognized as a GPIO0 button press instead of an MCU reset.

In any case, the ESP8266 will probably ignore a short glitch; it does seem to ignore it because the DTR and DSR do not go together exactly when I do ser.open() and ser.close() so even with the 2 transistor circuit there is still a tiny glitch on the MCU reset line. I didn't get a chance to measure how long the glitch lasts though. The MCU Reset timing should be in the ESP8266 datasheet somewhere.

I think the bottom line is, the port scan routine might still reset the board and the best you can do is immediately set DTR and RTS immediately after opening the serial port.

[BetaRavener]

ser.open(...)
rts=False
dtr=False

Why do you suggest this? Doing so will open port first, driving both lines low (default is true). Afterwards you would bring both lines back high. This will create glitch you described and it does cause reset on bare modules (tested).

Instead, first setting rts and dtr to false on serial object like it's done now and then opening it should start with both lines high immediately, without any glitches. The voltage level on those lines doesn't change until the port is actually opened. From pySerial documentation: "Set RTS line to specified logic level. It is possible to assign this value before opening the serial port, then the value is applied uppon open()."

I just can't see how I could be resetting the boards with transistors circuit. Could you share with me how are your boards normally connected? A photo would be also helpful. Also which version of NodeMCU board are you using? I might get one just for testing purposes. However, it will take a good amount of time to arrive, so it would be nice to solve this other way.

[vpatron]

Hi BR, ah, I didn't realize you can set the rts and dtr state first on serial object create. I just assumed there's no way to prevent resetting the bare boards. Ok, your logic sounds good.

But does port.close() make rts and dtr false? If yes then clicking Disconnect button would always reset the bare boards, right?

The 2 transistor circuit only resets whenever DTR=False and RTS=True.

I'll add the project to either github or my blog site I started and include pics of boards and transistor logic. Will let you know in a day or two when I have the article up.

[BetaRavener]

Actually I haven't observed any reset, but I'm mostly using WiFi.. I'll confirm it tomorrow when I have some time to test it and will also measure the levels in which the lines are left after closing the port. However, I'm afraid that this could differ from chip to chip, and now I mean USB-UART chips. NodeMCU uses CH340, while I'm using module with CP2102.