Better support for using esptool as Python module (ESPTOOL-157)

[projectgus]

Following on from #157. It would be good to provide more generic-code-access friendly methods, for people building UIs and the like.

Non-exhaustive list:

• Move module-level generic functions into classes where possible.
• Test cases of keeping a single ESPLoader object around and calling multiple functions on it.
• Allow redirection of output to status callbacks rather than only print().

[marcelstoer]

This is most welcome. It's actually gotten worse since 1.3. Below is a comparison of the code I had to write for marcelstoer/nodemcu-pyflasher to get the same results (incl. console output) as what esptool.py produces.

It even got worse (i.e. more code that I had to write) between the latest beta and the final 2.0.

[projectgus]

Thanks for showing that. I guess I'm not super surprised, given the number of additional features in 2.0. But in both cases you essentially have to "mock up" a command line, and that shouldn't be necessary. I'll use your code as an example when it comes to designing a more useful Python API.

[murilopolese]

This would be indeed much appreciated! I have been trying to avoid "mocking up" a command line but I wasn't able to find my way through the code on how to do it (I'm just starting with Python). Thanks @marcelstoer for the snippet! This is great!

Before finding this solution I was fiddling around with a local copy of esptool.py where the main() function accepts a non required Namespace as argument. In case I called main() with this argument I would use it instead of the result of parser.parse_args(). It worked pretty well as I could perform all the actions I needed with a fairly small amount of code.

My instincts tell me that is not a very good idea but I can't reason exactly why. Could you help me understanding why the main() function shouldn't accept an optional Namespace argument? Or maybe it could but what problems that wouldn't solve and which new problems that would create?

[Pimmetje]

It would also be nice to be able to catch the print output (callback). But i guess that would require a lot of refactoring on the esptool

[marcelstoer]

@Pimmetje for my PyFlasher I had to redirect stdout for that: https://github.com/marcelstoer/nodemcu-pyflasher/blob/master/Main.py#L165

[Pimmetje]

Thx @marcelstoer your code gave me a clue on how to make it work. I can now get my compiled projects from my CI server and upload them to a ESP with a few clicks :). I wished arduino had a python version of AVRdude. Remark i had to hack the code a bit because the Serial port was not closed after a upload.

[marcelstoer]

I recently got an issue about exactly that: marcelstoer/nodemcu-pyflasher#37

[Pimmetje]

Thx, i already found that one. I also updated the esptool but no joy on that end

[murilopolese]

@Pimmetje for my PyFlasher I had to redirect stdout for that: https://github.com/marcelstoer/nodemcu-pyflasher/blob/master/Main.py#L165

Just to update it as it's now pointing to an unrelated line of code: https://github.com/marcelstoer/nodemcu-pyflasher/blob/861f75415c3874e7e2f1000e8ea419da70f6fd19/Main.py#L188

[projectgus]

One more requirement (noted in #407) is for the ESPLoader object to be able to track something about the state of the chip (ie has bootloader connection been established). But still allow for case where serial port is connected to bootloader before the ESPLoader function is even called (this can probably be done because the bootloader will respond to SYNC packets at any time, I think.)

[doayee]

+1 on this particularly with regards to error handling. I just published my ESP32 flasher gui but am conscious it could handle errors much better. Particularly with the case where the supplied binary sizes and start addresses are detected to result in overlap, for some reason I couldnt catch the ArgumentError that this is supposed to throw at all, though I appreciate I am new to the game?

[jziolkowski]

Any progress on that one? I want to build a custom GUI app using esptool under the hood and I'd really hate to have to use subprocess or hacking the code to achieve that ;)

[someburner]

I broke out a very old version of esptool into just the class components required for flashing esp8266, and made it so I could interact with the esp object without needing to invoke commands from a shell, and allowed for custom return values. But for the ESP32 this is rather daunting since there are so many hacks and changes added to __init__.py.

Being able to interact with an esp object and perform multiple read/write/other operations without having to parse stdout and without having to leave stub mode would be so much cleaner.

Could probably also just write a special command that does everything I want it to do but then I have to use a custom esptool.py version. And because there are all kinds of dependency checks with esp-idf, it might not play nice when trying to use other tools like the nvs_gen.py utility. The issue is that for anything other than a super basic manufacturing process you really need to have a variety of custom io with the chip. All of it is already there in esptool, just not easily accessible to developers, since each command must be run individually.

For example, in our production process we need to: read flash_id, check if XMC, apply XMC fix, then set custom SPI parameters to check SPI connection to a device connected to VSPI, then read the MAC so we have it in our database for customers and/or print on label, then generate NVS partition data, then write the binaries.

If all of this could be done in python with a standard esp object/class, it would make life so much easier.

[radimkarnis]

Thank you @someburner for the valuable insight, it is appreciated.

I agree with your opinion and can see how this is getting worse with every feature/hack being added to esptool.

There wasn't any progress or driving force to do this since 2019, I will try to change that and put this higher on our priority list!

[someburner]

I might even suggest not trying to change esptool.py now, but rather just make a library that is like esp-serial-flasher but for python, and keeping extensibility for custom manufacturing scripts in mind. Seems like that way you eliminate all the argparse stuff and boil it down to what the classes actually need. Then later they can be reconciled.

We have manufactured in the 10s of thousands of device for esp8266 (ESP-12S) and moving to ESP32 soon, I'm happy to provide more detailed input / testing.

[marcelstoer]

not trying to change esptool.py now, but rather just make a library

It should be possible to refactor all essential functionality out of the esptool.py into something like esptool-lib.py. I expect the only thing left in the former will be the argparse stuff and the calls to the latter.

[M-Bab]

Just another example how useful a smart API could be: I am using esptool-ftdi which resets the ESP via RTS/CTS instead of RTS/DTS. This script uses the very little documented functionality to replace the serial communication object. I could get it working with esptool~=3.0 but not with esptool>=4 because the external_esp object is significantly more complex to overwrite than the serial object.

This is obsolete now. It works just as fine with esptool 4 now.

[someburner]

I've modified esptool to my liking, but just sharing in case this ever happens:

When using something like pyinstaller to generate a bundled application, it has a hard time locating references to the stub json files, since loader.py uses a relative path that I suppose is not inferred properly.

Instead of hassling with a bunch of custom pyinstaller/scripting, I just made a script to wrap all the json files into their own python folders and import them as a string instead.

Pretty sure this is how esptool used to do it back in the day anyways. I'm sure there's ways to configure static resources but those always seem to break or the api changes.

from .pystubs import *

def get_stub_json_str(chip_name):
    chip_name = re.sub(r"[-()]", "", chip_name.lower())
    chip_name = chip_name.replace("esp", "")
    py_mod = "stub_flasher_" + chip_name + "._STUBSTR"
    return eval(py_mod)

class PyStubFlasher:
    """Loading static json files doesnt play well with pyinstaller. Just load
    json strings from a python module instead.
    """
    def __init__(self, jsStr):
        stub = json.loads(jsStr)
        self.text = base64.b64decode(stub["text"])
        self.text_start = stub["text_start"]
        self.entry = stub["entry"]

        try:
            self.data = base64.b64decode(stub["data"])
            self.data_start = stub["data_start"]
        except KeyError:
            self.data = None
            self.data_start = None

...

stub = PyStubFlasher(get_stub_json_str(self.CHIP_NAME))

And a script to just generate them into esptool/pystubs. E.g. esptool/pystubs/stub_flasher_32.py looks like:

#
# stub_flasher_32.py
# generated python wrapper for stub_flasher_32.json
#
_STUBSTR = """
{
    "entry": 1074521516,
    "text": "CAD0CR7kCBn0AuFWoJQlkQtQreSSAA",
    "text_start": 1074520064,
    "data": "CMD8Pw==",
    "data_start": 1073605544
}
"""

[someburner]

@radimkarnis

Another related feature that would be helpful:

Currently read_flash requires a filename to write data to. In a custom esptool application it may be preferred to avoid writing data to disk and simply return the data read as a byte array, could be an option or a different command.

Similarly it may be useful to provide input data streams for write_flash, although it seems like that might already be possible with current arg structure.

[mickeyl]

Very good ideas here, sadly very little progress. I really wonder what people use in their production lines.