ESP8266 vs ESP32 compatibility?

[Spiritdude]

I just tried to test my LUA programs on the ESP32, and I realized the API of essential modules, with the same name, have changed: wifi and tmr in particular (there might be more, but those I encountered).

I ask, why? To use the same name "NodeMCU" and using the same module names, implies compatibility for me - is this because ESP32 NodeMCU is in development, and eventually compatibility is achieved or?

I coded JS with Mongoose OS last week, and uploaded the same JS code for ESP8266 on the ESP32, and it worked right away - I think NodeMCU as platform should achieve the same, unless this is disregarded by purpose.

Now, it won't be that hard to achieve better compatibility, so the old code works as well, e.g. tmr.now() and tmr.time() doesn't exist on ESP32 but those rather trivial to implement I assume. Yet, I can't assign tmr.now = function() on my own end in LUA (or let me know if I can), I have to choose another namespace/object name. For example I ended up with timer.* as wrapper for both https://github.com/Spiritdude/nodemcu-shell/blob/master/lib/timer.lua

Whoever decided to break the API - if this was the case - I want to say, this is a severe decision which has direct impact on the success of "NodeMCU" as a platform - perhaps lead developers might state the more long term plans of it.

I even thought to use LUA on Orange Pi Zero or Nano Pi Neo and alike devices running Linux like Armbian, writing for NodeMCU API with LUA once and running it on multiple IoT devices; but this only works when the API is stable.

Thinking in terms of

• NodeMCU/ESP8266
• NodeMCU/ESP32
• NodeMCU/Linux

Background: I'm new to ESP8266 & ESP32 and LUA (since a few weeks) and I really like it, especially with the NodeMCU platform - I otherwise would not have cared to code in LUA - I'm fine with JS as well (Mongoose OS or Espruino), but it discourages me that I have to keep two versions of the same LUA code or include arch=='esp8266' conditions because I run it on a newer processor and changed API. I currently work on a NodeMCU system platform with a shell (like bash) at its core, and an editor (like nano), running on the device itself: https://github.com/Spiritdude/nodemcu-shell (currently very experimental). I started to implement the same shell concept in JS on Mongoose OS (not yet released) and was happy to see my code running on both ESP8266 and ESP32, yet mJS (the JS engine) seems slower than LUA of NodeMCU and some important functionality is missing, like array sorting which I implemented in JS which slows down some of my code immensely, so I still consider LUA.

As an overall strategy I would propose (order of priority):

• making an effort to stay compatible or
• provide a compatibility layer (e.g. coded in LUA) for NodeMCU/ESP32 or
• provide a common layer for all platforms (if previous two points aren't considered, I might do this for my package anyway but it would be good if experienced developers with more LUA/ESP8266 experience leading that)
• only break API as an absolute last resort (just don't or rename the firmware)

Thoughts?

PS: Those who plan to use nodemcu-tool to upload LUA code to ESP32, I have a small patch which enables it to do so: https://github.com/Spiritdude/NodeMCU-Tool (the patch is under review and soonish in one way or another adapted in the official repo https://github.com/AndiDittrich/NodeMCU-Tool - btw, the reason nodemcu-tool didn't work for ESP32 NodeMCU was the missing node.info()).

[TerryE]

@Spiritdude Rene, I am one of the committers on the project. Thanks for your comments. A lot are valid. I've just been doing a lot of node.js / NodeRED coding for my Home Automation, and I am strruck by how similar in architecture the NodeNCU framework is to node.js, in that there are both non-blocking event-driven with similarities in the API callback approach.

In the case of the ESP8266 implementation this event-driven approach was forced on us by the event driven architecture of the non-OS SDK. However, the ESP32 uses an implementation of RTOS. and so has relaxed some of the constraints. But yes, I feel that we should unify the APIs except where the different architectures mean that the feature / functionality set of the ESP32 is far richer, and even in thee circumstances, we should provide an API superset. And one of our other committers has opened and RFD #2243 along similar lines of thinking.

I will post a more detailed response in the next few days.

[Spiritdude]

For those who code with ESP32, and want same API as of NodeMCU/ESP8266:

Lua: modules which provide some compatibility:

• https://github.com/Spiritdude/nodemcu-shell/blob/master/lib/http.lua : only http.get() for now
• https://github.com/Spiritdude/nodemcu-shell/blob/master/lib/gpiox.lua : gpiox.mode(), read(), write() and trig() (replaces gpio.*)
• https://github.com/Spiritdude/nodemcu-shell/blob/master/lib/timer.lua : timer.create() and its methods, (replaces tmr.*)
• https://github.com/Spiritdude/nodemcu-shell/blob/master/lib/rtctime.lua : rtctime module (set, get, epoch2cal)

Firmware: additionally I started to implement missing functions where Lua isn't suitable: https://github.com/Spiritdude/nodemcu-firmware/tree/dev-esp32

• tmr.now() and tmr.time() added (docs also updated), lib/timer.lua uses them if available
• gpio.mode() added (todo: pullup/opendrain)
• file.stat() added (size works, all other metadata doesn't work yet)

Regardless what will be decided about a common API, I add now code in my repos (either fw or lua) and we later can decide how those can be integrated (at some point the code has to come from somewhere).

Update

• 2018/02/11: gpio.mode() added
• 2018/02/12: file.stat() added

[Spiritdude]

@marcelstoer @TerryE @jmattsson

I need some guidance of you "old timers" of NodeMCU . . . as you saw I'm working on a clone/fork of the dev-esp32 at https://github.com/Spiritdude/nodemcu-firmware/tree/dev-esp32 and I head now to implement node.info() and try to do the right thing, e.g. provide some sane information as a return, as node.info() seems essential part of many external upload tools, and it needs to be addressed now as within the next couple of days or 1-2 weeks at max.

Now, there are several issues I like your input and guidance:

• node.chipid() (and therefore in future node.info()[chipid]) in NodeMCU/ESP32 breaks two implied conventions from NodeMCU/ESP8266:
  1. it not longer is integer decimal but already converted to hex with '0x' as prefix, a string now
  2. it no longer is trailer of the station MAC address (but 8 bit shifted to the right - irritating to me)
• node.info() should return the architecture (it seems most logical to me), e.g. 'esp8266' or 'esp32', and I propose the 9th and new element is such architecture; currently nodemcu-tool (the primary upload tool I use to code with NodeMCU/ESP32 does not work, only my patch tries to adapt but before my patch goes into the official repo, the node.info() must be decided and hopefully adapted).
• dev-esp32 should start to increment platform/include/user_version.h: NODE_VERSION_* to indicate progress and changes, right now it's 0.0.0.0.

@jmattsson: I started to walk through your contribution of dev-esp32, impressive work you did. I code again in C (after many years avoiding it) so NodeMCU/ESP32 becomes more usable - as you saw, I try to raise a small Lua system with a shell at the core, I require some consistency otherwise my Lua code becomes unmaintainable for both platforms - this is the reason I ask now.

I have no clue of each one's commitment in NodeMCU/* as of now, so I make no assumption of any way. If you say "I don't care", or whatever, would be good to say something, so I get an impression how much you (still) care of the direction this is heading for.

[TerryE]

@Spiritdude If you look at the commit history on the ESP32 branch, then you will see that Johnny and Arnim (@jmattsson and @devsaurus) are the main committers here.

I am still focusing on the ESP8266 and to be honest since I've spent the last three years building a house, I haven't had enough time for this let alone getting to grips with the ESP32. My current priorities are to do the LFS release for the ESP8266, then move on to getting Lua 5.3 out. At the moment the ESP32 Lua core is a fork of the ESP8266, and with 5.3 I want to move back to a single unified core codebase.

My particular core focus is the Lua engine internals.

[marcelstoer]

Thanks Rene, I don't work with the ESP32 but I see the issues you raised are all valid. Pretty much agree with you but please don't make Lua look like the acronym that it isn't (i.e. no caps) 😉

Looking forward to you PRs. I'm sure you already saw the contribution guidelines.

I have been a bit unhappy with node.info() for quite some time. There's IMO essential info missing. That's why NodeMCU built with my Docker image or the cloud builder spit out something like this at startup:

NodeMCU 2.1.0 built with Docker provided by frightanic.com
	branch: dev
	commit: 555b5574a24c21dd36e1e8c242ba7203c7d3af09
	SSL: true
 build created on 2018-02-07 22:24
 powered by Lua 5.1.4 on SDK 2.1.0(116b762)

So, whatever else you decide to add to node.info() is most welcome by me - as long as it's consistent on both chips. #1739 was raised by the community and is very much related.

[jmattsson]

Apologies for the disjointedness of my reply, but I don't have the time to make it better at the moment.

• To be able to augment the "tmr" module, you'll just have to replace the ROM table reference with a regular Lua table which in turn has the ROM module "inherited:

tmr = setmetatable({}, { __index = tmr })
tmr.foo = function() return 42 end

• In general, due to the huge differences in both hardware and SDKs between the ESP8266 and the ESP32, we made the call that for the native module interface we will forego "backwards" compatibility where it's the sane thing to do. We don't consider it a viable option to simply lift a Lua project from the 8266 and have it immediately work on the 32 - too many things are likely to be at least subtly different, and in those cases we have preferred to explicitly break them instead by changing the API. It's better to have things obviously not work, than have them silently fail sometime later. That said, we're certainly not against Lua overlay shims which do the necessary bits to provide the old API if it's possible to do so. When it comes to tmr.now() and tmr.time(), I originally omitted those because they aren't actually timer related, they're a) a counter, and b) system uptime, and as such don't really belong in that module (though maybe .now() does? it can certainly be argued). I'd prefer to see a node.uptime() to return the uptime, with a Lua shim to drag it back into the tmr as .time() only if necessary.

• And to further expand on the differences above; On the 8266 everything is based on the single core event driven architecture of the SDK. On the 32, we're instead limited (and free'd) by the dual-core RTOS approach. In the latter ordering guarantees that we took for granted on the 8266 no longer applies, and this at times bubble up into the Lua environment itself, where events may arrive out of order. The Lua VM is only one of many concurrently running tasks (e.g. the network stack runs concurrently), and while we've kept the VM itself single-threaded, the RTOS nature changes things significantly. The underlying feature set exposed by the SDK/IDF is also drastically different compared to the 8266, and some things simply aren't available, or are done in a very different manner. This too has necessarily impacted the module interfaces into Lua land.

• Due to lack of developer & committer time, we have lots of still-outstanding items on our ESP32 TODO list. We'd certainly welcome PRs :)

• For node.info(), I'd love to see us introduce a table-based return, e.g. obj = node.info(1) where we could then extend the returned information easily and without affecting backwards compatibility. Plus it's easier and clearer to do things like node.info(1).build_id vs _, _, _, _, _, _, _, _, build_id = node.info().

• The ESP32 branch hasn't actually seen any releases yet (afaik), hence why the version is still at 0.0.0. There's also the question of whether our release number should try to be in sync with the SDK / IDF versions.

[Spiritdude]

@jmattsson and others.

1. Thanks for the hint.

2. While it makes sense on the surface what you say, then I wonder for example, the gpio module, you changed the constants, instead to reuse the ones from ESP8266 you used new abbreviations for the same GPIO modes, why? gpio.mode() is ok, gpio.config() is ok as well, but now there is gpio.mode() in ESP8266 and gpio.config() for ESP32. I understand that some hardware and SDK likely dictates the way things work but just that example I wonder - what am I missing?

3. What is the biggest problem in ESP32 to interface Lua right now?

4. Thanks for the link.

5. node.info(1) agreed.

6. OK.

I think there is a misunderstanding between me and most of you committers, I assume NodeMCU is a brand, and stands for an API usable for ESP8266 and ESP32 - whereas for you it seems to stand for "making Lua run on the microcontroller using existing SDK" and caring little to nothing of Lua API compatibility between ESP8266 and ESP32, am I correct?

The past days I thought of what to reply, but I feel kind of blocked - I require tmr.now() and tmr.time() which are trivial to implement, as well math.sin,cos,tan,.. which seems commented out in lmathlib.c - and I don't know why. If you (@jmattsson and @devsaurus main coders of dev-esp32) like to have a choice whether tmr.time() or tmr.uptime() makes more sense then you operate already in the mode not to care of compatibility. I need the functionality, and best to have it compatible with NodeMCU/ESP8266, if you think to name it differently, and it may be just one function name or constant in a module doing the --same-- on both platforms, forces me to introduce an abstraction layer, and when I end up doing that for multiple modules I end up with a collection and I have to name such properly e.g. "NodeMCU/Universal" - since ESP32 has more RAM this is doable but avoidable:

I took your advise with metatable to do:
https://github.com/Spiritdude/nodemcu-shell/blob/master/lib/gpio.lua
https://github.com/Spiritdude/nodemcu-shell/blob/master/lib/tmr.lua (tmr.now() and tmr.time() actually need to be done in C to work in single process properly)

@jmattsson and @devsaurus Are you willing to change dev-esp32 API to be compatible with ESP8266 where suitable (e.g. coherent constant names and function names) or is this a no-go and you want to be 'free' to introduce new function names and constants and want to keep the existing function names and constants?

If you like to achieve better compatibility, I like to go with you into the details of each module and determine if NodeMCU/ESP8266 compatibility can be achieved (e.g. like tmr, gpio, node etc) and where the SDK underneath dictates another API altogether and the differences you introduced are justified (I may still make an effort to find a common API). Like making an overview of functions and constants (like a google document), and walk through them:

Functions/Consts  ESP8266      ESP32
gpio.mode()         x
gpio.config()                    x
gpio.read()         x            x
gpio.write()        x            x
gpio.IN                          x
gpio.INPUT          x
gpio.OUT                         x
gpio.OUTPUT         x
etc.

and then see if functions can be omitted or reimplemented in C or Lua level.

If you prefer another media like email to discuss this or have other preference, let me know, my email is spiritdude at gmail.

If anyone reading this has a better idea how to proceed, please let me know.

PS: the issue of node.info() resolved itself, NodeMCU-Tool repo accepted my patch - I was hoping to have better solution, but perhaps it's better to take the time to discuss general aims of compatibility, where node.info() is part of it.

[marcelstoer]

First of all...I like the idea behind nodemcu-shell, very cool. Looks like you're making good progress.

as well math.sin,cos,tan,.. which seems commented out in lmathlib.c - and I don't know why

That's due to resource constraints of ESP8266.

If you prefer another media like email to discuss this or have other preference

The fact that GitHub comments don't allow threading or branching off makes it sometimes really difficult to have effective discussions on topics like this one. However, we maintainers do pretty much everything in the open i.e. hardly ever in private emails or other closed sourced discussions.

I think there is a misunderstanding between me and most of you committers, I assume NodeMCU is a brand, and stands for an API usable for ESP8266 and ESP32 - whereas for you it seems to stand for "making Lua run on the microcontroller using existing SDK" and caring little to nothing of Lua API compatibility between ESP8266 and ESP32, am I correct?

It ain't so black and white I'd say. NodeMCU may be a brand (in China), I don't know, but the folks who started all this (devkits & this firmware) went off board in summer 2015. As Johny said "backwards" compatibility just wasn't the top priority for the ESP32 API. That doesn't mean that compatibility should be carelessly dropped when it'd be easy to maintain. On the other hand re-implementing something for a new platform, and that's why "backwards" is in quotes, gives you the nonrecurring opportunity to improve the API. Most of the essential modules where implemented way before the current maintainers came aboard.

The way I see it is that your nodemcu-shell use-case isn't a "mainstream" use-case. Anyone building Lua modules/libraries on top of NodeMCU would certainly be interested in a coherent API. End-users, however, wouldn't expect that and can easily be expected to adjust their Lua code I'd argue.

What hasn't been discussed here yet is the option of deprecating and eventually removing ESP8266 API. We've done this on a few occasions in the past (WiFi and net modules I believe).

I like to go with you into the details of each module and determine if NodeMCU/ESP8266 compatibility can be achieved

Personally I'd very much appreciate any effort from you to get this started. A GitHub wiki which is free for all to contribute would IMO be a good place to share and discuss: https://github.com/nodemcu/nodemcu-firmware/wiki/ BTW @jmattsson can we delete the ESP31-RTOS migration and drop the dev-rtos branch?
However, I think in your table it isn't obvious which functions are related. How about this:

GPIO

Feature	ESP8266	ESP32	Comments / Votes
setting mode	gpio.mode()	gpio.config()	Paul: Lua shim for ESP32

WiFi

Feature	ESP8266	ESP32
foo	wifi.foo()	wifi.bar()

And we only have to do this for current inconsistencies of course.

[TerryE]

It ain't so black and white I'd say. NodeMCU may be a brand (in China), I don't know, but the folks who started all this (devkits & this firmware) went off board in summer 2015.

Yes, NodeMCU started as a Chinese company with its own registered brand, and the owner and one of his colleagues did all of the initial work on porting Lua up to around version 0.91 if I recall correctly, but there wasn't enough money in the brand to make it financially viable to maintain the sort of commitment that it would take to continue to develop the firmware, so the owner actively transferred the day-to-day control to a bunch of the active contributors. All of the committers here do this work pro-bono, so it very much a community project, though all of us use the NodeMCU firmware for work or personal use.

[Spiritdude]

@marcelstoer the math functions are commented out in the dev-esp32 branch (as well), which I meant.

NodeMCU-Shell is an "app" (an example) which should run on ESP8266 and ESP32 using NodeMCU firmware, I assume many of us have ESP8266 and ESP32 devices, and I like to choose which hardware I need to run my code, I do not want to code specifically for one platform alone - I do want to reuse my code. I have Perl scripts which are 25 years old still working without changes, even running on a tiny ARM-based board. Lua has become for me a viable alternative for Perl - thanks to NodeMCU - but if I code in Lua, I like to reuse it from ESP8266, ESP32, ARM-based boards and on my servers - I don't mind writing file module to interface the io module on the larger systems because I see the benefit.

I started this wiki page https://github.com/nodemcu/nodemcu-firmware/wiki/ESP8266---ESP32-Compatibility (I used bullet list instead of table as a start, I'm not that experienced to do an elaborate table - feel free to change it if it's more suitable).

[pjsg]

My take is that we should have compatibility between both platforms unless it doesn't make sense. For example, the gpio constant names could have been the same. Arguably the new names are slightly better, but I don't think that is worth the incompatibility to change them.

I would be open to having a compiled in compatibility mode that created the extra functions and constants for ESP32 (to allow the running of code written for the esp8266).

What I really don't want is to have the same named API which does different things or takes different arguments.

[jmattsson]

My head really isn't anywhere near ESP32-space at the moment, so my recollections here may be off base - apologies if so.

Regarding the gpio module:

• The IN/OUT vs INPUT/OUTPUT look like oversights. I'd be happy to merge in an addition of INPUT / OUTPUT to get consistency.
• When it comes to config(), that's more complicated. The way of setting up GPIOs on the 32 changed compared to the 8266, in that it's an all-or-nothing approach. Thus, the old piece-meal way of the 8266 was no longer very workable without lots of hoops. The other big thing is the GPIO MUX capability of the 32, whereby just about any function can be routed to any pin. It's highly flexible, and when I looked at it I could conceive of no even remotely friendly way of exposing that functionality.
• As for math, I didn't try enabling it at the time, due to an outstanding issue with the 32 where task context switches could leave the FPU in an incorrect state, and corrupt results in return. Further, I was waiting for the 5.3 upgrade to become available since that would separate the floats from the ints, and allow for the different word sizes sensibly (iirc the 32 would otherwise end up with smaller max ints than the 8266 in return for faster floatingpoint performance - again, something I wasn't comfortable with at the time).

When I did the initial port of NodeMCU to the ESP32 it was done in the expectation that we would likely end up using the ESP32 at $work, and I was trying to get ahead of the curve. In the end we didn't go down the ESP32 path for various reasons, and my commitments elsewhere has meant I haven't had any time to dedicate to NodeMCU (be it 8266 or 32). Pull Requests for both fixes, improvements and new features are obviously welcomed, whether I'm around or not. I'm just the guy who got the ESP32 port off the ground, I am assuredly not the guy who will make it thrive, nor the Torvalds of NodeMCU on the ESP32 :)

[marcelstoer]

No activity in months an besides OP no one picked up on https://github.com/nodemcu/nodemcu-firmware/wiki/ESP8266---ESP32-Compatibility -> closing