RFC What is missing from the Lua Developer FAQ

[TerryE]

Background

We are about to implement the LFS and this fundamentally changes best practice for how Lua developers code their NodeMCU applications. Incidentally, I have just updated an issue (#1119) with a view to issuing the PR for this immediately after we release LFS and I realised that it is 2 years since we created this issue. This got me thinking about two questions that it is worth my posing, answering and our subsequenty discussing, because I want to reflect the conclusions of such discussion in our FAQ.

• Two years ago, in what way did my Lua "best coding practice" differ from typical coding style adopted by of most of our Lua developers? Is there any material understanding / support for my techniques with the other committers? If so, then how do we get the message across to our felllow developers?

• In what way have my own practices evolved over thia last 2 years, and again the same agree/ diasgree / how do we get this message across set of qQs.

Differences in best practice 2 years ago

The lifecycle of GCOjects

Pretty much all of the RAM used by the LVM are what runtime referes to as Garbage Colllectable Objects (GCObjects): Tables, Arrays, Functions, Strings, Userdata, ertc. The Lua Garbage Collector (LGC) handles all of these in a standard way: they are created during the program execution of the application. Once they have been fully dereferenced, then the LCG will collect and reuse the storage: birth, life and death.

The developer has very limited RAM in ESP applications, and to get decent scalability, then some care has to be put into ensuring that oject has as short a life as possible; that they are created in as lazy a way as possible and that they are dereferenced immediately after their use is done.

Because the LGC does is its in-use scan from a number of roots: the _G table, the Lua registry, and the Lua stack, it is very easy to leave dangling references to objects should be dead (for example storing the reference in a global or in the registry, or in a table which contains other live entries

Another aspect of GCOjects is that are all stored in variables by reference, so assigning a GCOject to anothe Lua variable is a very lightweight operation; referencing the variable doesn't dereference the underlying object if other references exit elsewhere.

Local and upval variables

Lua applications should always make maximum use of local and upval variables in preference to global refences etc. This is for two primary reasons:

• These are incredibly cheap to create and access in runtime terms. For example the compiler emits direct register-to-register instruction for references to local variables, and the LVM will typically execute these in less than 10 lines of C. In contrast a reference to a global ROM constant requires clawing down maybe haf a dozen subroutine levels, scanning ROM tables, etc and can take a 1,000+ lines of C code to execute. So for example, replacing node.chipid() in the following code by a local copy chipid will speed this code up by roughly 40 times:

  for j = 1,n do local x = node.chipid() end

• The other big plus about locals is that they have a defined scope, and therefore a built-in lifespan. A local goes out of scope when execution returns from the function, and any GCOjects referred to in these locals will have the reference remove, so you don't need to set the local to nil to dereference the object. With a little bit of thought, you get good housekeeping as a free side-effect.

Upvals are also an incredibly cheap and effective way of privately sharing variables between functions and across invocations of the function(s). You just have to a little careful and remember than an upval isn't avaliable for collection until all of the closures which use it are themselves dereferenced.

Floating vs Integer builds.

Many developers chose Integer build option over Floating Point(FP) for two reasons: RAM usage and performance, but then they also have to complicate their coding to do everything by integer evaluations. However:

• Integer values smalller than 2^53 are stored in denomalised form in the FP builds and the FP routines already include special case handling of such integers. So once you've allow for the other LVM execution overheads, the addition of a couple of integer values, say, in a FP build is a 100% or so slower than the corresponding Integer version, and whilst this is a runtime overhead, it is relatively small compared to the performace impacts of not using locals properly or having the LGC badly configured.

• The fundamental internal storage unit is the TValue is 8 bytes on an integer build and 16 on a FP build. And this is a real issue as TValues might typically be 15-25% of RAM use.

However note the with Lua 5.3 there are no performance or storage diffrences between the integer and FP implementations so I will be recommnedning dropping support for integer builds.

Differences in my coding practices over this last two years

Standard vs Emergency Gargage collection

Out default setting runs the EGC full collection after every storage operation. If the standard LGC has been properly configured then this is unnecessary and slugs kills the performance of most applications. It is very easy to set up the standard LGC to do everything that your applications needs with a low-impact incremental collection, that has perhaps 10× less overhead than node.egc.ALWAYS. In my view, the EGC should be just that and set to node.egc.ON_ALLOC_FAILURE.

Cross compiling

Even though the Lua RTS supports on-device compilation, the RAM overheads of doing this are significant, and so the maximum size of source compilation unit is limited. This is too much of a PITA. It is just a lot easier to use luac.cross and this enables you to have larger loadable lc files.

SPIFFS imaging

Using tools like ESPlorer to bootstap loading a PSIFFS image is also a PITA. Most of my applications make very limited use of SPIFFS if any for R/W data. My standard build use a SPIFFS that is "big enough", and I just image it down on the device for first use using esptool. Simple and fast.

Using a decent provisioning system and server-side compilation

I now handle udpates using my provisioning system. This maintains a shadow directory on the provisioning host. If I need any change a source component, then I just make the change in the shadow directory using my favourite editor, and then issue a reprovision command to my ESP applicaiton: maybe 30s later the ESP has rebooted into the new application version.

LFS

This is in many ways "more of the same" moving Protos and Strings out of RAM and into Flash. Now coding 2,000 line applications is simple and straightforward.

Having a decent remote gdb to develop new libraries and core components

This isn't relevant to pure Lua developers, but this represents a step change in capability for library developers.

Node Red

This one is perhaps a strange one for most contributors and merits some explanation. Node Red is an IBM graphical IDE for built around a node.js framework. I use this for implementing all my server-side Home Automation / IoT integration. node.js is archtecturally very similar to NodeMCU Lua.

• Lua and javascript are very close in language capabilities. The main diffrence is a cosmetic one: Lua uses a Fortran-like lanquage contruct where as javascript is C-like. There are other sutble difference such as the scoping of closure upvals, but in reality there is little to choose between the two langauages. The main difference is one of performance on IoT devices. Whilst there are small memory VMs for javascript, LVM is perhaps 10x+ faster.

• Both Node.js and the NodeMCU firmware use a single threaded event driven non-blocking IO architecture.

So what I have found is that I now I to code my ESP Lua apps in a more node.js-like style and v.v. And this does materially improve the layout and maintainability of my Lua code.

It's a shame that we don't have a Node red-style wrapper for our Lua develoeprs as this would make it a lot easier easier for them to grasp this event driven non-blocking cding style.

[marcelstoer]

Since the issue title asks for "What is missing from ... FAQ" I thought I'd reference #1861 and #1862 here.

[TerryE]

I am again overwhelmed by the rich response from the other committers, so it looks like the next round of updates is going to be another solo effort. 😒

One area where we do need more detail is in the Build Options discussion. We need to explain what the user_*.h files are and why a developer might need to change them. I also think that the sort of summary that I discussed in marcelstoer/nodemcu-custom-build#27 would be useful.

[marcelstoer]

it looks like the next round of updates is going to be another solo effort.

Maybe to lessen the disappointment...did you notice that I ironed out all the (Markdown) formatting problems just before the recent master drop? No material changes but improved legibility.

[TerryE]

@marcelstoer thanks for this. Perhaps you can do a similar editorial filter on my next update? Much appreciated 😄

[marcelstoer]

Sure, just tag me when you think there's something I should review.