LFS follow-up issues and missing features.

[TerryE]

Rather than following the Bugrep or Feature request templates, this issue is more of a post LFS merge follow up to #2292 (which really discussed trade-offs up to our LFS implementation and at 48 comments was large and becoming unwieldy to load). I will use this issue to track LFS-related feedback and functional gaps. We can use this for a catch-all for the next few weeks until our next LFS patch update, and after this we can use the normal Bugrep / FR templates to track any follow as individual issues.

[TerryE]

Support for luac.cross and spiffsimg

See #2315. We now have WSL and Cygwin methods which cover all WinX versions. Of course the other option is to buy yourself a $15 RPi Zero and load DietPi and make luac.cross and use a small ssh script (either WSL native SSH or OpenSSH) from your PC to offload the compiles to the RPi0.

Cloud builder generating LFS firmware

In progress. Completed. See marcelstoer/nodemcu-custom-build#29

Additional LFS-enabled Lua Modules

In addition to lua_examples/lfs I have added PR #2416 (Telnet) and #2417 (FTP). (Now merged.)

Compression Support

In progress. See #2409.

Updates to FAQs and LFS whitepaper

Pending feedback from users.

[HHHartmann]

I recently got my first LFS installation running. Here are some obstacles and suggestions I came across:

I used a docker build in standard configuration which built an integer and a float image.
I decided to flash the float image (which gets built before the int image)

I then found that aside the local/fs directory a local/lua directory is recognized by the tools make.

1. suggestion: add .gitignore file to local/lua to have it created automatically on checkout (just like in local/fs)

so I put some lua files in there and built an LFS image (make in tools directory, image stored in local/fs).

2. Suggestion: in the whitepaper you use LFS image once for a firmware image that is LFS enabled and once for an image that contains compiled lua and can be flashed in the firmware image. I had to read it several times to get it sorted out.

I then transferred the generated image to the ESP and called ''node.flashreload()''
No flashing took place and no reboot.
After a sleepless night it occurred to me that there could be a difference in int and float builds.
So building the int after the float version had overwritten the float version of luac.cross.
So I rebuilt the float version, rebuilt the LFS lua image and could load my first modules as LFS. chacka!

3. Suggestion: have only one version of loac.cross with a switch to generate int or float images.

4. Suggestion: have ''node.flashreload()'' return error messages if the given image does not fit for any reason (int/float mismatch, absolute/relative mismatch and size).

[TerryE]

@HHHartmann Gregor, thanks and in reply to your comments,

• The "standard" docker build. Marcel developed this as an evolutionary step from the Travis process. In Travis it makes sense to do the two build variants, but in a personal docker build, it make so sense at all to do both, IMO. You either use an Integer builds or a FP one as a matter of personal preference, but not both. You should just do the one that you need and this halves the build time or more. If it generates two builds then it should keep all variant-specific products that are going to be used. I don't use docker, so I haven't been exposed to this. It's on my TODO list, but this gains me nothing so its a low priority for me ATM. Are you interested in helping here? I can provide and guru bash help if you need it.

• Cross mode compilation. There are all sorts of subtle internal changes in the generated code for the integer and FP variants and inbuilt checking. Because the luac.cross build reuses the standard lua environment (and this is 90% of the luac.cross code), trying to support the other variant cross compilation (e.g. a integer output from a FP compiled luac) would be a nightmare. So sorry no: this is never going happen.

• Cross image detection. I added this after feedback from Phil and Johny. (I only use FP myself and only do integer builds for regression testing, so I missed this in the first LFS version.) If a problem is detected once the flash build has started you can't return to the app; you have to reboot, but there are a bunch of prologue checks that are done (e.g. is the image of the right type) that should return with a meaningful error. One for the next PR, so 👍 for this suggestion.

• LFS additions to the documentation and examples. Since both you and @vsky279 Lukáš are actively engaged here, I will open a new issue on the subject of LFS documentation so we can focus our documentation discussions there.

[nwf]

@TerryE a very minor thing, but running node.flashreload("luac.out") on my test board causes the next reboot to be a hardware WDT reboot (node.bootreason() returns 4, 1) rather than, say, a sofware reboot. This causes my init.lua to freak out and drop to its emergency wait state. The next time through, of course, things are fine.

[TerryE]

@nwf, you can't do a normal system_restart () because this is executed by the SDK after you have rolled back through the stack and returned control to the SDK, but you can't return from flash_restart() because the old ROstrt has been overwritten, so the Lua VM is inconsistent. So it's either a while (1) {} loop or using the boot ROM restart. The current version in dev does the former. I've switched to the latter in my latest PR

[TerryE]

@HHHartmann @joysfera et al, are you aware of the luac.cross -e option? This allows you run Lua scripts on the host using the cross compiler. I originally added a runtime capability to luac.cross for testing purposes -- it's a lot easier to test changes on the host using full gdb than the limited ESP8266 client. However I have also considered adding Lua File System to the host side, and a host-only LFS library to allow you to run host-side scripts to add ROTables to the LFS image. A good example of use would be if you wanted to add a bunch of resources such as CSS and JavaScript files to your LFS instead of having to load SPIFFS files, in your runtime code you could just refer to LFS.resource.css01 to access to string. (resource is just the name of the ROTable that you declared).

A good extra feature or not?

[NicolSpies]

Very good feature, I use a number of JSON SPIFF configuration files that needs to be preloaded.

[HHHartmann]

@TerryE Terry I found the -e option a while ago and was surprised to see it working (thought it was some leftover of whatever)
But without being able to use any nodemcu modules it is of not so big use expect for testing larger bits of calculation code.
Shouldn't it be fairly simple to include the hardware independent modules like cjson ...?
Having some kind of emulated file module being redirected to a local directory would sure help a lot too.
The LFS.ressource part could be done by some LUA Glue code fairly easy as I understand it. So that wouldn't be on the top of the list for me.
Some HW related modules would also be interesting like tmr or the rtc modules. But that would go direction esp8266 emulator.

[TerryE]

@HHHartmann Gregor, to be honest, the main reason that I added this was to enable you to test the NodeMCU LuaVM changes on the host -- it's a lot easier doing this with a decent debugger where you can set multiple HW breakpoints etc.

All of the app/modules are totally incompatible with a host execution environment so there is little to be gained in trying to port any. Creating an ESP emulator would be a horrendous task. Sorry.

If you look in app/lua/luac_cross then you will see that I have ported os and io. I will add 'luafs` but nothing else, at the moment. I am very ambivalent about exposing this further since it will just cause confusion for anyone other than very advanced developers. I am tending to the idea that I should really crack on with Lua 5.3 and proper ESP32 / ESP8266 integration. I might even get @devsaurus Arnim interested in this one 😄

[joysfera]

@HHHartmann @TerryE no need for an ESP emulator - there's an (already closed?) issue in NodeMCU where someone was porting the environment to other small MCUs and even to a Linux host - and the progress was quite good. So the luac -e could fit there nicely.

[joysfera]

@TerryE oh, I have just noticed the sentence about Lua 5.3 and proper ESP32 support by integrating with ESP8266 - absolutely go for it, please!

[HHHartmann]

@joysfera I read the announcement about the dev start of the emulator but was not aware that he got that far. I might give it a try these days.
For the interested it is hosted at https://github.com/Spiritdude/nodemcu-linux.

[devsaurus]

I might even get @devsaurus Arnim interested in this one

Oh, I'd love to see the ESP32 branch benefiting from all these goodies 😃 Just let me know how I can support.

[dtran123]

I am having trouble getting LFS to work. I have created an image based on the example files https://github.com/nodemcu/nodemcu-firmware/tree/dev/lua_examples/lfs and using Terry's tool: https://blog.ellisons.org.uk/article/nodemcu/a-lua-cross-compile-web-service/
After executing node.flashreload() , the firmware keeps saying that it has not loaded the lfs image.

Would it be possible to create a complete example that is simpler than found at https://github.com/nodemcu/nodemcu-firmware/tree/dev/lua_examples/lfs.
Basically, just a "Hello world" print statement in a lua module . The files should also include a sample init.lua file. The above lfs example doesn't have a sample init.lua. A step by step procedure to get it to work would be nice...assume the user just started with the ESP8266 and knows nothing about it.

In short, a LFS for dummy would be great.

I am dying to try out LFS because I need to use mutual authentication with services such as AWS. Without LFS, we don't have enough RAM to handle the certificates/cyphers.

Anybody who has successfully used LFS care to help us leverage this great feature ?

[TerryE]

Oops, my bad. The LFS format changed in dev to add compression. I need to add the extra cross complier versions. I'll do that this evening. Sorry

[HHHartmann]

If you want to continue now either use a docker image to have a compressing luac.cross or use a firmware built off master branch.

[NicolSpies]

I have been following the LFS development since the beginning and decided to finally abandon the flashmod method developed by dpwhittaker and test the process NodeMCU documentation and the myriad of related discussions and suggestions. The process to read, understand, implement, test and convert a major application took me about 4 days

Accolades to @TerryE and @marcelstoer for all the effort for this framework and tool. I is now possible to do things previously impossible.

I have modified my development process to the following:

1. Install Docker on Windows 10
2. Clone NodeMCU dev repository using GitBash for Windows
3. Modify the user_modules.h, user_config.h and user_config.h files
4. Use Windows PowerShell to pull @marcelstoer docker build-image container
5. Use Windows PowerShell to build ESP binary image using docker
6. Use Windows PowerShell to build LFS image using docker build
7. Flashed ESP binary image to ESP and confirm operation
8. Copied LFS image to SPIFFS
9. Used the LFS examples to modify the init.lua

My development starts with an LFS image empty of application code with functionality incrementally developed using an external text editor and uploaded to SPIFFS for testing and debugging. When the specific functionality is completed it is moved across to the LFS image and the next required functionality development is started. This continues until the whole lua application exists in the LFS image.

[TerryE]

@dtran123, I've got some glitch in the decompression library on the ARM targets. I am too tired to hunt this down tonight, so I will hunt this down when I am fresh tomorrow. Sorry.

@NicolSpies Nicol, Thanks for the feedback. As I've said before, in my case using a 128Kb LFS region for my code and still having 40+ Kb RAM for Lua variables has transformed the way that I code. The flashmod coding styles are effective but complicate the structure and readability of the code. No need to bother with that now.

[dtran123]

thx Terry. No sweat. I will follow HHartmann's suggestion to use the master branch instead for the LFS.

[dtran123]

Good news. I managed to get LFS to work using the master branch. I successfully solved one of my problems which was to connect to my mosquitto broker using full client+CA certificates. It used to run out of memory but not anymore. Still troubleshooting my connection to AWS IoT which is still failing...

@TerryE , @marcelstoer : thank you thank you thank you...LFS is really a game changer as it opens up new possibilities on the ESP8266 (mutual authentication, etc.)

My question now is: I configured the LFS to 128K, why am I still seeing E:M out of memory when I increased the MBEDTLS_SSL_MAX_CONTENT_LEN from 5120 to 6000...this small increase seems to bust the RAM limit causing E:M errors.

What can I do to safely increase MBEDTLS_SSL_MAX_CONTENT_LEN to larger values based on additional virtual RAM ?

[TerryE]

@dtran123, LFS move codes out of RAM and into the LFS region, but you can still load code from SPIFFS into RAM as well as any RW data which of course remains in RAM. You still need to be careful over variable usage and make sure that you are dereferencing any variables once done with them so they can be GCed. But this isn't an LFS issue, so doesn't belong on this issue 😄

[TerryE]

Oops, my bad. The LFS format changed in dev to add compression. I need to add the extra cross complier versions. I'll do that this evening.

This took a bit of tracking down. To save RAM uzlib/uzlib_deflate.c expands a compressed encoding of the compression code tables in genCodeRecs() which contains the line:

    m +=!(*c & 0x80) ? *c & 0x7F : 2 << *c;

The C array is denormalised: if the high-bit is clear then the remaining 7 bits is an increment; if it is set then the the remaining 7 bits is a shift value, so it is the false term that should be masked; masking off a zero bit on the true term is redundant. 😱

So why wasn't this picked up in testing: this code generates a shl %cl,%edx on the 64bit Intel and this only looks at the bottom 6 bits of %cl anyway, so the error is silent. The ARM code generator generates a lsl r3, r2, r3 instruction which uses the full byte and shifts all the bits out of the register. Durrhhh. I've changed this to:

    m += (*c & 0x80) ? 2 << (*c & 0x1F) : *c;

and now the ARM luac.cross also works fine.

[dtran123]

Thanks Terry. Let us know when we can resume using the following service for all
combinations of dev/master & int/float builds:
https://blog.ellisons.org.uk/article/nodemcu/a-lua-cross-compile-web-service/

I am considering building my own lua cross compiler so I don't have to rely on above service.

thx !

[dtran123]

BTW, @TerryE , I remember you had quoted that you can have 43K+ of RAM when you start new lua projects on the ESP8266. I wonder how you could get it. I tried both master and dev branches, but only manage to get 41K RAM. I had reduced the modules selected to a minimum (file, net, node, mqtt, timer, uart, wifi, tls). I am curious because right now, I am blocked with secured connections to AWS IoT. I am generally short with E:M 250 to E:M 500 errors. It seems that 2K more of RAM to start with will solve my problem.

[sonaux]

Moving common string constants into LFS will free additional 2 KB of RAM.

[jmd13391]

@dtran123,

"I am considering building my own lua cross compiler so I don't have to rely on above service."

Why not use http://hub.docker.com/r/marcelstoer/nodemcu-build ?

[TerryE]

@dtran123, this service is really there as a bootstrap. Once you are serious about using LFS, it is better to use your own local services, such as Docker -- it's all documented 😄

[dtran123]

thanks for the reminder that docker method has an option to build the lfs image.

[dtran123]

FYI, I produced my LFS image successfully using docker. I was able to save a few more bytes of RAM by updating LFS_dummy_strings.lua. Still no luck though getting enough RAM for my connections to AWS IoT.

Question for the ESP developers, how much RAM does the eLua take during run time ? Would it be safe and relatively easy to pick and choose what goes into eLua ? Depending on the use case, could we save a few KB of RAM by dropping some irrelevant components to a specific application ?

[joysfera]

@dtran123, if it is indeed a (lack of) memory issue what you're having then I think your best bet would be NodeMCU @ ESP32.

[TerryE]

@dtran123, this issue list is really for specific issues relating to bugs / feature enhancements (and specfically LFS enhancements here) , etc. rather than general "How to" dialogue. NodeMCU has little in common with eLua now, it has stripped RAM usage way above what eLua on its own offers, and with LFS there is about 48Kb RAM made available to the Lua VM of which about 44Kb is passed to the initialised application.

Read the documentation FAQ for info on how to limit RAM. If you can't properly GC unused variables then IMO the best options are to descope the ESP8266 application or as Petr suggests to move to the ESP32 platform.

I personally use a Node RED / mosquitto hub running on an RPi as my concentrator / control hub and integrate everything on that. I just use raw JSON over TCP sockets rather than MQTT for ESP8266 to RPi comms. I don't bother with HTTPS, because of its high RAM and Flash footprint, but I do sign all ESP8266 <-> RPi packets based on a shared secret, so these are trusted but can potentially be eavesdropped -- Trusted but not necessarily secret works fine for me.

[nwf]

Given that several people, if my memory is right, asked about SSL woes specifically as pertains to AWS IoT, it may be worth adding a bold disclaimer to the docs that we do not support it and, perhaps more generally, that we, as nodemcu developers, do not support integration with any commercial service. People who want us to try to solve problems with commercial platform integration are welcome to see #1010 and/or seek out a support contract, but, given the resource constraints of the 8266 (even neglecting the overhead of Lua), people should understand that certain things may be impossible, requiring workarounds, such as bridging to a beefier computer, as suggested by @TerryE.

[dtran123]

Thanks @nwf & @TerryE for your comments and help to make me a happy user of LFS. I decided to not pursue ESP8266/Lua integration with AWS and use Mongoose OS instead (which support secured ESP8266/AWS IoT) for such application. I am very thankful for LFS to enable secured connections (client+CA certs) to my MQTT broker. That is a big win and beggers are not choosers!

[TerryE]

@nwf Nathaniel, perhaps I just need to be a more proactive in documenting some application notes on my blog so that are searchable. At a later stage we can always précis the more popular such contributions from the active advanced developers into the online documentation.

[raffaelediana]

Hello.
I am having similar trouble than dtrasn123 getting LFS to work.

I have created an image based on the example files https://nodemcu.readthedocs.io/en/latest/getting-started/#compile-lua-into-lfs-image and using Terry's tool:

https://blog.ellisons.org.uk/article/nodemcu/a-lua-cross-compile-web-service/

I uploaded the .img file created (FLOAT MASTER) using UPLOAD of ESplorer tool.

After executing node.flashreload() , the firmware keeps saying that it has not loaded the lfs image (it says "No LFS image loaded").

Thanks in advance for the support.

[HHHartmann]

Hi @raffaelediana if I an not completely wrong the former DEV is on MASTER now. So try using a DEV LFS image.
@TerryE ?

Does your esp reboot after node.flashreload()?
If not the reload was not successful and the return values might help you to see what's wrong.

[raffaelediana]

HI @joysfera and @HHHartmann
Thanks for you answers.
I Simply created the img file using the web service in FLOAT mode because my firmware is a FLOAT LFS Enabled firmware created using cloud build.
After i uploaded it to an NODEMCU using ESplorer and i launched node.flashreload() and esp rebooted (for security i also restarted the device again).

It tells me "No LFS image loaded". So i suppose it didn't work.

[raffaelediana]

I have also tested DEV and not MASTER...same result.

[marcelstoer]

Guys, this issue was

to track LFS-related feedback and functional gaps

(we should IMO have closed it a while a ago I guess)

Please see https://nodemcu.readthedocs.io/en/latest/support/ for support options.

[TerryE]

@HHHartmann Gregor, let me check the builds and error logs. My bad, I haven't been keeping an eye on this. But this is more for my blog than here.

IMO, by far the easiest way for WinX users to use luac.cross is to install core Cygwin (if you haven't already got WSL loaded). Cygwin only takes up about ½Gb HDD compared to the 5Gb that WSL occupies, Download the latest zip that you want to use, and unzip it. Now edit the app/include/user_config.h to enable LFS and cd app/lua/luac_cross; make. The image is created in the root directory. Copy it somewhere safe and trash the zip and the source directory.

I haven't really used WinX myself for over a decade, though I do have an 8 year old laptop that I can dust off, power up and dual boot into and which I keep for when I do need to do something in Win7 like testing this.

[jmd13391]

I work in a Windows environment and find that @marcelstoer 's docker nodemcu-build and ESPlorer make easy work of building firmware images and uploading Lua applications to LFS.

[TerryE]

@jmd13391 Joe, you are right, the Docker build as maintained by Marcel and Gregor is another simple to use approach.

[raffaelediana]

Hi @TerryE i don't know if from your web service logs you can desume why the img i created is not seen in my nodemcu....
Is there any way to undestand why these img files are not recognized ?

Thanks,
Raffaele.

[TerryE]

@raffaelediana, a recent commit broke the luac_cross make, so the local versions on my blog server weren't getting updated and became out of sync with the firmware builds. I've put in a workaround to fix this. Sorry.

[raffaelediana]

No problem. Thanks.

[TerryE]

I think that this general buttlist is complete, so closing this generic issue.

[jyh29]

Why not setup a web service under pay for users to download simple beginning LFS examples, especially of _init.lua & dummy_strings.lua ?

[HHHartmann]

@jyh29 could you please stop cross posting this.