-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LFS follow-up issues and missing features. #2413
Comments
Support for luac.cross and spiffsimgSee #2315. We now have WSL and Cygwin methods which cover all WinX versions. Of course the other option is to buy yourself a $15 RPi Zero and load DietPi and make luac.cross and use a small ssh script (either WSL native SSH or OpenSSH) from your PC to offload the compiles to the RPi0. Cloud builder generating LFS firmware
Additional LFS-enabled Lua ModulesIn addition to Compression SupportIn progress. See #2409. Updates to FAQs and LFS whitepaperPending feedback from users. |
I recently got my first LFS installation running. Here are some obstacles and suggestions I came across: I used a docker build in standard configuration which built an integer and a float image. I then found that aside the local/fs directory a local/lua directory is recognized by the tools make.
so I put some lua files in there and built an LFS image (make in tools directory, image stored in local/fs).
I then transferred the generated image to the ESP and called ''node.flashreload()''
|
@HHHartmann Gregor, thanks and in reply to your comments,
|
@TerryE a very minor thing, but running |
@nwf, you can't do a normal |
@HHHartmann @joysfera et al, are you aware of the A good extra feature or not? |
Very good feature, I use a number of JSON SPIFF configuration files that needs to be preloaded. |
@TerryE Terry I found the -e option a while ago and was surprised to see it working (thought it was some leftover of whatever) |
@HHHartmann Gregor, to be honest, the main reason that I added this was to enable you to test the NodeMCU LuaVM changes on the host -- it's a lot easier doing this with a decent debugger where you can set multiple HW breakpoints etc. All of the If you look in |
@HHHartmann @TerryE no need for an ESP emulator - there's an (already closed?) issue in NodeMCU where someone was porting the environment to other small MCUs and even to a Linux host - and the progress was quite good. So the |
@TerryE oh, I have just noticed the sentence about Lua 5.3 and proper ESP32 support by integrating with ESP8266 - absolutely go for it, please! |
@joysfera I read the announcement about the dev start of the emulator but was not aware that he got that far. I might give it a try these days. |
Oh, I'd love to see the ESP32 branch benefiting from all these goodies 😃 Just let me know how I can support. |
I am having trouble getting LFS to work. I have created an image based on the example files https://github.com/nodemcu/nodemcu-firmware/tree/dev/lua_examples/lfs and using Terry's tool: https://blog.ellisons.org.uk/article/nodemcu/a-lua-cross-compile-web-service/ Would it be possible to create a complete example that is simpler than found at https://github.com/nodemcu/nodemcu-firmware/tree/dev/lua_examples/lfs. In short, a LFS for dummy would be great. I am dying to try out LFS because I need to use mutual authentication with services such as AWS. Without LFS, we don't have enough RAM to handle the certificates/cyphers. Anybody who has successfully used LFS care to help us leverage this great feature ? |
Oops, my bad. The LFS format changed in |
If you want to continue now either use a docker image to have a compressing luac.cross or use a firmware built off master branch. |
I have been following the LFS development since the beginning and decided to finally abandon the flashmod method developed by dpwhittaker and test the process NodeMCU documentation and the myriad of related discussions and suggestions. The process to read, understand, implement, test and convert a major application took me about 4 days Accolades to @TerryE and @marcelstoer for all the effort for this framework and tool. I is now possible to do things previously impossible. I have modified my development process to the following:
My development starts with an LFS image empty of application code with functionality incrementally developed using an external text editor and uploaded to SPIFFS for testing and debugging. When the specific functionality is completed it is moved across to the LFS image and the next required functionality development is started. This continues until the whole lua application exists in the LFS image. |
@dtran123, I've got some glitch in the decompression library on the ARM targets. I am too tired to hunt this down tonight, so I will hunt this down when I am fresh tomorrow. Sorry. @NicolSpies Nicol, Thanks for the feedback. As I've said before, in my case using a 128Kb LFS region for my code and still having 40+ Kb RAM for Lua variables has transformed the way that I code. The |
thx Terry. No sweat. I will follow HHartmann's suggestion to use the master branch instead for the LFS. |
Good news. I managed to get LFS to work using the master branch. I successfully solved one of my problems which was to connect to my mosquitto broker using full client+CA certificates. It used to run out of memory but not anymore. Still troubleshooting my connection to AWS IoT which is still failing... @TerryE , @marcelstoer : thank you thank you thank you...LFS is really a game changer as it opens up new possibilities on the ESP8266 (mutual authentication, etc.) My question now is: I configured the LFS to 128K, why am I still seeing E:M out of memory when I increased the MBEDTLS_SSL_MAX_CONTENT_LEN from 5120 to 6000...this small increase seems to bust the RAM limit causing E:M errors. What can I do to safely increase MBEDTLS_SSL_MAX_CONTENT_LEN to larger values based on additional virtual RAM ? |
@dtran123, LFS move codes out of RAM and into the LFS region, but you can still load code from SPIFFS into RAM as well as any RW data which of course remains in RAM. You still need to be careful over variable usage and make sure that you are dereferencing any variables once done with them so they can be GCed. But this isn't an LFS issue, so doesn't belong on this issue 😄 |
This took a bit of tracking down. To save RAM m +=!(*c & 0x80) ? *c & 0x7F : 2 << *c; The So why wasn't this picked up in testing: this code generates a m += (*c & 0x80) ? 2 << (*c & 0x1F) : *c; and now the ARM luac.cross also works fine. |
Thanks Terry. Let us know when we can resume using the following service for all I am considering building my own lua cross compiler so I don't have to rely on above service. thx ! |
BTW, @TerryE , I remember you had quoted that you can have 43K+ of RAM when you start new lua projects on the ESP8266. I wonder how you could get it. I tried both master and dev branches, but only manage to get 41K RAM. I had reduced the modules selected to a minimum (file, net, node, mqtt, timer, uart, wifi, tls). I am curious because right now, I am blocked with secured connections to AWS IoT. I am generally short with E:M 250 to E:M 500 errors. It seems that 2K more of RAM to start with will solve my problem. |
Moving common string constants into LFS will free additional 2 KB of RAM. |
"I am considering building my own lua cross compiler so I don't have to rely on above service." Why not use http://hub.docker.com/r/marcelstoer/nodemcu-build ? |
@dtran123, this service is really there as a bootstrap. Once you are serious about using LFS, it is better to use your own local services, such as Docker -- it's all documented 😄 |
thanks for the reminder that docker method has an option to build the lfs image. |
FYI, I produced my LFS image successfully using docker. I was able to save a few more bytes of RAM by updating LFS_dummy_strings.lua. Still no luck though getting enough RAM for my connections to AWS IoT. Question for the ESP developers, how much RAM does the eLua take during run time ? Would it be safe and relatively easy to pick and choose what goes into eLua ? Depending on the use case, could we save a few KB of RAM by dropping some irrelevant components to a specific application ? |
@dtran123, if it is indeed a (lack of) memory issue what you're having then I think your best bet would be NodeMCU @ ESP32. |
@dtran123, this issue list is really for specific issues relating to bugs / feature enhancements (and specfically LFS enhancements here) , etc. rather than general "How to" dialogue. NodeMCU has little in common with eLua now, it has stripped RAM usage way above what eLua on its own offers, and with LFS there is about 48Kb RAM made available to the Lua VM of which about 44Kb is passed to the initialised application. Read the documentation FAQ for info on how to limit RAM. If you can't properly GC unused variables then IMO the best options are to descope the ESP8266 application or as Petr suggests to move to the ESP32 platform. I personally use a Node RED / mosquitto hub running on an RPi as my concentrator / control hub and integrate everything on that. I just use raw JSON over TCP sockets rather than MQTT for ESP8266 to RPi comms. I don't bother with HTTPS, because of its high RAM and Flash footprint, but I do sign all ESP8266 <-> RPi packets based on a shared secret, so these are trusted but can potentially be eavesdropped -- Trusted but not necessarily secret works fine for me. |
Given that several people, if my memory is right, asked about SSL woes specifically as pertains to AWS IoT, it may be worth adding a bold disclaimer to the docs that we do not support it and, perhaps more generally, that we, as nodemcu developers, do not support integration with any commercial service. People who want us to try to solve problems with commercial platform integration are welcome to see #1010 and/or seek out a support contract, but, given the resource constraints of the 8266 (even neglecting the overhead of Lua), people should understand that certain things may be impossible, requiring workarounds, such as bridging to a beefier computer, as suggested by @TerryE. |
Thanks @nwf & @TerryE for your comments and help to make me a happy user of LFS. I decided to not pursue ESP8266/Lua integration with AWS and use Mongoose OS instead (which support secured ESP8266/AWS IoT) for such application. I am very thankful for LFS to enable secured connections (client+CA certs) to my MQTT broker. That is a big win and beggers are not choosers! |
@nwf Nathaniel, perhaps I just need to be a more proactive in documenting some application notes on my blog so that are searchable. At a later stage we can always précis the more popular such contributions from the active advanced developers into the online documentation. |
Hello. I have created an image based on the example files https://nodemcu.readthedocs.io/en/latest/getting-started/#compile-lua-into-lfs-image and using Terry's tool: https://blog.ellisons.org.uk/article/nodemcu/a-lua-cross-compile-web-service/ I uploaded the .img file created (FLOAT MASTER) using UPLOAD of ESplorer tool. After executing node.flashreload() , the firmware keeps saying that it has not loaded the lfs image (it says "No LFS image loaded"). Thanks in advance for the support. |
Hi @raffaelediana if I an not completely wrong the former DEV is on MASTER now. So try using a DEV LFS image. Does your esp reboot after node.flashreload()? |
HI @joysfera and @HHHartmann It tells me "No LFS image loaded". So i suppose it didn't work. |
I have also tested DEV and not MASTER...same result. |
Guys, this issue was
(we should IMO have closed it a while a ago I guess) Please see https://nodemcu.readthedocs.io/en/latest/support/ for support options. |
@HHHartmann Gregor, let me check the builds and error logs. My bad, I haven't been keeping an eye on this. But this is more for my blog than here. IMO, by far the easiest way for WinX users to use luac.cross is to install core Cygwin (if you haven't already got WSL loaded). Cygwin only takes up about ½Gb HDD compared to the 5Gb that WSL occupies, Download the latest zip that you want to use, and unzip it. Now edit the I haven't really used WinX myself for over a decade, though I do have an 8 year old laptop that I can dust off, power up and dual boot into and which I keep for when I do need to do something in Win7 like testing this. |
I work in a Windows environment and find that @marcelstoer 's docker nodemcu-build and ESPlorer make easy work of building firmware images and uploading Lua applications to LFS. |
@jmd13391 Joe, you are right, the Docker build as maintained by Marcel and Gregor is another simple to use approach. |
Hi @TerryE i don't know if from your web service logs you can desume why the img i created is not seen in my nodemcu.... Thanks, |
@raffaelediana, a recent commit broke the luac_cross make, so the local versions on my blog server weren't getting updated and became out of sync with the firmware builds. I've put in a workaround to fix this. Sorry. |
No problem. Thanks. |
I think that this general buttlist is complete, so closing this generic issue. |
Why not setup a web service under pay for users to download simple beginning LFS examples, especially of _init.lua & dummy_strings.lua ? |
@jyh29 could you please stop cross posting this. |
Rather than following the Bugrep or Feature request templates, this issue is more of a post LFS merge follow up to #2292 (which really discussed trade-offs up to our LFS implementation and at 48 comments was large and becoming unwieldy to load). I will use this issue to track LFS-related feedback and functional gaps. We can use this for a catch-all for the next few weeks until our next LFS patch update, and after this we can use the normal Bugrep / FR templates to track any follow as individual issues.
The text was updated successfully, but these errors were encountered: