Limited WinX support for luac.cross and spiffsimg

[TerryE]

Missing feature

This is really a corollary to #2292. We need a process for supplying native WinX executables for luac.cross and spiffsimg.

Justification

With the availability of Marcel's Cloud build service, very few ESP Lua developers need to have a NodeMCU firmware build environment. Also whilst both luac.cross and spiffsimgare considered essential for heavyweight Lua developers, most developers seem happy to do without them.

LFS changes all this because you must use an LFS-enabled version of luac.cross to generate your images. You also need spiffsimg if you prepare your own FS images for downloading.

Luckily the Makefiles for both app/lua/lua_cross and tools don't callback to parent makes, and so these can be easily built standalone from the relevant directory. So what we need to do is to document how a WinX user can build these, preferably with free software. Whatever this process is, these makefiles should contain the necessary conditional statement to work in both environments. Perhaps the easiest approach might be to use MinGW or Cygwin, both of which use the GNU toolchain.

Workarounds

I can't do this work because I don't have any Windows platforms (purely Linux (and Android) so I have no need of workarounds. Perhaps another alternative would be for one of the core developers to do a private build and we follow the PHP practice of making a zip available with the current working release of the WinX binaries for these components.

[cwrseck]

I think there's a significant need for luac.cross, which isn't well enough known. I've been working on a fairly small application for temperature monitoring, and despite all my tinkering I rapidly ran out of space to compile it (to bytecode) on the ESP8266. luac.cross, now I've gotten around to using it, has allowed code size to increase considerably and made debugging a lot easier.

A windows version of luac.cross is definitely needed.

Will

[TerryE]

now I've gotten around to using it, has allowed code size to increase considerably and made debugging a lot easier.

@cwrseck, this is partly a result of the history of the use of the Lua stack on NodeMCU. The early versions (before 1.1 IIRC) didn't have a working luac.cross and used the node.compile() function to generate all compiled lua files. I've been working steadily to improved the core over the last few years. I got the traceback working properly, then did he LCD patch with allowed you to strip out the debug info that traceback didn't need, plus used compression to reduced the line info by ~ 10×, so that retaining line info became a sensible option. I also added luac.cross.

Unfortunately the node.compile() function doesn't use the stripdebug options, so using this loses all traceback. What you can do if you don't have luac.cross is to use the string.dump(loadfile(name)) approach is a separate "compile only" boot.

But as far as I am concerned life is too short. My provisioning system handles all of this and does the compiles on the host. I just tweak any files on that modules shadow directory on the host and command the app to reprovision itself. 5 seconds later I am running the new version on the ESP module.

As I said, I only run Linix, but the big problem here that most ESP developers still stick to WinX. I think that they are crazy, but I respect their right to choose crazy. 😄

[karrots]

Windows 10 has this solved ;)

https://docs.microsoft.com/en-us/windows/wsl/install-win10

[TerryE]

@karrots Jooathan, I think that you would be doing your fellow WinX developers a great service if you wrote a simple short How To that took a lua developer step by step through the whole process from installing an Ubuntu subsystem, fetching the repo, and either

• installing build-essential and git, building standalone Xtensa toochain using pfalcon's toochain, pulling or fetching the firmware, configuring the app/include/user*.h files and them making the firmware (including luac.cross and spiffimg, or
• installing build-essential and git, getting and upacking a latest release then making the app/lua_cross and tools directories. Or
• Doing a Windows docker install as per Marcel's NodeMCU Build on Docker

Yes, I could sketch this out, but whoever does this needs to do a full dress rehearsal on a WinX machine to validate the script before issuing this as a PR to link into our doc tree. So I can't do this.

There may be other simpler options that I haven't though of, but given that the only Win$ that I touch nowadays is cleaning and upgrading the PCs of many of all my IT-illiterate friends and family. There are other issues like: how much space does this take up on the system drive, how do you run lua.cross from the Windows command shell, etc.

If we do use docker then AFAIK, the docker container is set up do a fresh build on each execution, and essentially only the bin directory is preserved. This means that the process takes a few mins on a reasonably powerful PC. This is essential for its primary Travis use, but this is a total pain here. It should be able to set a switch so that the entire NodeMCU tree is cashed persistently in the container, so that makes are truly incremental and only take a couple of seconds.

So I suggest that this should belong to you and Marcel .

[cwrseck]

Reference luac.cross:

Unfortunately loadfile() throws the pervasive E:M error (I wish I knew what
that meant) on this file; what's more, it doesn't throw it reliably, so it's
never quite clear whether the code will run or not. I'd guess that to be an
effect of the GC, except that a reset doesn't improve things. Hence my
enthusiasm for luac.cross; so far, code that compiles on Linux will run on
the ESP8266.

dofile() and node.compile() are certainly quick and easy ways to compile small
applications, but before your improvements building large chunks of ESP8266
code could be pretty frustrating, and it might be worthwhile (once the Windows
binaries are available) to recommend luac.cross in the FAQ, with a note that
if it is locally built, it must be with LUA_NUMBER_INTEGRAL undefined.

Will

[TerryE]

The E:M error is thrown by the compiler when it runs out of RAM during the compilation. There is a fairly low compilation unit size, but nothing (other (than being a PITA) to stop you breaking your app into lots of small compilation units. But using luac.cross certainly eases this problem.

With the LFS patch, once deployed, luac.crossis just part of the standard make and doesn't matter if LUA_NUMBER_INTEGRAL is defined or not -- so long as the same flavour is used for both the firmware and the cross compiler.

[karrots]

I don't have a windows machine at home either. I might be able to cook up some instructions using the work machine.

Re Docker for Windows, unfortunately, docker isn't an emulation layer. So you can't move a docker image made for Linux and run it on Windows. That said the Windows Linux subsystem based on the link posted earlier looks like more or less a full install of Ubuntu or another distro of your choice. I haven't tried it yet.

[TerryE]

I don't have a windows machine at home either.

Bugger!. Another real PC user. 😄 When I retired early, I returned all of my work PC kit to HP. The next couple of laptops I bought were dual boot, but the only time I ever seemed to boot into Windows was to do the occasional security upgrade (which always took ages) and so it became just easier to use single boot Ubuntu with headless VMs for playpens.

We could really use a true Windows-only developer to do this learning curve.

[joysfera]

If there are no Windows developers left then why bother with supporting dev tools on Windows? ;-)

[TerryE]

It's like iPad/iPhone vs Android: both platforms have their strength and weaknesses, and in general people end up choosing one or the other depending on what fits them. This isn't a matter of right and wrong, but rather one of personal preference. The same argument applies for OSX vs Linux vs WinX: what an individual developer uses is up to her or him. The underlying issue here is that we have two communities of developers who use or are involved with NodMCU:

1. The application developers who are building IoT projects and products using the NodeMCU platform. I suspect that a large majority of theses use Marcel's Cloud Service to generate their firmware, and a significant majority of the ~200K builds are a straightforward master build using a very limited set of core modules. And most of these are Windows users.
2. The very small number of these developers that do C development and are active Github contributors to the project. IMO, there are no more than 20 such developers at any one time, and these people do their own builds and use Linux-based development platforms. (As I said earlier, I don't even own a Windows platform.)

But if developers in this first category want to use LFS then they either need to have windows-executable NodeMCU luac.cross and spriffsimg binaries, or access to a cloud service where they can upload a zip file of their application and receive a SPIFFS image in returm.

[joysfera]

OK. As @karrots wrote earlier I believe the Ubuntu on Win10 (WSL) is the best way.
The first steps are documented here:
https://tutorials.ubuntu.com/tutorial/tutorial-ubuntu-on-windows

This is documentation from the other vendor:
https://docs.microsoft.com/en-us/windows/wsl/install-win10

Unfortunately, Docker does not run on WSL directly but luckily there are workarounds:
https://stackoverflow.com/questions/48008675/run-docker-on-wsl-windows-subsystem-for-linux-ubuntu
https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly
https://medium.com/@sebagomez/installing-the-docker-client-on-ubuntus-windows-subsystem-for-linux-612b392a44c4
or an official and more complicated way with a relay:
https://blogs.msdn.microsoft.com/commandline/2017/12/08/cross-post-wsl-interoperability-with-docker/

The last step would be Marcel's https://hub.docker.com/r/marcelstoer/nodemcu-build/

I would put the documentation together from these tested and proven bits of information instead of writing something from scratch. The closer it gets to the workflow of a Linux user the better it will work in the long run.

A cloud service building SPIFFS images would be far better, though. I bet even Linux/Mac users would be using it (just like they currently use the cloud service for building firmware images).

[joysfera]

I got sidetracked completely, sorry. WSL way will not build native Windows binaries.
Either back to Cygwin/MingW or resort to running Linux binaries on Windows (either using WSL or emulator like Virtual box).
This github issue asks for native binaries so the WSL/virtualization is out of question, I'm afraid.

[TerryE]

Petr, you like me are probably very comfortable with virtualization. (In fact I used to be contributor to Virtualbox and was a moderator on their user forums back when Sun owned them.) but again needing to use this sort of technology is a step too far for many IoT developers. I think that the best compromise wouldbe to get the app/lua/lua_cross and the tools build processes Mingw compatible and to make these binaries available to Windows users.

[joysfera]

OK, until somebody cross compiles the cross compiler: I have just created a Debian package of luac.cross and spiffsimg binaries. Installing it on Windows 10 with WSL is a piece of cake: simply go to Windows Store, buy Ubuntu (for free), it installs automagically within few seconds and since then you can run your trusted Linux binaries in MS Windows. Then you're just two steps away from the Lua cross compiler for NodeMCU in MS-Windows (run these commands in your new Ubuntu shell, of course):

wget https://pstehlik.cz/data/lua-nodemcu.deb
sudo dpkg -i lua-nodemcu.deb

That's all. And if you didn't know that, your Windows C:\ drive is fully accessible in /mnt/c/ path so if you wrote your Hello world Lua program in Notepad.exe and stored it to C:\Temp\hello.lua you can now call (still in the Ubuntu window)

luac.cross /mnt/c/Temp/test.lua

to get your bytecode for NodeMCU. You'll find it in C:\Temp\luac.out file.

In the Debian package I also included manual pages, documentation of spiffsimg and last but not least, step-by-step instructions how to rebuild the binaries (somebody familiar with docker should review it eventually).

Enjoy!

[TerryE]

@joysfera Petr, in fact there are three separate build variants of luac.cross Integer, FP (16byte Tvalues), FP (12byte Tvalues); though I doubt anyone will use the 16byte TValue version soon as it just wastes 25% of your RAM for no gains.

An issue with WSL is that it is only supported on the later Win10 versions, so anyone on Win 8.x or Win 7 can't use it.

[joysfera]

@TerryE Terry, I could update my Debian package to include the FP 12byte variant - if somebody needs it just let me know.

As for WSL: Windows 7 will reach the end in January 2020, Windows 8.x in January 2023. If somebody builds the cross compiler natively sooner then great. If not then the WSL might be an easy way how to use the Lua cross compiler under Windows after 2023 :-)

[TerryE]

@joysfera, Petr to be honest, I never use integer builds, simply because when I do need decimal maths, it's just a PITA in the integer build variants. Also the runtime delta is pretty small in practice since the FP routines handle the denormalised integer values using integer computation, and the extra runtime is lost compared to the overheads of GC and ROtable lookup unless you code to minimise these.

IMO, the main advantage of the Int builds was really the 8 vs 16 byte TValue overhead. With LFS allowing you to move most or all of your code into flash, and 12 byte TValues developers would need to start building far larger apps before starting to hit RAM limits.

[NicolSpies]

@joysfera, I have followed your Debian package process described earlier and it is as easy as you said with the following small correction to locate the output bytecode:

The luac.cross /mnt/c/Temp/test.lua command reads the test.lua file in the C:/Temp directory but creates the bytecode in the directory where luac.cross is located.

To create the bytecode file test.out in the Temp directory use the following:
luac.cross -o /mnt/c/Temp/test.out /mnt/c/Temp/test.lua

[TerryE]

or (since this is a bash shell)

luac.cross -o /mnt/c/Temp/test.{lc,lua}

since you should adopt the lc extension convention for bytecode files and img for LFS images:

luac.cross -f -o /mnt/c/Temp/lfs.img /mnt/c/Temp/lfs/*.lua

Also -l emits an assembler listing to stdout in both cases. Grepping this for GLOBAL is a good way to pick up missed local declarations.

[TerryE]

I dug out an old laptop which still has a Win 7 HomePro boot partition on it and booted into Win7. I then installed the CYGWIN core + gcc-core + gnu make, and downloaded a zip of dev, tweaked the user_config.h to enable LFS, then did a cd app/lua/luac_cross; make. Note that no support for the xtensa tooolchain is needed as the luac_cross make uses standard gcc.

I had to add a couple of small #ifdefs to handle the fact that the CYGWIN ld builds images slightly differently. I will add these to the next LFS-related PR, so that the make works out of the box.

But this is a 10-min method of creating a version of luac.cross that can run on pre-WSL Windows versions (albeit in a Cygwin command window) without getting into VMs or Docker.

[joysfera]

While trying to compare MSYS to Cygwin I happened to find Interix = Microsoft Services For Unix (or so) that is supposedly still updated and even included in Windows:
http://wiki.tcl.tk/11329
Just FYI. No personal experience.

[TerryE]

Petr,

Thanks for the help, but it's a hill that I don't want to climb.

I did look at Cygwin vs. MinGW. In my pre-Linux days over 15 years ago (when my main *nix use was Solaris) I used Cygwin for all of my hobby projects and it was 100% solid on Win XP. I also used it briefly on Win 7 without any problems. More to the point is that it presents a good POSIX environment so I knew that there would be very little to do to get luac.cross and spiffsimg working. MinGW has some POSIX-like support, but its RTS is really a thin veneer on the native WinX RTS so getting this working would be far more involved. For your average WinX developer, installing Cygwin is far less of a hurdle than getting into VMware, VirtualBox or Docker, IMO, and the default installation is ~400Mb in a single folder hierarchy.

I had hoped that some other contributor had the bandwidth to look into this, but most of the other committers have other work priorities at the moment, so if anything needs doing then it seems to be down to me -- especially when it comes to the Lua internals. The Cygwin (or WSL for Win10Pro users) meets the community users need for in WinX cross compilation without diverting my priorities.

[joysfera]

I know the Cygwin vs MinGW, that's why I wanted to check out the MSYS that is supposed to be better than MinGW, to see if it isn't a bit more POSIX than MinGW (probably not). Anyway, my previous message was about Interix because that should also be POSIX (from what I've read in 5 minutes) and even part of Windows (so no need to install Cygwin?)

I'm not telling you to give it a try, you're lucky that the Cygwin gave you a working binary in a moment without much work, but perhaps someone else would be interested in exploring the Interix (if they didn't know it, like me).

FYI, I used to be using Cygwin many years ago as well but later my OSS projects had to include Win specific code so now MSYS can build the binaries just fine.

[TerryE]

I have now pushed those small changes (PR ##2419). The steps are:

• Download the Cygwin setup file from the Cygwin site and run:
• Choose a drive where you have at least 1Gb free, and install the default + GCC core + GNU make + unzip. (Current user install).
• Use your browser to download the latest dev source zip.
• Open Cygwin termial Add a symlink to your Downloads directory and make

ln -s /cygdrive/c/Users/SomeUser/Downloads
unzip Downloads/nodemcu-firmware-dev.zip
# Now edit nodemcu-firmware-dev/app/includes/user_config.sys to enable LFS
# using vi or your prefered Windows editor
cd nodemcu-firmware-dev/app/lua/luac_cross; make
cd ../../../tools;make

You can now move nodemcu-firmware-dev/luac.cross and spiffsimg into your home directory.
You can optionally rm -R nodemcu-firmware-dev, and the Cygwin subdirectory in your downloads folder. The install is about 400Mb, which is a bit bloaty and could be trimmed a lot, but this takes less than 10mins and is still a lot less HDD space than a docker or VM install.

These utilities can be run from the Cygwin command line and there are various SO articles to explain how to wrap up their execution in batch files.

PS. I've just watched a review of WSL and it is a 4-5 Gb install, so maybe 400 Mb isn't so bad after all. 😆