cx_freeze 5 support on MS Windows

[totaam]

Follow up from #678, needed for #1527.

The version of cx_Freeze currently shipped by msys2 is 4.3.3, and it works well enough for our needs.
Version 5 breaks all sort of things, and contains some unfixed bugs... but we need it to be able to create a service (#1527), and msys2 is bound to update eventually - we'd better be ready for that.

Issues:

• we have to install from source...
• installation from source fails because Missed cxfreeze-postinstall script in source distribution - what a mess, download it by hand:

wget https://bitbucket.org/anthony_tuininga/cx_freeze/raw/e1ba6ec6eaad090a76b7095eeb12d15f803a3bde/cxfreeze-postinstall

• sqlite3.dll not shipped looks in the wrong place, ugly workaround:

mkdir /c/msys32/mingw32/DLLs
cp /c/msys32/usr/lib/python2.7/lib-dynload/_sqlite3.dll \
    /c/msys32/mingw32/DLLs/sqlite3.dll

• Fall back on sys.real_prefix for Python 2.7, workaround by defining sys.base_prefix
• to be able to create windows services, cx_freeze must be built from source with cx_Logging installed in a very specific location (cx_Logging must be side by side with the cx_Freeze-5.0.XX source directory we build from..), Windows service executable missing in distribution - suggests the dependency should be dropped as the package is unmaintained...
• some options have been removed - completely undocumented too, no migration information at all

Scary how many of the tickets on the cx_freeze bug tracker are old and unanswered.

The resulting installation is still completely broken... so maybe we need some of the ignored bugs / pull requests:

• Added a few MSYS/MINGW fixes and changed a bit of the PyQt4/PyQt5 hooks
• Finder can now deal mixed-slashes paths
• Application is blocked by import calls within functions: we need to be able to use comtypes..
• cryptography module - we do need that too
• Support for Win32 service dependencies - we probably don't need that

cx_freeze can generate an MSI installer, but there is no documentation on how that works:
Documentation of how to create desktop or program menu shortcut in Windows

What a mess.

[totaam]

2017-05-23 15:33:39: antoine commented

---

Looks like the actual bug tracker to use is not sourceforge or bitbucket, but this one now: [https://github.com/anthony-tuininga/cx_Freeze/issues].

New issues:

• Missing modules in numpy and other packages - we need a working numpy
• No module named startup: the libraries end up in the wrong location (lib and lib/python2.7.zip), looks like this may be solved by Allow storing all Python packages in a subdirectory on Windows - problem is that trunk (cx_freeze 6) no longer supports python2, PITA. The v5.x branch doesn't help us here either.
• Python 3.4 + win32GUI causes sys.argv to be empty - this would break the launcher file association

Shame we can't use another project, pyinstaller requires pywin32.. and py2exe doesn't work with our mingw setup.

[totaam]

2017-05-23 16:02:40: antoine uploaded file cx-freeze-lib.patch (0.5 KiB)

fix cx_freeze to load libraries from the directory it created

[totaam]

2017-05-23 16:28:54: antoine commented

---

The patch attached ([/attachment/ticket/1528/cx-freeze-lib.patch]) is based on the workaround suggested in No module named startup: I can confirm that copying/moving the files from the "lib" directory into same directory as the exe file does work.
Extra hacks in r15933 for #1527.

With the cx_freeze 5 workarounds from r15932, we can now generate an installation directory that mostly works.
Still TODO:

• cryptography: [https://github.com/Cryptography module marcelotduarte/cx_Freeze#138]
• numpy: [https://github.com/Missing modules in numpy and other packages marcelotduarte/cx_Freeze#232]
• sound subprocess doesn't work (closed file descriptors?), this may also affect printing and other tools we exec

[totaam]

2017-05-25 07:10:29: antoine commented

---

This is too much work, too much breakage - stub service removed in r15960.
Will use a shim for writing the service instead, see #1527#comment:4.

We can re-visit when we really have no other choice, and hopefully the cx_freeze 5.x branch will have matured a bit by then.

[totaam]

2017-06-21 10:29:12: antoine commented

---

MSYS2 has started pushing version 5.. PITA.

So until this is all fixed, we need to pin the package with:

echo "IgnorePkg   = mingw-w64-i686-python3-cx_Freeze mingw-w64-i686-python2-cx_Freeze" >> /etc/pacman.conf

For new installations, it is a bit more tricky as downgrades are not possible with pacman, so r16111 switches to using setuptools to install version 4 - just have to hope that this results in a working installation and that the PKGBUILD for version 4 won't need to be revived from version control. (untested - I dare not mess with this stuff since my system works!)

[totaam]

2017-07-03 14:04:46: antoine commented

---

I've copied an archive of the working cx freeze versions here:
[http://xpra.org/pkg/]

And as of r16175, that's the package we install.

[totaam]

Blocker for #1574. Raising.

[totaam]

2017-07-17 14:48:04: antoine commented

---

It might be easier to switch to [http://www.pyinstaller.org/].
Their documentation is a lot better.

[totaam]

2017-09-28 08:21:53: antoine commented

---

As per comment:1, we can't use pyinstaller as we can no longer use pywin32.

Despite some ugly workarounds in r16970 and using [/attachment/ticket/1528/cx-freeze-lib.patch], cx_freeze 5.0.2 still fails to package things properly: No module named startup and the workarounds suggested require messing with the pacman build files as building with pip fails on MSYS2...

Even after manually installing the git v5.x version, some problems remain:

• lots of modules are not packaged properly: "Gtk" / "Glib" / etc, "numpy", "cryptography", "opencv"
• ships "Adwaita" theme and a gazillion icons

[totaam]

• r16978: sqlite3 dll packaging problems
• r16979 + r16981: CUDA 9 and pycuda packaging
• r16982 + r16983: numpy packaging

[totaam]

Note: the 64-bit builds now use cx_Freeze version 5.x (with manual patching since 5.0.2 is so buggy..)

[totaam]

The python2 / GTK2 builds will remain on the 4.x branch, the 5.x branch works for python3 / GTK3.

[totaam]

r20018 re-enables cx_freeze from the msys2 repositories rather than holding it back at an older version.
Packaging works for both python2 and python3 with this version (5.1.1-2)