Web installer for Windows / Linux

[Hecatron]

Hi,
I've written a small constructor script for generating an installer for cadquery
This can be used to generate an exe (windows) or sh (Linux)
which will install miniconda / cadquery onto a windows or linux system
The general idea is to make it easier for windows users to setup without accidentally overriding they're default paths.

The installer doesn't directly include cadquery but instead runs a post install script instead
To download / install the bits needed for cadquery such as opencascade etc
This way the installer is kept to a small size instead of being 2Gb.

The settings for windows are set to not to add to the system path by default
Or override an existing python install in terms of the PATH athough there are tick boxes to change that in the installer gui
I spotted that was something someone mentioned in one of the other issues as a problem, with miniconda changing they're default python.

It may also work for macos as well I'm not sure
There was some suggestion in the constructor docs that the post install stage doesn't work for macos but at the same time they provide an example, so it may or may not work for that one I'm not sure.

I'm not quite sure how to do azure workflows at the moment but I've tried to make it as easy as possible to call
The first parameter being the version number for the installer, the second being the github tag that will actually be installed

conda install jinja2 constructor
cd cadquery/conda/web-installer
python build.py 2.2 master

To Activate the environment

# Under Windows
condabin\activate.bat
# Under Linux
source ~/cadquery/bin/activate

My eventual plan is to see if it's possible to access the dll's from python from outside of conda
Since these installers put cadquery into a predictable directory path that can be searched for.
using os.add_dll_directory / sys.path.append, although I haven't quite managed that yet.

If nothing else it makes it easier to install and possibly might increase adoption a bit.

[Hecatron]

I should probably add so far I've tested this on windows 10 and the latest ubuntu 21.10 in a VM

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.27%. Comparing base (23de5cc) to head (db45761).
Report is 380 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1000   +/-   ##
=======================================
  Coverage   96.26%   96.27%           
=======================================
  Files          40       40           
  Lines        9342     9357   +15     
  Branches     1103     1103           
=======================================
+ Hits         8993     9008   +15     
  Misses        205      205           
  Partials      144      144           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[adam-urbanczyk]

Thanks, what is the point of not bundling cadquery directly? You have to download it anyway...

[Hecatron]

Github has a limit on the size of files that can be put into a release tag (2Gb)
So if you wanted to add this to the releases page you wouldn't be able to.
"Each file included in a release must be under 2 GB. There is no limit on the total size of a release, nor bandwidth usage."

[Hecatron]

Post Install Fixes

I've removed the update to conda in the post install script as I think that seems to break things for some reason
conda update -q conda

Also I've added in the "-c conda-forge -c cadquery" options, I thought those wouldn't be needed due to the conda_default_channels setting, but it looks like they still are for things to work correctly

Using from an external non conda python

One upside to this is that I've discovered you can now use cadquery from a separate pip based python environment without the need for wheels.

Installation:
This part is optional as these packages can be picked up on sys.path from the conda install

# First install some depends in the python virtual env (some of these might be for cq-editor)
pip install multimethod nptyping typing_extensions ipython ezdxf pyparsing>=2.1.9 logbook pyqt5 pyqtgraph spyder path.py requests

# install cadquery
pip install git+https://github.com/CadQuery/cadquery@master

Add the following to you're python code prior to "import cadquery" (this is the important bit)

sys.path.append("C:/Users/ric/USERNAME/cadquery/lib/site-packages")
os.add_dll_directory("C:/Users/USERNAME/cadquery/Library/bin")

This allows you to use the OCP dll's and dependencies installed via conda
The only downside is you'd need to manually update OCP in the conda install when needed since it won't auto update when used from a regular virtenv based python, but I'm more than happy with that.

[jmwright]

@grbd I'm just getting around to trying this on Linux. It worked just fine on Linux, and I plan to try Windows and MacOS as well. By default the installer installs jupyter-cadquery, which brings a lot of other packages along with it (i.e. zeromq and tornado) that many users won't ever need. Could the jupyter-cadquery install be made optional as part of the install wizard?

[Hecatron]

I couldn't see a way to add things to a menu within constructor so I've removed jupyter-cadquery from the install scripts
I'm guessing it might be matplotlib that might be bringing a lot of stuff in that it depends on

[jmwright]

Ok, thanks. Here are a few other notes/questions.

1. This installer also works on an Intel-based Mac with MacOS Mojave installed. I think you can just tack MacOS onto the Linux instructions.
2. How would we distribute this? Will the installers be attached to a GitHub release?
3. How will updates work? Will the user need to delete the cadquery directory and re-run the installer?
4. The main readme should probably be updated to mention this installation method.

[Hecatron]

1. Now fixed

2. For distribution the way I see it is you could have it attached to the github release
   it's only around 70Mb and you could probably automate it partially with a workflow
   I'm not familiar with azure workflows myself but something like this
   followed by a task to attach the end result to the github release

build.py release_version git_version
build.py 2.2 2.2

3. For updates you'd ether have to manually update it inside the conda environment

cd C:\ProgramData\cadquery
condabin\activate.bat
conda install -c conda-forge -c cadquery -y cadquery=master

Or uninstall / re-install the self installer
Under windows for example if you try and install to the same directory with a setup exe it will complain that the directory already exists (it doesn't try to uninstall the old version)

4. I wasn't sure if to wait until the github release part was automated before doing that.

[jmwright]

Since the web installer can install any tag of CadQuery (including master), I guess the installer does not have to be tied to a release. It just needs to be available for download from somewhere. The installer code itself probably won't change that much, will it?

A couple of other things that I forgot to mention.

1. On Windows, the installer dialog mentions Python 3.1, but installs Python 3.10.
2. The installer has a version number of 2.2 in the file name, which is the unreleased version of CadQuery. Is that meant to be the CadQuery version, or the installer version? I don't think the installer needs to be tied to a specific CadQuery version since it can install several versions (like the rvm (Ruby) or nvm (Node) utilities).

[Hecatron]

For the 3.1 issue, that might be a constructor related bug maybe it's picking up

specs:
  - python 3.10*

then truncating the version string perhaps

The "installer version" can be anything you want, I just used 2.2 as an example it's fairly arbitrary. It's just a label that shows up at least under windows in the list of installed apps in the control panel / name of the installer.

However you would need a different installer for each release as you can't just tell it to install a different version when actually doing the install in the wizard, just when you build the installer.
When you build / generate the installer using build.py it puts the version of cadquery to download / use into the post install script using a simple jinja template

Example 1

So for example if I run

build.py randomtext 2.1

• This would generate a file called "cadquery-randomtext-Windows-x86_64.exe"
• Under the windows control panel it'd show up as "cadquery randomtext (Python 3.10.0 64-bit"
• The actual version of cadquery it would try to install would be "2.1" (I'm guessing this would fail due to lack of 3.10 support)

Example 2

build.py someothertext_v2.x master

• This would generate a file called "cadquery-someothertext_v2.x-Windows-x86_64.exe"
• Under the windows control panel it'd show up as "cadquery someothertext_v2.x (Python 3.10.0 64-bit)"
• The actual version of cadquery it would try to install would be "master"

So for a "master" version you could just always use the same installer but for specific versions you'd need one per version.

[jmwright]

However you would need a different installer for each release as you can't just tell it to install a different version when actually doing the install in the wizard, just when you build the installer.

Ah, ok. Yes, I wasn't thinking that through.

So it sounds like we could create version 2.1 web installers for each OS and then retroatively attach them to the existing 2.1 release. After that the release installer builds could be automated. Then we can create a web installer set to "master" and use that as the development installer, but we'll have to figure out where to store it on the Web.

[Hecatron]

Yes thats correct
I think the only thing you'd have to change for 2.1 is the python version from 3.10* to something lower in the "construct.yaml.jinja2" file

[jmwright]

I am able to get web installers working well for all three major OSes, so I'm inclined to merge this. The biggest outstanding question is how we will distribute the installers, but that does not have to hold up this PR. @adam-urbanczyk @marcus7070 @lorenzncode do you have any thoughts?

[marcus7070]

LGTM, although I am unable to test it out.

If we attach this to a GH release, eg. v2.2, then at the time the installer is built v2.2 will not be published to conda-forge yet. So we can't write a CI workflow that will test this installer, right? Which isn't a dealbreaker, just trying to get it clear in my head.

[jmwright]

So we can't write a CI workflow that will test this installer, right?

I think that's correct. We'd have to attach the installers to the releases after the fact.

I'm kind of wondering if we should make the installer for the current master (that always installs the bleeding edge) a download from a general location rather than having to attach it to a release.

[jmwright]

I have created web installers for the master branch and put them in a temporary download location. I'll post this to Discord to ask for testers.

Windows Download
MacOS Download
Linux Download

[jmwright]

I've used these scripts and they're available to the community at a temporary download location. I'm inclined to merge this as the installers have worked fine on all three OSes. @adam-urbanczyk @lorenzncode what do you think?

[adam-urbanczyk]

+1

[lorenzncode]

I gave the Linux Download script a try.

If it's useful for Windows users, or someone unfamiliar with conda, that's great.

Perhaps there can also be an option to install cq-editor automatically (yes by default).

[jmwright]

@lorenzncode Is that a request for the future, or are you suggesting that we delay merging until that's added?

[lorenzncode]

My understanding is this method is intended for users that don't know conda (yet). The description says it will be easier to set up without "accidentally overriding their default paths."

Say the user would like to run cq-editor - it must be installed with conda anyway. I wasn't sure whether this method is then much easier compared to a standard miniforge/miniconda install.

Is that a request for the future

Yes, a future enhancement perhaps after more feedback.

[jmwright]

Thanks @grbd !

[Hecatron]

Just to add, another reason for doing this was to try and install cadquery on a default path that's easy to find under windows (or linux as far as the home directory). That way it would open up the possibility of using the conda install from within a pip environment with something like this

# cadquery_vloader/__init__.py

import sys
import os
from os import environ, path

# This allows for using cadquery from an external non conda python installation
# Such as a python pip virtual environment


def find_cadquery_conda_dir():
    """Search for installed cadquery conda instance"""

    # Search the Home directory
    cadquery_home = path.join(path.expanduser("~"), "cadquery")
    if path.isdir(cadquery_home):
        return cadquery_home

    # Search the ProgramData directory for system wide installs
    if sys.platform == "win32":
        cadquery_progfiles = path.join(environ["ALLUSERSPROFILE"], "cadquery")
        if path.isdir(cadquery_progfiles):
            return cadquery_progfiles

    return None


def import_cadquery_conda(condapath=None):
    """Attempt to locate and import cadquery from an external conda install"""
    if condapath is None:
        condapath = find_cadquery_conda_dir()
    if condapath is not None:
        site_dir = path.join(condapath, "lib/site-packages")
        sys.path.append(site_dir)
        bin_dir = path.join(condapath, "Library/bin")
        os.add_dll_directory(bin_dir)


import_cadquery_conda()

Then you can just use it like this

# testpart.py

import cadquery_vloader
import cadquery as cq

def create_part():
    length = 80.0  # Length of the block
    height = 60.0  # Height of the block
    thickness = 5.0  # Thickness of the block
    result = cq.Workplane("XY").box(length, height, thickness)
    return result

create_part()

You could run into problems keeping the binaries installed under conda and the python files installed via pip in sync. But it would avoid having lots of multiple installs of conda for each virtual environment (which are quite big), instead you can just use one for all virtual pip envs.

I've been planning on putting the above example into a seperate python package on github but just haven't gotten around to it yet.