Package as standalone binary #93

atomotic · 2017-11-14T21:14:07Z

hint: more than the docker image a standalone and multiplatform binary (containing the app and the database) could be simpler to distribute.

i would like to investigate the possibility to package everything with pyinstaller adding the database as a data file

simonw · 2017-11-14T21:47:02Z

Even without bundling in the database file itself, I'd love to have a standalone binary version of the core datasette CLI utility.

I think Sanic may have some complex dependencies, but I've never tried pyinstaller so I don't know how easy or hard it would be to get this working.

simonw · 2017-11-14T22:09:13Z

Looks like we'd need to use this recipe: https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Setuptools-Entry-Point

atomotic · 2017-11-14T22:42:16Z

tried quickly, this seems working:

~ pip3 install pyinstaller
~ pyinstaller -F --add-data /usr/local/lib/python3.6/site-packages/datasette/templates:datasette/templates --add-data /usr/local/lib/python3.6/site-packages/datasette/static:datasette/static  /usr/local/bin/datasette

~ du -h dist/datasette
6.8M	dist/datasette
~ file dist/datasette
dist/datasette: Mach-O 64-bit executable x86_64

simonw · 2017-11-14T22:51:46Z

That didn't quite work for me. It built me a dist/datasette executable but when I try to run it I get an error:

$ pwd
/Users/simonw/Dropbox/Development/datasette
$ source venv/bin/activate
$ pyinstaller -F --add-data datasette/templates:datasette/templates --add-data datasette/static:datasette/static /Users/simonw/Dropbox/Development/datasette/venv/bin/datasette
$ dist/datasette --help
Traceback (most recent call last):
  File "datasette", line 11, in <module>
  File "site-packages/pkg_resources/__init__.py", line 572, in load_entry_point
  File "site-packages/pkg_resources/__init__.py", line 564, in get_distribution
  File "site-packages/pkg_resources/__init__.py", line 436, in get_provider
  File "site-packages/pkg_resources/__init__.py", line 984, in require
  File "site-packages/pkg_resources/__init__.py", line 870, in resolve
pkg_resources.DistributionNotFound: The 'datasette' distribution was not found and is required by the application
[99117] Failed to execute script datasette

atomotic · 2017-11-14T23:06:33Z

i will look better tomorrow, it's late i surely made some mistake
https://asciinema.org/a/ZyAWbetrlriDadwWyVPUWB94H

simonw · 2017-11-14T23:56:35Z

It worked!

$ pyinstaller -F \
    --add-data /usr/local/lib/python3.5/site-packages/datasette/templates:datasette/templates \
    --add-data /usr/local/lib/python3.5/site-packages/datasette/static:datasette/static \
    /usr/local/bin/datasette

$ file dist/datasette 
dist/datasette: Mach-O 64-bit executable x86_64
$ dist/datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette!

Options:
  --help  Show this message and exit.

Commands:
  serve*   Serve up specified SQLite database files with...
  build
  package  Package specified SQLite files into a new...
  publish  Publish specified SQLite database files to...

simonw · 2017-11-14T23:58:07Z

It's a shame pyinstaller can't act as a cross-compiler - so I don't think I can get Travis CI to build packages. But it's fantastic that it's possible to turn the tool into a standalone executable!

atomotic · 2017-11-15T08:09:41Z

actually you can use travis to build for linux/macos and appveyor to build for windows.

simonw · 2019-10-06T15:56:01Z

Relevant conversation on Twitter: https://twitter.com/simonw/status/1180866651962560512?s=21

simonw · 2021-01-04T20:59:20Z

Updated pyinstaller recipe - lots of hidden imports needed now:

pip install wheel
pip install datasette pyinstaller

BASE=$(python -c 'import os; print(os.path.dirname(__import__("datasette").__file__))') \
    pyinstaller -F \
        --add-data "$BASE/templates:datasette/templates" \
        --add-data "$BASE/static:datasette/static" \
        --hidden-import datasette.publish \
        --hidden-import datasette.publish.heroku \
        --hidden-import datasette.publish.cloudrun \
        --hidden-import datasette.facets \
        --hidden-import datasette.sql_functions \
        --hidden-import datasette.actor_auth_cookie \
        --hidden-import datasette.default_permissions \
        --hidden-import datasette.default_magic_parameters \
        --hidden-import datasette.blob_renderer \
        --hidden-import datasette.default_menu_links \
        --hidden-import uvicorn \
        --hidden-import uvicorn.logging \
        --hidden-import uvicorn.loops \
        --hidden-import uvicorn.loops.auto \
        --hidden-import uvicorn.protocols \
        --hidden-import uvicorn.protocols.http \
        --hidden-import uvicorn.protocols.http.auto \
        --hidden-import uvicorn.protocols.websockets \
        --hidden-import uvicorn.protocols.websockets.auto \
        --hidden-import uvicorn.lifespan \
        --hidden-import uvicorn.lifespan.on \
        $(which datasette)

simonw · 2021-01-04T21:00:14Z

(pyinstaller-datasette) pyinstaller-datasette % file dist/datasette
dist/datasette: Mach-O 64-bit executable x86_64
(pyinstaller-datasette) pyinstaller-datasette % ls -lah dist/datasette
-rwxr-xr-x  1 simon  wheel   8.0M Jan  4 12:58 dist/datasette

I'm surprised it's only 8MB!

simonw · 2021-01-04T21:05:57Z

That BASE= trick seems to work with zsh but not with bash.

simonw · 2021-01-04T21:06:49Z

Works on Linux/Ubuntu too, except I had to do export BASE= on a separate line. I also did this:

apt-get install python3 python3-venv
python3 -m venv pyinstaller-venv
source pyinstaller-venv/bin/activate
pip install wheel
pip install datasette pyinstaller

export DATASETTE_BASE=$(python -c 'import os; print(os.path.dirname(__import__("datasette").__file__))')

pyinstaller -F \
    --add-data "$DATASETTE_BASE/templates:datasette/templates" \
    --add-data "$DATASETTE_BASE/static:datasette/static" \
    --hidden-import datasette.publish \
    --hidden-import datasette.publish.heroku \
    --hidden-import datasette.publish.cloudrun \
    --hidden-import datasette.facets \
    --hidden-import datasette.sql_functions \
    --hidden-import datasette.actor_auth_cookie \
    --hidden-import datasette.default_permissions \
    --hidden-import datasette.default_magic_parameters \
    --hidden-import datasette.blob_renderer \
    --hidden-import datasette.default_menu_links \
    --hidden-import uvicorn \
    --hidden-import uvicorn.logging \
    --hidden-import uvicorn.loops \
    --hidden-import uvicorn.loops.auto \
    --hidden-import uvicorn.protocols \
    --hidden-import uvicorn.protocols.http \
    --hidden-import uvicorn.protocols.http.auto \
    --hidden-import uvicorn.protocols.websockets \
    --hidden-import uvicorn.protocols.websockets.auto \
    --hidden-import uvicorn.lifespan \
    --hidden-import uvicorn.lifespan.on \
    $(which datasette)

simonw · 2021-01-04T21:23:13Z

(pyinstaller-venv) root@dogsheep:/tmp/pyinstaller-venv# dist/datasette --get /-/databases.json
[{"name": ":memory:", "path": null, "size": 0, "is_mutable": true, "is_memory": true, "hash": null}]
(pyinstaller-venv) root@dogsheep:/tmp/pyinstaller-venv# ls -lah dist/datasette 
-rwxr-xr-x 1 root root 8.9M Jan  4 21:05 dist/datasette

simonw · 2021-01-04T21:28:01Z

As an experiment, I put the macOS one in a zip file and attached it to the latest release:

mkdir datasette-0.53-macos-binary
cp dist/datasette datasette-0.53-macos-binary
zip -r datasette-0.53-macos-binary.zip datasette-0.53-macos-binary

It's available here: https://github.com/simonw/datasette/releases/tag/0.53 - download URL is https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip

So I can test #93

simonw · 2021-01-04T21:35:37Z

I tested it by running a tmate session on the GitHub macOS machines, and it worked!

Mac-1609795972770:tmp runner$ wget 'https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip'
--2021-01-04 21:34:10--  https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip
Resolving github.com (github.com)... 140.82.114.4
Connecting to github.com (github.com)|140.82.114.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/107914493/74658700-4e90-11eb-8f3b-ee77e6dfad90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210104T213414Z&X-Amz-Expires=300&X-Amz-Signature=6f3c54211077092553590b33a7c36cd052895c9d4619607ad1df094782f64acf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107914493&response-content-disposition=attachment%3B%20filename%3Ddatasette-0.53-macos-binary.zip&response-content-type=application%2Foctet-stream [following]
--2021-01-04 21:34:14--  https://github-production-release-asset-2e65be.s3.amazonaws.com/107914493/74658700-4e90-11eb-8f3b-ee77e6dfad90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210104T213414Z&X-Amz-Expires=300&X-Amz-Signature=6f3c54211077092553590b33a7c36cd052895c9d4619607ad1df094782f64acf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107914493&response-content-disposition=attachment%3B%20filename%3Ddatasette-0.53-macos-binary.zip&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.43.164
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.43.164|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8297283 (7.9M) [application/octet-stream]
Saving to: ‘datasette-0.53-macos-binary.zip’

datasette-0.53-maco 100%[===================>]   7.91M  --.-KB/s    in 0.1s    

2021-01-04 21:34:14 (73.4 MB/s) - ‘datasette-0.53-macos-binary.zip’ saved [8297283/8297283]

Mac-1609795972770:tmp runner$ unzip datasette-0.53-macos-binary.zip 
Archive:  datasette-0.53-macos-binary.zip
   creating: datasette-0.53-macos-binary/
  inflating: datasette-0.53-macos-binary/datasette  
Mac-1609795972770:tmp runner$ datasette-0.53-macos-binary/datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette!

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect
  install    Install Python packages - e.g.
  package    Package specified SQLite files into a new datasette Docker...
  plugins    List currently available plugins
  publish    Publish specified SQLite database files to the internet along...
  uninstall  Uninstall Python packages (e.g.
Mac-1609795972770:tmp runner$ datasette-0.53-macos-binary/datasette --get /-/versions.json
{"python": {"version": "3.9.1", "full": "3.9.1 (default, Dec 10 2020, 10:36:35) \n[Clang 12.0.0 (clang-1200.0.32.27)]"}, "datasette": {"version": "0.53"}, "asgi": "3.0", "uvicorn": "0.13.3", "sqlite": {"version": "3.34.0", "fts_versions": ["FTS5", "FTS4", "FTS3"], "extensions": {"json1": null}, "compile_options": ["COMPILER=clang-12.0.0", "ENABLE_COLUMN_METADATA", "ENABLE_FTS3", "ENABLE_FTS3_PARENTHESIS", "ENABLE_FTS4", "ENABLE_FTS5", "ENABLE_GEOPOLY", "ENABLE_JSON1", "ENABLE_PREUPDATE_HOOK", "ENABLE_RTREE", "ENABLE_SESSION", "MAX_VARIABLE_NUMBER=250000", "THREADSAFE=1"]}}
Mac-1609795972770:tmp runner$

simonw · 2021-01-04T23:35:13Z

Next step is to automate this all!

simonw · 2021-11-21T07:00:21Z

Closing this in favour of Datasette Desktop: https://datasette.io/desktop

simonw added feature ci labels Jul 10, 2018

simonw added a commit that referenced this issue Jan 4, 2021

tmate session mac

ab7767a

So I can test #93

simonw changed the title ~~package as a standalone binary~~ Package each release as standalone binarys Jan 4, 2021

This was referenced Jan 4, 2021

GitHub Actions workflow to build and sign macOS binary executables #1171

Open

GitHub Actions workflow to build manylinux binary #1173

Open

atomotic changed the title ~~Package each release as standalone binarys~~ Package as standalone binary Jan 5, 2021

This was referenced Aug 24, 2021

Build Datasette with PyInstaller and bundle it into an installable Electron app simonw/datasette-app#7

Closed

Build a macOS PyInstaller binary of Datasette plus the app-support plugin simonw/datasette-app#17

Closed

This was referenced Aug 31, 2021

Bundle Python and install datasette in a virtual environment on startup simonw/datasette-app#27

Merged

Build a working distributable macOS application (signed and notarized) simonw/datasette-app#20

Closed

simonw closed this as completed Nov 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Package as standalone binary #93

Package as standalone binary #93

atomotic commented Nov 14, 2017

simonw commented Nov 14, 2017

simonw commented Nov 14, 2017

atomotic commented Nov 14, 2017

simonw commented Nov 14, 2017

atomotic commented Nov 14, 2017

simonw commented Nov 14, 2017

simonw commented Nov 14, 2017

atomotic commented Nov 15, 2017

simonw commented Oct 6, 2019

simonw commented Jan 4, 2021 •

edited

Loading

simonw commented Jan 4, 2021

simonw commented Jan 4, 2021

simonw commented Jan 4, 2021 •

edited

Loading

simonw commented Jan 4, 2021

simonw commented Jan 4, 2021

simonw commented Jan 4, 2021

simonw commented Jan 4, 2021

simonw commented Nov 21, 2021

Package as standalone binary #93

Package as standalone binary #93

Comments

atomotic commented Nov 14, 2017

simonw commented Nov 14, 2017

simonw commented Nov 14, 2017

atomotic commented Nov 14, 2017

simonw commented Nov 14, 2017

atomotic commented Nov 14, 2017

simonw commented Nov 14, 2017

simonw commented Nov 14, 2017

atomotic commented Nov 15, 2017

simonw commented Oct 6, 2019

simonw commented Jan 4, 2021 • edited Loading

simonw commented Jan 4, 2021

simonw commented Jan 4, 2021

simonw commented Jan 4, 2021 • edited Loading

simonw commented Jan 4, 2021

simonw commented Jan 4, 2021

simonw commented Jan 4, 2021

simonw commented Jan 4, 2021

simonw commented Nov 21, 2021

simonw commented Jan 4, 2021 •

edited

Loading

simonw commented Jan 4, 2021 •

edited

Loading