Generate AppImage

[coryschwartz]

This generates an AppImage, which bundles the libraries into a portable executable.

As with https://github.com/filecoin-project/lotus/pull/6202/files this is intended to run on Ubuntu 18.04 which is easily available in CI. AppImageBuilder.yaml is generaed using appimage-builder https://appimage-builder.readthedocs.io/en/latest/ on Ubuntu 18.04, but if you want to generate images yourself and do not have this OS installed, you can re-generate it.

possible solution to #6030

However, anyone can can run the resulting appimage wihtout worrying about installing dependencies.

https://gist.github.com/coryschwartz/552a0983925d50f575c2fd17a8b5032a

just showing that this is, indeed, lotus:

◉ ./Lotus-v1.8.0-x86_64.AppImage 
NAME:
   lotus - Filecoin decentralized storage network client

USAGE:
   lotus [global options] command [command options] [arguments...]

VERSION:
   1.11.0-dev+mainnet+git.dbd191239.dirty

COMMANDS:
   daemon   Start a lotus daemon process
   backup   Create node metadata backup
   version  Print version
   help, h  Shows a list of commands or help for one command
   BASIC:
     send    Send funds between accounts
     wallet  Manage wallet
     client  Make deals, store data, retrieve data
     msig    Interact with a multisig wallet
     paych   Manage payment channels
   DEVELOPER:
     auth          Manage RPC permissions
     mpool         Manage message pool
     state         Interact with and query filecoin chain state
     chain         Interact with filecoin blockchain
     log           Manage logging
     wait-api      Wait for lotus api to come online
     fetch-params  Fetch proving parameters
   NETWORK:
     net   Manage P2P Network
     sync  Inspect or interact with the chain syncer

GLOBAL OPTIONS:
   --help, -h     show help (default: false)
   --version, -v  print the version (default: false)

and that this does not require installation of dependencies:

◉ ldd Lotus-v1.8.0-x86_64.AppImage 
        not a dynamic executable

[coryschwartz]

master was broken, so I updated the gist to have lotus v1.8.0

[coryschwartz]

this isn't ready yet. I'm changing this to a draft PR to work out an issue. There is a bug that I need to troubleshoot, when this is run on another machine, go.rice file embeds are not working correctly.

$ ./Lotus-v1.8.0-x86_64.AppImage  daemon
2021-05-07T16:48:54.733Z        INFO    main    lotus/daemon.go:214     lotus repo: /home/vagrant/.lotus
2021-05-07T16:48:54.735Z        INFO    repo    repo/fsrepo.go:122      Initializing repo at '/home/vagrant/.lotus'
panic: could not locate box "proof-params"

goroutine 1 [running]:
github.com/GeertJohan/go%2erice.MustFindBox(...)
        /home/vagrant/go/pkg/mod/github.com/!geert!johan/go.rice@v1.0.0/box.go:110
github.com/filecoin-project/lotus/build.ParametersJSON(0xc000e3a3c0, 0x3598ea0, 0xc000e3acc0)
        /home/vagrant/go/src/github.com/filecoin-project/lotus/build/parameters.go:6 +0xf3
main.glob..func3(0xc000e3a3c0, 0x0, 0x0)
        /home/vagrant/go/src/github.com/filecoin-project/lotus/cmd/lotus/daemon.go:234 +0x1db4
github.com/urfave/cli/v2.(*App).RunAsSubcommand(0xc000276900, 0xc000e05e40, 0x0, 0x0)
        /home/vagrant/go/pkg/mod/github.com/urfave/cli/v2@v2.2.0/app.go:432 +0x9bf
github.com/urfave/cli/v2.(*Command).startApp(0x5ec9600, 0xc000e05e40, 0x30e5e8c, 0x7ffcc0552da0)
        /home/vagrant/go/pkg/mod/github.com/urfave/cli/v2@v2.2.0/command.go:279 +0x6bb
github.com/urfave/cli/v2.(*Command).Run(0x5ec9600, 0xc000e05e40, 0x0, 0x0)
        /home/vagrant/go/pkg/mod/github.com/urfave/cli/v2@v2.2.0/command.go:94 +0x9cd
github.com/urfave/cli/v2.(*App).RunContext(0xc000276480, 0x3598ee0, 0xc000056090, 0xc00000e060, 0x2, 0x2, 0x0, 0x0)
        /home/vagrant/go/pkg/mod/github.com/urfave/cli/v2@v2.2.0/app.go:306 +0x81f
github.com/urfave/cli/v2.(*App).Run(...)
        /home/vagrant/go/pkg/mod/github.com/urfave/cli/v2@v2.2.0/app.go:215
github.com/filecoin-project/lotus/cli.RunApp(0xc000276480)
        /home/vagrant/go/src/github.com/filecoin-project/lotus/cli/helper.go:35 +0x7c
main.main()
        /home/vagrant/go/src/github.com/filecoin-project/lotus/cmd/lotus/main.go:75 +0x645

[coryschwartz]

I took another look at this.

The way the rice box files were included in the original binary didn't carry over to the appimage binary. Embedding the files fixes this.

If you want to give this file a go.

https://gist.github.com/coryschwartz/aff1f89b2ccca40d025addb271279562/raw/63c027e0b2ab92cee26e4aa22ba5281b284f2eda/Lotus-current-x86_64.AppImage

If this seems reasonable, I can wire up CI to build and release this automatically.

In case you want to build this yourself, the AppImageBuilder.yml file assumes we are building from Ubuntu 18, and this script has the dependencies needed to run make appimage.

https://gist.githubusercontent.com/coryschwartz/aff1f89b2ccca40d025addb271279562/raw/fb7b738ed4c3f46e212120d4b20674b4e7fe7bd6/setup_vm.sh

[magik6k]

This will generate a massive Go file which slows down builds, and likely IDEs.

At this point we should probably just switch builtin Go embed (https://golang.org/pkg/embed/)

We only use rice in builtin/ in a few places:

build/bootstrap.go
18:     b := rice.MustFindBox("bootstrap")

build/openrpc.go
31:     data := rice.MustFindBox("openrpc").MustBytes("full.json.gz")
36:     data := rice.MustFindBox("openrpc").MustBytes("miner.json.gz")
41:     data := rice.MustFindBox("openrpc").MustBytes("worker.json.gz")

build/parameters.go
6:      return rice.MustFindBox("proof-params").MustBytes("parameters.json")

[coryschwartz]

Good point about the huge source file.

I have swapped out go rice for go:embed for everything under build. There is still a rice box in the townhall and fountain and that can be removed as well if we want to switch all embeded files to go:embed.

This feature is only available on go 1.16+ a

[coryschwartz]

@magik6k here's the appimage generated with go:embed in case you want to try it out.

https://gist.github.com/coryschwartz/ec99152d4425cf2c69797abd9f1742d4/raw/d51611bd0f9059c34d6628a9e27afa675c4be1be/Lotus-current-x86_64.AppImage

[magik6k]

(rebased on latest master)

[magik6k]

This is really cool. I'm guessing the next step here will be building those images in CI and including in releases