Bringing back lively.app

[linusha]

In the past, we ran into situations where a missing rebuild or an outdated version of git lead to problems. This happened multiple times and with technically-able users. As lively.next has the explicit goal of fostering collaboration between designers and programmers, I think this constitutes a call for action on our side:

Retrieving a working lively.next setup and keeping it up to date needs to become easier. Now, it is a bit ironic that we only recently scrapped the attempts of a working docker setup we had. But in any way I do not think docker constitutes a good solution to this problem (and it never meant to), as the end result would again be people needing to copy-paste some docker commands they do not understand.

I though about two ways to tackle this:

1. Bringing back a lively executable, that takes care of installing/updating lively and spins up a server.
2. Provide a stable lively server that people can just connect to with their browser.

---

Server Driven Approach

I think we would actually have everything ready for this to function. Actually promoting this would probably require us to spend (significant?) amounts of time thinking about user separation and other security measures, but for a select group of users that would be of less importance, I'd think.
However, I see some major downsides to this approach:

1. It would require us to actually administer a server, which might or might not be much work. With the current velocity with which we proceed, it would definitively be something that we would need to think of often.
2. What I think is more problematic: It actually distances the users quite a bit from what they actually do, most significantly from their file-system. As lively.project is still in its infancy, I have worries that users might loose data and cannot easily restore/save/backup parts of their work when lively.project falls flat under them.

Stuff like spontaneous builds would also become much harder, forcing each retrieval of an artifact to go through GitHub. I have automagical actions for this nearly ready, but it would still constitute an annoyance.

lively.app

This leaves us with the option of a prepared lively application that can just be installed/executed like any other program.
Concretely, this would mean providing an electron app or something similar (more on that later).

Dependencies

The most obvious hurdle to take is that, at the moment, we require quite a some dependencies. As these are to be installed as native applications, we cannot just bundle them up (at least not with far too much involvement, i.e., thinking about platforms, following release cycles,...). Let's consider our list from the README:

• node.js v18.12.1
• git

These are actually important. More on that below.

• sed or gsed on MacOs
• ss or netstat on MacOs
• perl
• python3 with sultan installed

These are only used inside of various scripts (mainly the bulk testing script and the artifact checker). I'd argue that these are not important for the average user. The overall goal of this undertaking should not be to move everyone onto the app, but the workflow of us as core developers will probably remain unchanged. Thus, we can just ignore those, when thinking about app support.

• brotli

Here, I am unsure. Looks like the freezer already uses this via a npm library - correct, @merryman?

• aspell

We provide spelling correction via a command inside of lively. While I think this feature is overall quite bare-bones at the moment and could use some love in general, we do not have any special requirements that I am aware of, so we will be able to switch to some JS library (there are plenty of options).

How to Create an Executable

As we need to bundle a node application, we will need to use something like electron, nw.js or pkg. There are other tools available, but these are the ones I looked at. Using any one of these will take care of our node dependency.

Which tool to use?

From looking at it, electron seems to be the best documented of the ones listed here. I'd prefer using that, to be honest. An additional benefit of using electron would be that we would probably come quite far with the lively.app basis that we already had in the past. The only advantage I would see with using nw.js is that we were thinking about bundling a complete lively app at one point, that utilizes the option of calling the node API in the browser, to get an improved debugging experience inside of lively. Pursuing this idea would then require another switch.

git

This leaves us with our dependency on git, which has become quite the dealbreaker since the introduction of lively.project. While I think installing this would be alright, even for non-technical users (we could still improve our installation instructions), we could also experiment with getting rid of calls to a native git application from within the client.
For this, I looked at the available JS/node libraries for git. The best option we have (since nodegit seems to be a nightmare in with regards to interoperability) seems to be isomorphic-git. With that it is possible to operate directly on a git repository in a git compatible way, i.g. actual files get manipulated, which is what we'd want. I identified the following blockers:

1. isomorphic-git does not provide all operations git provides and the output is obviously different. Before proceeding we thus should:

• check all git operations we need and see if we can map them
• if we were to proceed with this, we would need to spend some time enhancing the code where ever we use git in order to account for the different output

2. isomorphic-git needs a FileSystem in order to operate. On node side, one can just use fs. This is the easier option, but it would mean that we would need to use evaluateOnServer quite often and I am undecided how desirable that is. The other option would be to experiment with writing our ShellClientResource in a way that it fulfills the requirements that isomorphic-git has with regards to a file system. The list of operations that one needs to support is the following (taken from here):

Using the "promise" API (preferred)

A "promise" fs object must implement the same set functions as a "callback" implementation, but it implements the promisified versions, and they should all be on a property called promises:

    [fs.promises.readFile(path[, options])](https://nodejs.org/api/fs.html#fs_fspromises_readfile_path_options)
    [fs.promises.writeFile(file, data[, options])](https://nodejs.org/api/fs.html#fs_fspromises_writefile_file_data_options)
    [fs.promises.unlink(path)](https://nodejs.org/api/fs.html#fs_fspromises_unlink_path)
    [fs.promises.readdir(path[, options])](https://nodejs.org/api/fs.html#fs_fspromises_readdir_path_options)
    [fs.promises.mkdir(path[, mode])](https://nodejs.org/api/fs.html#fs_fspromises_mkdir_path_options)
    [fs.promises.rmdir(path)](https://nodejs.org/api/fs.html#fs_fspromises_rmdir_path)
    [fs.promises.stat(path[, options])](https://nodejs.org/api/fs.html#fs_fspromises_stat_path_options)
    [fs.promises.lstat(path[, options])](https://nodejs.org/api/fs.html#fs_fspromises_lstat_path_options)
    [fs.promises.readlink(path[, options])](https://nodejs.org/api/fs.html#fs_fspromises_readlink_path_options) (optional [¹](https://isomorphic-git.org/docs/en/fs#footnote-1))
    [fs.promises.symlink(target, path[, type])](https://nodejs.org/api/fs.html#fs_fspromises_symlink_target_path_type) (optional [¹](https://isomorphic-git.org/docs/en/fs#footnote-1))
    [fs.promises.chmod(path, mode)](https://nodejs.org/api/fs.html#fs_fspromises_chmod_path_mode) (optional [²](https://isomorphic-git.org/docs/en/fs#footnote-2))

¹ readlink and symlink are only needed to work with git repos that contain symlinks.

² Right now, isomorphic-git rewrites the file if it needs to change its mode. In the future, if chmod is available it will use that.

3. There seems to be different experiences with regards to how well SSH works with isomorphic-git. We should

• check whether we actually rely on SSH or if we are good to go with HTTPS only (my notes are inconclusive about this 🤡)

4. Using isomorphic-git for pushing and pulling makes the use of a proxy mandatory. isomorphic-git provides one that can be used for "small projects and testing". It might also be that makeProxied suffices.

• investigate this!

---

Summing up, I think a good approach would be to provide an electron app and utilize isomorphic-git in order to not need a native git installation for lively.project to work.
However, I understand this post merely as an (extensive) starting point for a discussion on how to proceed. I deem this problem to be quite complex and would like to minimize the risk of quick shots.

[merryman]

Here, I am unsure. Looks like the freezer already uses this via a npm library - correct, @merryman?

Yes, but that is a javascript based implementation which is rather slow. I dont think we are using that at the moment.

[linusha]

I briefly thought about bundling a browser with lively as well, thinking that that would make a lot of the file handling for lively.project a lot less painful. However, that is not a good option. The goal of this operation should be ending up with something that is completely compatible to the current lively.next experience, i.e. the current workflow should not be affected!

[linusha]

From looking at it, electron seems to be the best documented of the ones listed here. I'd prefer using that, to be honest. An additional benefit of using electron would be that we would probably come quite far with the lively.app basis that we already had in the past. The only advantage I would see with using nw.js is that we were thinking about bundling a complete lively app at one point, that utilizes the option of calling the node API in the browser, to get an improved debugging experience inside of lively. Pursuing this idea would then require another switch.

We began experimenting with recreating the capability needed to get lively.context (the "lively debugger") to work eventually. The goal of the experiment was to be able to retrieve the value of a variable inside of a closure at rumtime through nodes inspector API and the Runtime Domain of the Chrome Dev Tools Protocol. We were able to get that to work inside a .mjs script running in node.js.
From what we gathered about the process separation model of electron, getting this to work inside of electron might be possible (but not even that is sure), but in any case it requires a lot more work than with nw.js.
Experimenting with translating the node experiment into nw.js are currently blocked by problems with nw.js.

The code of the experiment can be found on the enhancement/app branch.