Fix Meilix iso generation pipeline [GSoC 2019 draft]

[r0mflip]

Abstract:

The main goal of Meilix is to generate an iso image with the specific given packages, this has yet been incomplete. In its current stage, Meilix has a lot of unused code and resources lying in it's repository. Changes are being made to both meilix and meilix-generator repos but most changes include aesthetic changes, functionality is barely improved. This proposal targets various parts of the pipeline introducing changes like:

1. Documentation of generation request API
2. Conditional package installation
3. Optionally move meilix-generator to services like Zeit serverless platform for the webapp
4. Use Lubuntu iso images instead of debootstrap

1. Package installation

This part of the proposal aims to implement the following solutions in the package installation and customization of meilix:

1. ISO file generation request API
2. Conditional package installation

1.1. ISO file generation request API [proposal]

The script in meilix-generator, build.py, obtains a fair amount of configuration info from the webapp and sends it to travis build script. But this info is being in-consistently changed and is not documented. This API needs to be documented so that tools like curl can be used to make requests to generate the iso file. The JSON structure needs to be documented. The Travis script should also fail with respective HTTP error message if wrong configuration is supplied.

1.2. Conditional package installation

This scripts on Travis installs various packages inside chroot. Current implementation of the chroot scripts installs only the packages that are required to just start the meilix iso. No additional packages are being installed based on the request. This needs to be completed by introducing an additional script for package installation that performs all the required installations conditionally.

2. Consider using iso file source instead of debootstrap [optional, proposal, experimental]

The downloadable iso file of any linux distro can be used to obtain the filesystem of that distro from the filesystem.squashfs. This can then be chrooted into so as to make changes.

$ sudo mount -o loop,ro lubuntu-18.10-desktop-amd64.iso ./mnt
$ sudo rsync --exclude=/casper/filesystem.squashfs -a ./mnt ./image
$ sudo unsquashfs mnt/casper/filesystem.squashfs

Advantages of using iso file:

1. Maintains the base Lubuntu distro image
2. No manual installation and configuration of display server/environment
3. In sync with Lubuntu distro packages and sources
4. Upgrade to new release is easy (replacing iso image might suffice)

Inspired from https://help.ubuntu.com/community/LiveCDCustomization

3. Webapp migration [proposal, optional]

Currently the webapp meilix-generator is being deployed to Heroku platform. Heroku platform fulfills the requirements for the generator. App containers are called Dynos. Current Heroku service plan is free, in which Dynos sleep after 30 minutes of inactivity and are constrained by free dyno hours.

My [biased] alternative is Zeit's Now platform. The Now V2 platform is based on the concept of serverless which allow multiple language endpoints. A python endpoint can be created, the equivalent of build.py in current deployment that works the same.

Pros of Now platform:

1. Zero startup time, always available endpoint
2. Static assets are cached around global CDN's

Cons:

1. Only 1,000 serverless invocations per day (including builds and deployments)

Conclusion

After all the changes in the proposal are made the repo should be more clean and the pipeline is made simple. Note: Proposed as solution to issue#445

[abishekvashok]

It's a good proposal. I have some comments:

Use Lubuntu iso images instead of debootstrap

This might make the process more complicated and take more time for the system to build. We are using the meilix iso and using debootstrap to modify it inside the Travis CI container.

No additional packages are being installed based on the request. This needs to be completed by introducing an additional script for package installation that performs all the required installations conditionally.

I guess currently we currently have optional packages, don't we?

[r0mflip]

@abishekvashok: This might make the process more complicated and take more time for the system to build. We are using the meilix iso and using debootstrap to modify it inside the Travis CI container.

debootstrap tries to generate the fs locally on which we are chrooting. This is fine [whispering: not so fine, even on Travis] but very hard for local development. It download thousands of packages and processes them. ISO file comes pre-configured. Current setup hardly targets local development. If the image is somehow cached then the network on Travis would be saved.

@abishekvashok: I guess currently we currently have optional packages, don't we?

chroot_reconst.sh it's complete yet, even chroot.sh doesn't install packages conditionally. I was targetting all the scripts.

I've been targetting this change since a while. Thought it might fit into a GSoC timeline, but nevermind anyways.

[xeon-zolt]

we can create a make script instead of bash to only continue from step it last failed on so that it saves time for local development
I was doing something like that https://github.com/xeon-zolt/meilix/tree/newstart

[r0mflip]

I would choose shell scripts over makefiles, they are more flexibility and are easy to understand and develop (by beginners especially).

Local development is possible by using only shell scripts where you can just test for the environment and ignore any Travis specific ops like uploading the iso file and getting configuration from command line.