Reference Livy alternative in Sparkmagic Readme file

[pdambrauskas]

Hi, since it looks like Apache Livy is not maintained anymore: new features are not accepted, Spark version is not being updated, we decided to implement our own solution - Lighter.
Lighter supports Spark batch applications on Yarn and Kubernetes, and PySpark Sparkmagic sessions.

Some more information on how it works can be find here. As well as on documentation pages in our Github repo.

Would it be ok to add a link to our project on the Sparkmagic readme file, so that other sparkmagic users would know that there is an alternative to Livy, which allows to use Sparkmagic with latest PySpark version on K8s (and also Yarn)?

[devstein]

Hi @pdambrauskas thanks for making an issue! Absolutely. Alternatives to Apache Livy are much needed, so I'm excited to see how Lighter evolves.

Just make a PR to add it to the README. In general, always feel to contribute to Sparkmagic. We are always looking for more contributors and maintainers, especially people that work on related projects in the industry.