feat: wasm transformers

[HarikrishnanBalagopal]

LFX link

https://mentorship.lfx.linuxfoundation.org/project/ec286a9e-e48d-4c83-a991-9c79a4ec213a

Move2Kube is a command-line tool for automating creation of Infrastructure as code (IaC) artifacts. It has inbuilt support for creating IaC artifacts for replatforming to Kubernetes/OpenShift. Move2Kube has a very plugin friendly architecture, users can write custom logic in the form of "Transformers" that Move2Kube can integrate seamlessly into its transformation pipeline. So far we have support for both Starlark and container image based transformers. We would like to support writing transformers as WASM modules that Move2Kube can run. WASM provides extensive sandboxing for security, it allows writing transformers in different language stacks like Rust, C/C++, etc. other than Golang, and WASM is just as lightweight and fast as Starlark.

• Expected Outcome:
  • Implement a feature in Move2Kube CLI to allow running WASM modules as custom transformers.

Prerequisites

Please familiar yourself with the topics and try compiling simple Golang programs to WASI (preferably those that use functions imported from the host) with TinyGo so that you are familiar with the workflow and challenges involved.

• Some in-depth knowledge of WASM/WASI and Golang/TinyGo.
• (Optional) In-depth knowledge of Move2Kube will also help a lot but we can help guide you through that.

Getting Started

• If you are familiar with Golang you can start by trying to build the Move2Kube CLI https://github.com/konveyor/move2kube
  Follow the tutorial on how to use it to do a transformation https://move2kube.konveyor.io/tutorials/cli
• Custom transformers
  • tutorial https://move2kube.konveyor.io/tutorials/customizing-the-output/custom-annotations
  • documentation https://move2kube.konveyor.io/transformers/external
  • source code (look at Starlark Transformer code for inspiration) https://github.com/konveyor/move2kube/tree/817b8c901e9d3622a7fcbf6db55a34a08427ca90/transformer/external
• If you are not familiar with WASM and WASI, there are lots of tutorials online. I suggest the TinyGo tutorial https://tinygo.org/docs/guides/webassembly/
• For a more complete example of a Golang program running in the browser you can take a look at https://github.com/HarikrishnanBalagopal/asciidonut
• https://tinygo.org/docs/guides/compatibility/

Challenges

This is not an exhaustive list, just the ones we have found so far.

• Async host function calls. QA engine might require asynchronous function call support in the runtime.
• Compiled function with arguments has wrong signature tinygo-org/tinygo#3010 (comment)
• Tinygo need serialization tools tinygo-org/tinygo#447

Tasks

Beta Give feedback

1. Add support for custom transformers written in WASM

[kanha-gupta]

@HarikrishnanBalagopal any resources available to learn about wasm support ?

[kmehant]

@kanha-gupta You can start with WebAssembly Specifications, Hari will add more to this.

https://webassembly.org/specs/

[HarikrishnanBalagopal]

@HarikrishnanBalagopal any resources available to learn about wasm support ?

Have updated above. Can add more details if necessary.

[mohamedawnallah]

I'm really interested in this project. I wanted to provide you with an update on my progress with the Move2Kube WASM integration for custom transformers.

So far, I have successfully built the Move2Kube CLI and have been using it and (UI) to perform built-in and custom transformations as described in the Move2Kube documentation. Also, I've reported Inconsistent Planning Results Between Move2Kube UI and CLI bug in UI while following the Move2Kube tutorials. It has been an exciting experience so far, and I'm impressed with the plugin-friendly architecture that Move2Kube offers.

Currently, I am dedicating time to learn about TinyGo, WASM specifications, and how to compile Golang programs to the WebAssembly System Interface (WASI) API, with a particular focus on functions imported from the host.

I will let you know if any progress or questions about the project!

[mohamedawnallah]

Hi @HarikrishnanBalagopal @kmehant,

I wanted to share with you my recent experience with the Web Assembly using TinyGo. I took the time to understand the differences between TinyGo and Go Default Compiler and compiled a simple Golang program for WASI using TinyGo.

The program involved image processing for a checkerboard, and I imported functions from the host into JavaScript. Specifically, I configured the checkerboard size to be 1000 pixels x 1000 pixels and compared its performance with the vanilla JavaScript version. The results were impressive, as the computationally intensive task performed with Go and WebAssembly was nearly twice as fast as the JavaScript version. I have attached a demo for you to see where the website is refreshed every half second to illustrate the performance differences of checkboard generation.

checkboard_image_processing.mp4

Moreover, I noticed that the performance margin increased until it hit the threshold of the machine's memory, which highlights the efficiency of implementing such tasks in Go using WebAssembly.

After going through this experience, I have a solid understanding of why WebAssembly is an excellent option for handling computationally intensive tasks. It clearly demonstrates the advantages of combining TinyGo and WebAssembly, as they prove to be lightweight and remarkably fast together.

[mohamedawnallah]

@HarikrishnanBalagopal @kmehant Now I'd like to know where is the Move2Kube transformation engine source code for custom transformers located ?

[DhruvSinghiitmandi]

Hi @kmehant @HarikrishnanBalagopal
I am really interested in this project and issue. How can i start learning and contributing to this issue? . So far I have built the move2kube CLI and tried the WASM and WASI TinyGO Tutorial ,currently I'm trying to figure out how Transformers and Custom Transformers function

[HarikrishnanBalagopal]

@mohamedawnallah @DhruvSinghiitmandi
Thanks for your interest, please apply on the LFX portal https://mentorship.lfx.linuxfoundation.org/project/ec286a9e-e48d-4c83-a991-9c79a4ec213a
You can use your findings to design a project proposal in the list of tasks in the LFX portal.

@HarikrishnanBalagopal @kmehant Now I'd like to know where is the Move2Kube transformation engine source code for custom transformers located ?

Have added the link to tutorial, code and documentation to the Getting started section in the first post.

Hi @kmehant @HarikrishnanBalagopal
I am really interested in this project and issue. How can i start learning and contributing to this issue? . So far I have built the move2kube CLI and tried the WASM and WASI TinyGO Tutorial ,currently I'm trying to figure out how Transformers and Custom Transformers function

Have added some more links to the custom transformers docs and code in the Getting started section. If you have any specific questions regarding code/documentation please post it in this thread or ask in the #konveyor channel on the Kubernetes Slack workspace https://kubernetes.slack.com/archives/CR85S82A2

[Srishti969]

Hi @kmehant @HarikrishnanBalagopal !
What will the selection process be for this project ? Will you be looking for the progress made on this specific project?

[HarikrishnanBalagopal]

Hi @kmehant @HarikrishnanBalagopal ! What will the selection process be for this project ? Will you be looking for the progress made on this specific project?

There is a task assigned to all candidates to design a technical proposal on how they will solve this issue.
There may also be an interview with the shortlisted candidates.

Progress made on the issue may also be considered but not a requirement. We are only expecting candidates to have a good understanding of the issue, technologies involved, and the code base.

[vishalrajofficial]

I am Vishal Raj, a final year CSE student. The Konveyor: Move2Kube project for implementing WASM Transformers sounds like an interesting challenge. I'm excited to contribute to this CLI tool and support the running of WASM modules as custom transformers. My familiarity with Golang and knowledge of WASM/WASI will be valuable assets in tackling this project. I'm eager to explore this plugin-friendly architecture and leverage the extensive sandboxing and language versatility of WASM to enhance Move2Kube's capabilities. Let's work together to make Move2Kube even more powerful and efficient with custom WASM transformers!

[DhruvSinghiitmandi]

@kmehant @HarikrishnanBalagopal
How is wasmer for loading and executing wasm modules(transformer) ?

[SVKREP]

Thanks for accepting me as a mentee in the LFX program. I am currently working on this.