API Extensibility with existing and new algorithms

Vision to make Karabo extensible and useable for a wide range of use cases

This page shows ideas, that could be implemented, to make karabo more extensible with the rest of the radio astronomy software world.

Polymorphic and clearly defined data structures

Python is fully implementing multiple inheritance and shows through that potential to interface with many programs, without the user having to know the internal data structure. Karabo might implement two different imaging algorithms with differently formatted data inputs; however, both requiring mathematically the same input. Let's propose both imaging algorithms use some sort of measurement set for their input data structure. However, one imaging algorithm uses dask-ms and another uses their own implementation for the memory representation of the measurement set (for example, rascil has their own implementation). This is an issue and would require to use specific transformations, so the data structure can be used in both algorithms. By implementing both of these interfaces into a single data structure, the user can focus on the science and not have to worry about transformations.

An Example:

# read the ms file with the karabo provided function
ms = karabo.read_ms('my-funny-ms.ms')

# polymorphic towards dask-ms
image = dask_ms_implemented_imager(ms)

# polymorphic towards rascil
image = rascil_imager(ms, params=params)

# polymorphic towards ur implementation
image = bluebuild(ms)

Containerized extension for your code

You have created an amazing algorithm and want to use the provided tools of karabo-pipeline in your tests. So far you have been working with a single script that you manually start with python3 my_code test.ms --param=3 or inside a Jupyter notebook. Your code takes in an ms file and returns some fits images.

What you can do is simply install the karabo-pipeline into the same environment and start using it with your code. As long as the output and input of your algorithm is readable by the karabo-pipeline.

To make your life easier, karabo has an interface to containerize your code and let interop with the rest of karabo flawlessly. U write a small piece of code declaring your container, for example, like this.:

my_container = Karabo.Container("my_code") #make a new container
my_container.script = "./my_script.py" # set an existing script as the execution in the container
my_container.script = my_function # set a defined function (for example in the current notebook)
my_container.env = this # extract currently installed packages in this python environment to install them also into the container

my_container.parameters = [
{ 'ms_file': Karabo.MS }, # define the inputs to make sure karabo can read the inputs and outputs
{ 'log': number }
{ ... }
]

my_container.build('singularity') # create container image as a singularity image
my_container.push() # push image to karabo container registry

Use the code anywhere with karabo now. On a different machine only with karabo installed, you can now run this, execute your code.:

my_script = Karabo.container.get("my_code")
result = my_script.run(ms,2) # simply use the defined interface like it was a normal python function