You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We don't currently support creating full image cubes in either WscleanDirtyImager (combine_across_frequencies=False) or WscleanImageCleaner, even though this is a quite commonly used feature (already came up as a requirement from Devin / ETH and in SRCNet). We should therefore implement this. The relevant WSClean parameter is this one: -channels-out <count> Splits the bandwidth and makes count nr. of images. Default: 1.
While in principle, users can use this parameter through karabo.imaging.imager_wsclean.create_image_custom_command, the function currently fails when using the parameter because this changes the naming of the output files, see https://wsclean.readthedocs.io/en/latest/making_image_cubes.html.
Another blocker that needs to be addressed to solve this issue is that currently, imagers return Image objects which load the full image data into memory. This quickly leads to OOM problems with full image cubes. For example, the full image cube of the PI24 SRCNet Karabo run with 2717 channels and an image size of 20000x20000 would be around 4 TB.
Find a smart solution to deal with image cubes that don't fit into memory. Maybe make type of output configurable with a reasonable default: return full cube, return MFS image (weighted sum over channels), return specific channel, return nothing. Or find a way to stream image into memory on demand when trying to access it downstream.
Properly support -channels-out in create_image_custom_command. Current problems: output_filenames parameter, return type / value.
Support -channels-out in WscleanDirtyImager, possibly change combine_across_frequencies config parameter, change output_fits_path parameter. Specifics depend on task 1.
Support -channels-out in WscleanImageCleaner as well: change dirty_fits_path and output_fits_path parameters, support generating full image cube for both scenarios: when creating a cleaned image from visibilities and from an existing dirty image cube.
We don't currently support creating full image cubes in either WscleanDirtyImager (combine_across_frequencies=False) or WscleanImageCleaner, even though this is a quite commonly used feature (already came up as a requirement from Devin / ETH and in SRCNet). We should therefore implement this. The relevant WSClean parameter is this one:
-channels-out <count> Splits the bandwidth and makes count nr. of images. Default: 1.
While in principle, users can use this parameter through karabo.imaging.imager_wsclean.create_image_custom_command, the function currently fails when using the parameter because this changes the naming of the output files, see https://wsclean.readthedocs.io/en/latest/making_image_cubes.html.
Another blocker that needs to be addressed to solve this issue is that currently, imagers return Image objects which load the full image data into memory. This quickly leads to OOM problems with full image cubes. For example, the full image cube of the PI24 SRCNet Karabo run with 2717 channels and an image size of 20000x20000 would be around 4 TB.
The text was updated successfully, but these errors were encountered: