Calibrator in selfcal and facet images

[duyhoang-astro]

I have a bright calibrator in the field which should be subtracted properly with direction-dependent gain. The calibrator in the selfcal images looks fine, but not ok in the facet image. As I would expect they are the same since they are corrected with the same solutions. Would it be something wrong during the transferring solutions to the facet data? The other sources in the facet image seem not to be properly deconvolved either.

(left is selfcal calibrator, right is calibrator in facet image, the two images have similar resolutions, color scale is the same for both images)

Factor branch: master (current version)
LOFAR version 2.9
wsclean version: 2.5

[rvweeren]

The other sources in the facet image seem not to be properly deconvolved either.

I think the deconvolution is -not- a real problem. Currently it will not fully deconvolve because of the artifacts. If the artifact problem (suggesting sort of an 'applycal' / correction issue) is solved I suspect the deconvolution will work fine.

The last image of the calibrator in the DDE selfcal cycle should just be very similar to the full facet image with that calibrator. After all the solutions do not change anymore for that facet and the data should be the same as well, except that some additional sources are present (apart from some different data averaging settings, but that should not change how the calibrator source looks, otherwise there is something wrong...)

[darafferty]

I suspect something has gone wrong when the selfcal solutions are merged. Duy, can you send me the *.merge_selfcal_parmdbs and *.convert_merged_selfcal_parmdbs parmdbs in the results/facetselfcal/facetname directory?

[duyhoang-astro]

Sure, here is the link to the parmdbs: https://owncloud.strw.leidenuniv.nl/index.php/s/0NUJ0fHlg5atpWc

[darafferty]

Indeed, there is a difference between the original selfcal solutions and the merged ones (in which all effects have been combined into a single gain). See below: left is merged, right is original -- there is an offset of 0.5 radians or so between them. I'm trying now to figure out where the offset is coming from...

[darafferty]

Nevermind -- I was confused about which frequency parmdbplot.py uses to convert TEC to phase. Once I plotted it correctly, the two are identical. So, still searching...

[rvweeren]

David, do you I in general observe that the calibrator in the facet image roughly has the same quality as in the last DDE selfcal image?

Wondering if the averaging is too aggressive..as I notice some sort WSClean aliasing issue? Or there is a mixup with calibration tables between blocks...? Renormalization of gains going wrong? (error pattern in the image sort of suggests something with the slow-gain solutions not being the same between the last DDE selfcal image and the facet image)

[darafferty]

Yes, usually the calibrator and facet image are very similar, but I have seen a few cases where the facet image looks worse (though not as bad as this one).

I see what you mean about the aliasing (the artifacts to the north and south of the bright source, right?). The averaging can be controlled with the max_peak_smearing parameter (under the [imaging] section of the parset), so it might be worth experimenting with that.

At any rate, I agree that the artifacts suggest something's wrong with the amplitudes but they look fine in the parmdbs that Duy provided. I'm checking them now in more detail to see if they're really the same.

[rvweeren]

I see what you mean about the aliasing (the artifacts to the north and south of the bright source, right?).

Yes.

[duyhoang-astro]

The averaging can be controlled with the max_peak_smearing parameter (under the [imaging] section of the parset), so it might be worth experimenting with that.

Below are the images of the calibrator in the selfcal (left) and in the facet image (right). The first row is from the run with max_peak_smearing = 0.15 (default value), the second row is from the run with max_peak_smearing = 0. The results are similar (beside very small difference in the statistics within the calibrator region).

Here is the full facet image including the facet layout.

[twshimwell]

And you think this is calibration rather than deconvolution? What do the model images and masks look like for the selfcal and facet imaging steps?

[rvweeren]

Wondering about the normalization, since this quite of looks like an amplitude normalization error. David you do not calculate these normalization factors separately per timechunk (or freqblock)?
(because that will certainly lead to problems....)

[duyhoang-astro]

And you think this is calibration rather than deconvolution? What do the model images and masks look like for the selfcal and facet imaging step?

Tim: I did a test with the deconvolution: multiscale clean as we discussed. Below are the model of the calibrator in the selfcal and facet images without and with multiscale clean. The model of the calibrator in the facet model using single scale clean seems not to be deconvolved properly. It has just a few clean components. But with multiscale clean, there are more clean components in the facet model (close to the model in the selfcal model). And the facet image with multiscale clean seems to be a bit better. But still lots of artifacts, maybe something else causing the problem too. Maybe I should try to add a proper mask, as this facet cleaning uses the entire facet mask?

David: is it possible to input a mask in Factor to replace the facet mask (*premask)? I can add CASA region file in the direction file. But I am not sure if it also accepts CASA mask. Another thing is that the the multiscale options (e.g. selfcal_multiscale_scales_pixel = [0, 7]; facet_multiscale_scales_pixel = [0, 7]) in the factor parset do not seem to propagate through the pipeline when it generates the selfcal parset (results/facetselfcal/facet_patch_163/pipeline.parset). I had to change in the factor/pipeline/parsetsfacetselfcal_pipeline.parset. (I am not sure if this is due to my local environment settings though)

[darafferty]

David you do not calculate these normalization factors separately per timechunk (or freqblock)?

Yes, the solutions for all the time and freq chunks are combined before doing the normalization. However, there is a step in which the amplitudes are all set to 1.0: the one that makes the "preapply" parmdb that is preapplied to all the other directions before selfcal. I wonder whether it could be the one used in facet imaging for some reason instead of the normal parmdb? Duy, can you check the contents of results/facetimage/facetname/mapfiles/expand_merged_parmdbs.mapfile? It should have the *.convert_merged_selfcal_parmdbs file and not the *.create_preapply_parmdb one.

David: is it possible to input a mask in Factor to replace the facet mask (*premask)?

No, unfortunately it only accepts a casa region file.

One other thing you might try is to use the other facetimage pipeline by setting automask_facet_image=False (or True if you already have it as False). This should help to rule out the possibililty that the deconvolution is the issue, as the two facetimage pipelines use different ways to do the masking (automasking vs. pybdsf).

Another thing is that the the multiscale options (e.g. selfcal_multiscale_scales_pixel = [0, 7]; facet_multiscale_scales_pixel = [0, 7]) in the factor parset do not seem to propagate through the pipeline when it generates the selfcal parset (results/facetselfcal/facet_patch_163/pipeline.parset).

I'll check the multiscale options. Must be a bug...

[duyhoang-astro]

I wonder whether it could be the one used in facet imaging for some reason instead of the normal parmdb? Duy, can you check the contents of results/facetimage/facetname/mapfiles/expand_merged_parmdbs.mapfile? It should have the *.convert_merged_selfcal_parmdbs file and not the *.create_preapply_parmdb one.

Yes, it does. The results/facetimage/facetname/mapfiles/expand_merged_parmdbs.mapfile contains *.convert_merged_selfcal_parmdbs, not the *.create_preapply_parmdb one.

[duyhoang-astro]

One other thing you might try is to use the other facetimage pipeline by setting automask_facet_image=False (or True if you already have it as False). This should help to rule out the possibililty that the deconvolution is the issue, as the two facetimage pipelines use different ways to do the masking (automasking vs. pybdsf).

Do you mean that setting automask_facet_image=False to make the facet image that is in the, e.g., facetimage_c1.5r-0.25t0.0u200.0? And this will use pybdsf to make a mask instead of automasking with wsclean?

This option, automask_facet_image=False, is not used for imaging the facet during the selfcal, right? As I tried this option, but the automask is still on in the results/facetselfcal/facet_patch_xxx/pipeline.parset:

wsclean_image_full.argument.auto-mask = 3
wsclean_image_full.argument.auto-threshold = 1.0
wsclean_image_full.argument.local-rms-window = 50
wsclean_image_full.argument.local-rms-method = rms-with-min

[darafferty]

Do you mean that setting automask_facet_image=False to make the facet image that is in the, e.g., facetimage_c1.5r-0.25t0.0u200.0? And this will use pybdsf to make a mask instead of automasking with wsclean?

Yes, that's right. Perhaps I was confused -- are the facet images shown above made in the facetselfcal pipeline or later in the facetimage one? I was thinking they were the latter. If they're from facetselfcal, then yes, automasking is always used and automask_facet_image won't have any effect. In this case, it could be the imaging parameters aren't well tuned for really bright sources in the facet imaging of selfcal -- e.g., you could try changing wsclean_image_full.argument.niter to 100000 or so and wsclean_image_full.argument.auto-threshold=0.3 (both of these in the factor/pipeline/parsets/facetselfcal_pipeline.parset, as you did before).

[duyhoang-astro]

are the facet images shown above made in the facetselfcal pipeline or later in the facetimage one?

Yes, these facet images above are from the facetselfcal pipeline.

[duyhoang-astro]

To check the deconvolution issue, I have run wsclean on the data with similar imaging parameters as in the facetselfcal pipeline. But I use a mask that is manually made with pybdsf, instead of using the automasking in wsclean. The bright calibrator is properly deconvolved in the zoom-in image below (left is the calibrator in the selfcalfacet image, right is the mask).

So, maybe for bright sources like this one (>25 Jy), it would be good to have an option to force the facetselfcal pipeline to use automask in wsclean or a mask made with pybdsf?

By the way, I also see some fainter sources that are not deconvolved in other facets. Perhaps, adding the option to select automasking with wsclean or making mask with pybdsf is good for these cases.

[rvweeren]

Can you show that image next to the last image from the DDE selfcal cycle ? (on the same color scale and zoom and showing the same FoV as the selfcal image)

[duyhoang-astro]

yes. (left is the calibrator in the selfcal image, middle is the calibrator in selfcal facet image (factor uses automask), right is the calibrator is the facet image made with pybdsf mask (as in my above comment))

Btw, below is the psf image (that has similar pattern as the noise in the selfcal facet image.)

[rvweeren]

Good comparison and debugging! So it's a masking issue/deconvolution, interesting this makes such a big difference. The remaining aliasing effects are WSClean related I think.

[duyhoang-astro]

The remaining aliasing effects are WSClean related I think.

do you think increasing the image size would reduce the aliasing effects in these images?

[rvweeren]

Yes, worth trying.

[duyhoang-astro]

@darafferty: Is there any quick trick that I can replace the *premask with an input mask for facet imaging in the facetselfcal pipeline?

[darafferty]

You can try to replace the current step with something like (replace your_mask.fits with the full path to you input mask):

premask.control.kind        = plugin
premask.control.type        = addListMultiMapfile
premask.control.hosts       = {{ hosts }}
premask.control.files       = [your_mask.fits]
premask.control.mapfile_dir = input.output.mapfile_dir
premask.control.filename    = mypremask.mapfile

It should then (hopefully) use your mask.