Merging crystal maps with not_indexed points results in removal of phase

[hakonanes]

Discussed in #636

^{Originally posted by Erlendos12 April 10, 2023}
I have been trying out kikuchipy 0.8.4 which now keeps the not_indexed points after refinement. However now that I try to merge refined crystal maps which contains not_indexed points, all points in that crystal map will now become not_indexed.

E.g. I do hough indexing and refinement on a Super duplex stainless steel with ferrite and austenite.
After Hough indexing, I create maps from the data:

xmap_austenite = kp.indexing.xmap_from_hough_indexing_data(
    data=data,
    phase_list=phase_list,
    data_index=0,  # Phase ID for austenite in phase_list
    navigation_shape=nav_shape,
    step_sizes=(1.0, 1.0),
    scan_unit="um",
)
xmap_ferrite = kp.indexing.xmap_from_hough_indexing_data(
    data=data,
    phase_list=phase_list,
    data_index=1,  # Phase ID for ferrite in phase_list
    navigation_shape=nav_shape,
    step_sizes=(1.0, 1.0),
    scan_unit="um",
)
print(xmap_austenite, "\n")
print(xmap_ferrite)

Phase   Orientations       Name  Space group  Point group  Proper point group     Color
    0  3600 (100.0%)  austenite        Fm-3m         m-3m                 432  tab:blue
Properties: fit, cm, pq, nmatch
Scan unit: um 

Phase   Orientations         Name  Space group  Point group  Proper point group       Color
   -1    405 (11.2%)  not_indexed         None         None                None           w
    1   3195 (88.8%)      ferrite        Im-3m         m-3m                 432  tab:orange
Properties: fit, cm, pq, nmatch
Scan unit: um

Note here that the austenite map is fully indexed, but not the ferrite one. I do refinement of orientations on each crystal map, and the number of orientations for each phase are the same as before, as expected. However when merging the refined crystal maps which contains not_indexed in the ferrite crystal map, the merged crystal map is completely missing ferrite.

refined_xmap_austenite = s.refine_orientation(
    xmap=xmap_austenite,
    detector=det,
    master_pattern=mp_austenite,
    energy=20,  # kV
    navigation_mask=None,
    trust_region=[1, 1, 1],
    # The following are default values
    method="minimize",
    method_kwargs=dict(method="Nelder-Mead", tol=1e-4),
    compute=True,
)

refined_xmap_ferrite = s.refine_orientation(
    xmap=xmap_ferrite,
    detector=det,
    master_pattern=mp_ferrite,
    energy=20,  # kV
    navigation_mask=None,
    trust_region=[1, 1, 1],
    # The following are default values
    method="minimize",
    method_kwargs=dict(method="Nelder-Mead", tol=1e-4),
    compute=True,
)
print(refined_xmap_austenite)
print(refined_xmap_ferrite)

refined_xmap = kp.indexing.merge_crystal_maps(
    [refined_xmap_austenite, refined_xmap_ferrite]
)
print("MERGED MAP", refined_xmap)

...
Phase   Orientations       Name  Space group  Point group  Proper point group     Color
    0  3600 (100.0%)  austenite        Fm-3m         m-3m                 432  tab:blue
Properties: scores, num_evals
Scan unit: um
Phase   Orientations         Name  Space group  Point group  Proper point group       Color
   -1    405 (11.2%)  not_indexed         None         None                None           w
    1   3195 (88.8%)      ferrite        Im-3m         m-3m                 432  tab:orange
Properties: scores, num_evals
Scan unit: um

MERGED MAP
Phase   Orientations         Name  Space group  Point group  Proper point group     Color
    0   1429 (39.7%)    austenite        Fm-3m         m-3m                 432  tab:blue
    1   2171 (60.3%)  not_indexed         None         None                None         w
Properties: scores, merged_scores
Scan unit: um

I have also seen that when both refined crystal maps contains not_indexed points, it results in a crystal map where all points are not_indexed. The same also happens without refining the crystal maps before merging (e.g. using scores_prop = "fit") if they contain not_indexed points.

Could this have been overlooked in the release of 0.8.4?

---

Thank you for reporting this issue, @Erlendos12. It seems like I introduced a regression in the fix in 0.8.4. Will have a look.

The same also happens without refining the crystal maps before merging (e.g. using scores_prop = "fit")

Note that a lower pattern (mis)fit is better, so you have to pass merge_crystal_maps(..., greater_is_better=False, scores_prop="fit") in this case.

[Erlendos12]

Note that a lower pattern (mis)fit is better, so you have to pass merge_crystal_maps(..., greater_is_better=False, scores_prop="fit") in this case.

Thanks, I forgot to add that.