Need per-image pixel mask #651
Comments
|
@graeme-winter points to a similar discussion about shadow masks in the HDRMX working group: I like the delineation of static mask vs. dynamic mask. Might be better than having it implicitly baked into the dimensions of pixel_mask... |
|
That sounds like a straight forward application of the (not exceedingly formalised) NeXus scan rules. Explicitly mentioning this in the documentation would be a small editorial change.
Cheers
Tobias
…________________________________
From: "Aaron S. Brewster" <notifications@github.com>
Sent: Tuesday, 26 March 2019 21:23
To: nexusformat/definitions
Cc: Subscribed
Subject: [nexusformat/definitions] Need per-image pixel mask (#651)
The NXmx specification of pixel_mask is:
pixel_mask[i, j, k]: (optional) NX_INT
This specifies a single mask applicable to all images in the dataset. This is a valid use case. I believe it could also be defined as:
pixel_mask[np, i, j, k]: (optional) NX_INT
i.e., it is valid for the user to provide a separate mask for each image. This is the case for the EuXFEL.
Therefore, the pixel mask always has a number of dimensions equal to the dimensionality of the data OR the dimensionality of the data -1, where the former is for a mask for each image and the later is a mask for the whole dataset.
Thoughts?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<#651>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AA0ReJ-MC_o8n3p2kgi7YbBUlCULcWqzks5vaoHGgaJpZM4cMXwX>.
|
|
To me this looks reasonable. If there is enough interest we can do
informal HDRMX dinner or lunch meetings at this summer's
ACA and ECM meetings and then do a formal HDRMX meeting at DLS in the fall
to discuss this and other additions to the NXmx
metadata to achieve a consensus on what should be optional, what should be
mandatory, and what should be part of the "Gold Standard".
Hopefully we will achieve a consensus that will make it easier to exchange,
process and archive datasets.
…On Tue, Mar 26, 2019 at 4:23 PM Aaron S. Brewster ***@***.***> wrote:
The NXmx specification of pixel_mask is:
pixel_mask[i, j, k]: (optional) NX_INT
This specifies a single mask applicable to all images in the dataset. This
is a valid use case. I believe it could also be defined as:
pixel_mask[np, i, j, k]: (optional) NX_INT
i.e., it is valid for the user to provide a separate mask for each image.
This is the case for the EuXFEL.
Therefore, the pixel mask always has a number of dimensions equal to the
dimensionality of the data OR the dimensionality of the data -1, where the
former is for a mask for each image and the later is a mask for the whole
dataset.
Thoughts?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#651>, or mute the
thread
<https://github.com/notifications/unsubscribe-auth/AEPiATYwS9ybLpjF6f5rOpZQ15e5yRzYks5vaoHGgaJpZM4cMXwX>
.
|
|
I can see an argument for dynamic and static masks explicitly, in that you could be using both: a backstop / bad pixel mask which is static and a second computed mask which handles, in this example, the shadow. The latter could be interpreted as a suggestion or guide, or best effort. Keeping these separate has value to me... |
|
That is kind of solved via the mask being a bit field. A pixel can carry multiple reasons for being excluded.
Tobias
Data Management and Software Centre
European Spallation Source, Lund, Sweden
…________________________________
From: Graeme Winter <notifications@github.com>
Sent: Wednesday, 27 March 2019 17:09
To: nexusformat/definitions
Cc: Tobias Richter; Comment
Subject: Re: [nexusformat/definitions] Need per-image pixel mask (#651)
I can see an argument for dynamic and static masks explicitly, in that you could be using both: a backstop / bad pixel mask which is static and a second computed mask which handles, in this example, the shadow. The latter could be interpreted as a suggestion or guide, or best effort. Keeping these separate has value to me...
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<#651 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AA0ReG-mhAMjNNIRsdNQ3uVCaTpqddu_ks5va5fRgaJpZM4cMXwX>.
|
|
Keeping them separate though lets us do things like hard linking to a never-changing known static mask (the pixel_mask) while also generating a runtime specific mask (the dynamic_mask). |
|
That's an option, sure. Linking the static mask saves 1 image, while you're writing a 1000 dynamic one's, though.
And it is neglecting aspects of NeXus that have been introduced by demand of the MX community. Of the community can ask for something different now, if it is worth it.
Tobias
…________________________________
From: "Aaron S. Brewster" <notifications@github.com>
Sent: Wednesday, 27 March 2019 17:25
To: nexusformat/definitions
Cc: Tobias Richter; Comment
Subject: Re: [nexusformat/definitions] Need per-image pixel mask (#651)
Keeping them separate though lets us do things like hard linking to a never-changing known static mask (the pixel_mask) while also generating a runtime specific mask (the dynamic_mask).
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<#651 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AA0ReOLwYfrKvon0tPLNXw7DheucaWKkks5va5uSgaJpZM4cMXwX>.
|
|
We use both static and dynamic masks in our single crystal diffuse scattering code. The argument for keeping them separate is that you may need to regenerate the dynamic mask as you refine your analysis. For example, we use dynamic masks to eliminate Compton scattering halos around Bragg peaks, generating them by an algorithm that calculates the Bragg peak positions. If we discover that our assumed space group was wrong, we might need to regenerate the mask, but we would not then have a record of the bad detector pixels. |
|
I think @zjttoefs is saying that record could persist through the use of bit fields. bit 8: user defined mask (e.g. around beamstop) vs bit 1: dead. Right? |
|
Correct!
…________________________________
From: "Aaron S. Brewster" <notifications@github.com>
Sent: Wednesday, 27 March 2019 18:05
To: nexusformat/definitions
Cc: Tobias Richter; Mention
Subject: Re: [nexusformat/definitions] Need per-image pixel mask (#651)
I think @zjttoefs<https://github.com/zjttoefs> is saying that record could persist through the use of bit fields. bit 8: user defined mask (e.g. around beamstop) vs bit 1: dead. Right?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#651 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AA0ReNYBvBCTn9EQLtJbJ3hEENfO1SHjks5va6TAgaJpZM4cMXwX>.
|
|
There are two distinct issues being raised here:
1. For any given mask, it may be the same for every frame or it may
differ frame to frame, in which case it should be indexed by frame.
2. There may be multiple types of masks, in which case the bit fields
may be needed, or, if those are inadequate we may need different mask names.
To see if item 2 requires new names -- please look at the bit definitions,
and propose whatever different types may be needed.
Regards,
Herbert
…On Wed, Mar 27, 2019 at 12:09 PM Graeme Winter ***@***.***> wrote:
I can see an argument for dynamic and static masks explicitly, in that you
could be using both: a backstop / bad pixel mask which is static and a
second computed mask which handles, in this example, the shadow. The latter
could be interpreted as a suggestion or guide, or best effort. Keeping
these separate has value to me...
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#651 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEPiAa6LfnRAWcwpIoo-bZ9JiIwt-6H2ks5va5fRgaJpZM4cMXwX>
.
|
|
Folks - in discussion with @phyy-nx yesterday I was reminded of another motivator for the two masks - the pixel mask as traditionally defined (i.e. 2D version) is available directly from the camera and is known at acquisition time - the shadow / dynamic mask may only be computed a postori or at the same time as the data acquisition, so could be linked into the master file in anticipation of the existence of the file. At the moment (see HDRMX link above) the calculation of the mask takes longer than acquiring the images... this is also advisory and could be refined so I do think there is a good argument for separate mask data sets. |
Also sync documentation between NXmx and NXdetector.
|
I've added a new pull request, #655, that uses a dynamic mask instead of making pixel_mask mult-image (as in #654). I'm happy with either #654 or #655 as solutions, but @graeme-winter has shown use cases where as separate mask is preferable. Other comments? |
|
Being dynamic is a separate issue from whether a given mask or other
per-pixel array needs to be separate from the current pixel_mask. --
Herbert
…On Fri, Mar 29, 2019 at 1:12 PM Aaron S. Brewster ***@***.***> wrote:
I've added a new pull request, #655
<#655>, that uses a
dynamic mask instead of making pixel_mask mult-image (as in #654
<#654>). I'm happy with
either #654 <#654> or #655
<#655> as solutions, but
@graeme-winter <https://github.com/graeme-winter> has shown use cases
where as separate mask is preferable. Other comments?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#651 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEPiAXV_BuCFsLDV7GW07OrU5i5dWp9Qks5vbkmOgaJpZM4cMXwX>
.
|
|
There is a specific use case where the dynamic mask will be computed at the same time as data are acquired, and referenced from the master file. In this case there will be a need for two pixel masks, one 2D one from the detector and a second which will appear later which is 3D. Avoids holding up acquisition for the shadow calculation... |
Allow multi-image masks. Resolves #651.
|
Leaving this open as there's still a desire to add a separate mask, dynamic_mask (#655). Probably best to discuss at next telco. |
Also sync documentation between NXmx and NXdetector.
The NXmx specification of pixel_mask is:
pixel_mask[i, j, k]: (optional) NX_INTThis specifies a single mask applicable to all images in the dataset. This is a valid use case. I believe it could also be defined as:
pixel_mask[np, i, j, k]: (optional) NX_INTi.e., it is valid for the user to provide a separate mask for each image. This is the case for the EuXFEL.
Therefore, the pixel mask always has a number of dimensions equal to the dimensionality of the data OR the dimensionality of the data -1, where the former is for a mask for each image and the later is a mask for the whole dataset.
Thoughts?
The text was updated successfully, but these errors were encountered: