GalSim noisy galaxy cases highly sensitive to noise_fraction in load_active_pixels!
#482
Assignees
Comments
|
Just an idea: when Jeff got rid of tiles, he also changed the way we
decided on active pixels. Previously, we would render a noiseless image at
its original parameters and select all the pixels that would be within some
proportion of the noise, he looked at the actual, noisy image, and selected
all the pixels that were brighter than some proportion of the noise. I
think the new way induces selection bias in the noise and is dependent on
having an accurate background estimate. The old way relies on having a
good guess of initial parameters, though.
…On Mon, Dec 19, 2016 at 11:44 AM, gostevehoward ***@***.***> wrote:
I noticed this past weekend that estimated brightness and HL radius are
way off for noisy galaxy cases in the GalSim benchmarks:
│ Row │ label │ field │ expected │ single_infer_actual │ joint_infer_actual │
├─────┼─────────────────────┼──────────────────────────────┼────────────┼─────────────────────┼────────────────────┤
│ 1 │ "galaxy_with_noise" │ "X center (world coords)" │ 0.00541833 │ 0.00542928 │ 0.00542929 │
│ 2 │ "galaxy_with_noise" │ "Y center (world coords)" │ 0.00514056 │ 0.00513782 │ 0.00513782 │
│ 3 │ "galaxy_with_noise" │ "Minor/major axis ratio" │ 0.4 │ 0.415016 │ 0.415019 │
│ 4 │ "galaxy_with_noise" │ "Angle (degrees)" │ 15 │ 14.7824 │ 14.7822 │
│ 5 │ "galaxy_with_noise" │ "Half-light radius (pixels)" │ 6.31313 │ 25.6438 │ 25.6095 │
│ 6 │ "galaxy_with_noise" │ "Brightness (nMgy)" │ 15 │ 37.1759 │ 37.158 │
│ 7 │ "galaxy_with_noise" │ "Probability of galaxy" │ 1 │ 0.995 │ 0.995 │
I sadly hadn't noticed this before, but it's present in some old
benchmark results <#435>
(click the pastebin link). I spent a while manually tracking this down (git
bisect wasn't possible because the benchmark harness was often broken) and
I think it started in 5878520
<5878520>
(the change to eliminate tiles).
Here's the inferred brightness and radius for various values of
noise_fraction, for the "galaxy_with_noise" case (true brightness 15,
true radius 6.31):
noise_fraction brightness HL radius
-0.9 14.9161 5.87814
-0.5 15.2943 6.13193
-0.4 15.6574 6.37313
-0.3 16.3363 6.83299
-0.2 17.5608 7.77823
-0.1 24.4099 14.7574
0 26.7117 16.9204
0.1 29.3006 19.2417
0.2 31.951 21.5993
0.5 37.1759 25.6438
Note that for "galaxy_with_all", a test case identical in every way except
with no poisson noise, the estimates are fine:
│ Row │ label │ field │ expected │ single_infer_actual │ joint_infer_actual │
├─────┼───────────────────┼──────────────────────────────┼────────────┼─────────────────────┼────────────────────┤
│ 1 │ "galaxy_with_all" │ "X center (world coords)" │ 0.00541833 │ 0.00541833 │ 0.00541833 │
│ 2 │ "galaxy_with_all" │ "Y center (world coords)" │ 0.00514056 │ 0.00514056 │ 0.00514056 │
│ 3 │ "galaxy_with_all" │ "Minor/major axis ratio" │ 0.4 │ 0.401808 │ 0.401808 │
│ 4 │ "galaxy_with_all" │ "Angle (degrees)" │ 15 │ 14.9999 │ 14.9999 │
│ 5 │ "galaxy_with_all" │ "Half-light radius (pixels)" │ 6.31313 │ 5.92177 │ 5.92177 │
│ 6 │ "galaxy_with_all" │ "Brightness (nMgy)" │ 15 │ 14.8672 │ 14.8672 │
│ 7 │ "galaxy_with_all" │ "Probability of galaxy" │ 1 │ 0.995 │ 0.995 │
I can try and dig further into what's going on but wanted to see if you
guys have any ideas off the top of your head @jeff-regier
<https://github.com/jeff-regier> @rgiordan <https://github.com/rgiordan>.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#482>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AESWPvbHnCd0OBj_vsx7eu0SxsF7Rd2_ks5rJt6LgaJpZM4LRHHU>
.
|
|
Yes, that's it. It adds selection bias unless |
|
OK, hopefully I'll be able to solve this along with #157 then. Why doesn't this affect SDSS results, or the stripe 82 benchmark? |
|
For what it's worth I noticed the "probability of galaxy" estimate in the simple_star test is also thrown off by this issue. |
gostevehoward
added a commit
to gostevehoward/Celeste.jl
that referenced
this issue
Dec 20, 2016
sadly this unit test was implemented in a completely broken way so it wasn't really checking parameter values. this fixes that. i've also added a few additional cases and commented out the two that are currently broken due to issue jeff-regier#482.
|
I think it does affect SDSS and stripe 82, but not that much, in part because an 8x8 grid around the center of the source always gets included no matter how bright the light source is. Also, the |
jeff-regier
pushed a commit
that referenced
this issue
Dec 21, 2016
sadly this unit test was implemented in a completely broken way so it wasn't really checking parameter values. this fixes that. i've also added a few additional cases and commented out the two that are currently broken due to issue #482.
This was referenced Jan 18, 2017
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I noticed this past weekend that estimated brightness and HL radius are way off for noisy galaxy cases in the GalSim benchmarks:
I sadly hadn't noticed this before, but it's present in some old benchmark results (click the pastebin link). I spent a while manually tracking this down (git bisect wasn't possible because the benchmark harness was often broken) and I think it started in 5878520 (the change to eliminate tiles).
Here's the inferred brightness and radius for various values of
noise_fraction, for the "galaxy_with_noise" case (true brightness 15, true radius 6.31):Note that for "galaxy_with_all", a test case identical in every way except with no poisson noise, the estimates are fine:
I can try and dig further into what's going on but wanted to see if you guys have any ideas off the top of your head @jeff-regier @rgiordan.
The text was updated successfully, but these errors were encountered: