Automatic normalization bounds produce sub-optimal images

[atruskie]

Expected behaviour

False colour spectrograms, rendered for the zooming and fixed-scale visualisations should be similar or identical.

Actual behaviour

Due to the fact that the IndexPropertiesConfig.Zooming.yml config file uses automatic normalization bounds and the IndexPropertiesConfig.yml does not, the images rendered for zooming look different from those done by normal generation. In particular, they generally look very red,/green and at lower scales, very green.

If you look on Ecosounds and correlate the data there with images we produce offline, you'll find plenty of examples where the colour profiles are sub-optimal.

Here is one demo: This data was used in the workshop and is attributed to Liz Znidersic. It is from the Tasmania Mez site, on the 2015-11-14.

Here is a manually composited image of the zooming tiles:

And the standard rendering (through Towsey.Acoustic):

Steps to reproduce the behaviour

• What command did you use?
  • The standard commands for generating indices (audio2csv and Towsey.Acoustic.yml) and generating zooming images (DrawZoomingSpectrograms)
• What config file did you use?
  • The standard configs
  • A local copy here: \\Sef-bigdata-10\d$\tasmania_mez\Towsey.Acoustic.Zooming.yml
• Where can we find the source data used?
  • \\Sef-bigdata-10\d$\tasmania_mez has a copy of the zooming indices and tiles

Any other details

This issue affects all images used for zooming spectrograms on the Ecosounds website. The changes made to normalization bounds should be tested for multiple zoom levels, and multiple sites.

[atruskie]

@towsey did some work on this in ad1f0f8

[atruskie]

@towsey - I remember what we were talking about now.

We were concerned that the histograms output for the zooming indices were very flat (and thus not useful for automated normalisation).

• you were going to investigate why the BGN histogram had a weird gap in its distribution

• we were going to check the histogram functions for soundness given the very large number of values involved (possible int32 overflow?)

• Phil and I were discussing this:

  • We may need to calculate the distributions separately for each zoom level
  • Are confidence intervals useful here?
  • If you thought it was useful, we maybe should calculate index values and distributions for a significant (and random) subset of our entire audio collection (periodically) and use those fixed values in all zooming rendering. This would be an example of a offline analysis

[towsey]

Hi Anthony, Thanks for reminding me about this. Yes, it also occurred to me that we might have to calculate bounds for each ICD level separately. And yes these could be calculated based on an off-line sample of a representative selection of samples. I need to think about this some more. It would take a bit of experiemnting. Which I would like to do some time! By taking a sample, we would avoid the integer overflow problem - if that is indedd one of the problems. Cheers, Michael

[atruskie]

More work is required for this.

We need to see

• the 24-hour image generated from the zooming analysis
• and the new distribution image

Keep in mind, this issue when closed will trigger a solid 6-9 months of computation as we update all the images on Ecosounds.

[atruskie]

Closed in #233