Implementation of 'hdr-compute-blacklevel' that adapts to the Darkest Pixel in Scene and Spreads the Tone-Mapping Curve reaching Enhanced Contrasts

[stonix015]

EDIT:
See my second and last comment for a solution that is useable, but not "clean and nice" regarding the code implementation

Hey @haasn ,

I am thinking of a feature that measures the lowest spot in the scene and scales the scenes, so that this spot appears as a 0 nits // 0 through RGB values...
...Like the opposide of the hdr-compute-peak feature:
Setting the target-contrast in mpv raises the black level of the output format to the specs of the desired display by upscaling the scene from the src black-level to the display black-level.
In the actual version of libplacebo, the src black-level is always assumed to be 0 meaning that a constant upscaling is taking place for every movie and every scene in it.
But what if a scene only has pixels that are above 0?
In this scene, the black-level scaling of mpv would not only be insufficient, but also harming the image quality by a visible greyish look.
A black-level-detection could work as the opposide pendant to the hdr-peak-detection:
In this case, the scene would be less upscaled, adapting its specific darkest pixel to the desired display black-level.

My personal Case (Could be indirectly solved by using the feature)
Surprisingly but disappointingly lots of movies have black levels in the greyish area. Even in darkest scenes, a full black level will not be reached. Instead of, the scene looks washed out.
Using some gamma tricks to raise the details in the dark (e.g. to make movies watchable in projector setups) makes the problem worse: Now the "grey" level of the movie looks bright grey instead of dark grey.
But those settings are the only values with which some other dark movies can be enjoyed.

The result would be a contrastier image, which quality would be uniform and indepent from the image quality of the movie.
Very much greetings,
Lukas

[stonix015]

Hey,

I tried my best the last two days understanding the code.

My try was to search for the peak-detect feature and have a look how it works:

From my understanding, the function is measure_peak in colorspace.c:1027 .
The part of the function ...

for (int k = 0; k < SLICES; k++)
    next += data->frame_hist[k][i];
if (next < target_pixel) {
    sum = next;
    continue;
}

... looks for me like pixels with a raising brightness are counted, until there are so many pixels counted as the peak-percentile values suggests (target_pixel).

From that perspective i think, that data->frame_hist[k][i] stores the number/count of pixels sorted as a 2 dimensional array regarding a rising PQ brightness aquivalent, where data->frame_hist[0][0] means "0 nits" and data->frame_hist[SLICES][HIST_BINS] means "10'000nits".

From that perspective, i thought of a function like:

static float measure_min(const struct peak_buf_data *data)
{

    //Src min Black level
    for (int k = 0; k < SLICES; k++)
    {
        if (data->frame_hist[k][0] >= 1) {continue;}
    }
    const float ratio = k / SLICES;

    const float pq_low  = (float) HIST_PQ(-64) / PQ_MAX;
    const float pq_high       = (float) HIST_PQ(-63) / PQ_MAX;
    const float min_black = PL_MIX(pq_low, pq_high, ratio);

    return new_black;
}

where just the lowest bin (-64) is refered with its slices from 0 to SLICES.
I empirically determined, that this should reach for brightness values from 0 nits to around 0,1nits. That might do the trick for most movies with slightly greyish dark scenes where a total black_level of 0 is NOT reached. (Thats what i want to implement this feature for).

Now i have some troubles: I am missing the idea how to transfer the new_black value to the spline function // tone_mapping.c.
There are also some plot holes in my thoughts, e.g. why can HIST_PQ (x) be counted from x=-64 to x = 64, while data->frame_hist[0][0] is counted from zero -> Shouldn't the i be shifted of those value of -64 at colorspace.c:1063 :

// PQ luminance associated with count_low/high respectively
const float pq_low  = (float) HIST_PQ(i)     / PQ_MAX;
float pq_high       = (float) HIST_PQ(i + 1) / PQ_MAX;

I think that's all i can do at this place.
My head is crushed now. :/

Very many greetings!
Lukas

[stonix015]

Hey @haasn ... if you had interest in the feature, we can discuss this via discord. I made it to work now with a few workarounds:

In my comment above, i already stated an idea how to implement this thing. The only thing was missing to me was knowledge how to bring the value from colorspace.c to the tone_mapping.c.
Now i worked around that! I used the input_avg variable instead and declined the adaptive avg-feature of the spline filter.
(This is not worse, as i already worked with the linear tone-mapping before, there are multiple reasons why the spline tone-mapping does not sufficient on my very dark projector-setup)

Creating the new black level value (shaders/colorspace.c)
First, i had to change the histogram enum-function to create a histogram from 0nits to HDR completely:

enum {
    SLICES = 12,
    PQ_BITS     = 14,
    PQ_MAX      = (1 << PQ_BITS) - 1,
    HIST_BITS   = 7,
    HIST_BIAS   = 0, //Change here!
    // In the version before, the histogram was splitted into <100nits and >100nits. 
    // Everything under 100nits wasnt practically useable.
    HIST_BINS   = (1 << HIST_BITS) - HIST_BIAS,
#define HIST_PQ(bin) (((bin) + HIST_BIAS) << (PQ_BITS - HIST_BITS))
};

secondly, i had to create a measure_black function, which works similar to the measure_peak:

static float measure_black(const struct peak_buf_data *data, float percentile)
{
    unsigned next = 0;
    float slice_ratio = 0;
    for (int i = 0; i < HIST_BINS; i++) {
        for (int k = 0; k < SLICES; k++){
            next += data->frame_hist[k][i];
            if (next < 10000) //Counterpart to the hdr-peak-percentile, the first 10'000 pixels are ignored.
                slice_ratio = k; // will be updated until 10'000 pixels are reached. Yes! The if function will be entered every time until that.
            //most movies have slight noise, therefore this is senseful to clip them.
        }
        if (next < 1000)
            continue;
        const float ratio = slice_ratio / SLICES;
        const float pq_low  = (float) HIST_PQ(i) / PQ_MAX+0.000001f; // somehow, a total zero would throw the peak detection out.
        const float pq_high = (float) HIST_PQ(i+1) / PQ_MAX;
        return PL_MIX(pq_low, pq_high, ratio);
    }
}

Then, i had to change the avg-function in the update_peak_buf

float avg_pq, max_pq;
if (frame_wg_active) {
    avg_pq = measure_black(&data, params->percentile);
    max_pq = measure_peak(&data, params->percentile);
} else {
    // Solid black frame
    avg_pq = max_pq = PL_COLOR_HDR_BLACK;
}

Using the new black-level value (tone_mapping.c)
After that, i just had to use the value in the tone_mappin.c :

First, i want to limit the feature to the specified output_min level.
If the scene black-level appears to be above the display-spec, it should still appear as "grey":

const float new_input_min = PL_MIN(params->output_min, params->input_avg);

It should be said firstly, that a use in the linear-function (rescale_in and rescale_out) didnt have any effect. So i decided to transfer the spline function into a kind of linear function.

Therefore, i had to modify the st2094_pick_knee function:

[...]
// New Line here:
const float new_input_min = PL_MIN(params->output_min, params->input_avg);
// Modified line here:
const float src_min = pl_hdr_rescale(params->input_scaling,  PL_HDR_PQ, new_input_min);
const float src_max = pl_hdr_rescale(params->input_scaling,  PL_HDR_PQ, params->input_max);
// Modified line here:
const float src_avg = pl_hdr_rescale(params->input_scaling,  PL_HDR_PQ, new_input_min); 
const float dst_min = pl_hdr_rescale(params->output_scaling, PL_HDR_PQ, params->output_min);
const float dst_max = pl_hdr_rescale(params->output_scaling, PL_HDR_PQ, params->output_max);
[...]

And then, i had to modify the spline function:

[...]
// New Line here:
const float new_input_min = PL_MIN(params->output_min, params->input_avg);
// Modified Line here:
float slope = (dst_pivot - params->output_min) / (src_pivot - new_input_min);
float ratio = params->input_max / params->output_max - 1.0f;
ratio = fclampf(params->constants.slope_tuning * ratio,
                params->constants.slope_offset,
                1.0f + params->constants.slope_offset);
slope = powf(slope, (1.0f - params->constants.spline_contrast) * ratio);
// Normalize everything the pivot to make the math easier
// Modified Line here:
const float in_min = new_input_min - src_pivot;
const float in_max = params->input_max - src_pivot;
const float out_min = params->output_min - dst_pivot;
const float out_max = params->output_max - dst_pivot;
[...]

Thats all.
The spline function works similar to the linear function now, and the minimum black level in picture will be detected.
From practical use, i can say that LOTS OF scenes appear more contrasty than before.

[stonix015]

Some examples of the feature (it might help to show the screenshots in fullscreen and Alt+Tabbing them):
Prey with the feature enabled.

Prey with the old version of libplacebo

And also as a movie which already has 0-black levels:
Bodies Bodies Bodies with the feature enabled

And with the feature disabled:

You can clearly see the difference in Prey, which is very washed out. You can also see that this feature adapts to the scenes black level, as it doesnt do anything in Bodies Bodies Bodies.

[haasn]

Hey; sorry for not replying yet as I have been rather busy. I do find this feature worth consideration; and I can help implement it properly. (The underlying logic is straightforward as is just an inverse of the histogram peak)

I will get back to you on this

[stonix015]

No stress, if you want we can discord and talk about that (in german).

would be very cool!

:)
Lukas