render: Preliminary support for paletted textures

[ccawley2011]

This extends the limited support for paletted textures in the DirectFB render driver to allow support to be added to other platforms, and to allow the palette to be changed after the texture has been created, which is useful in older games.

This is currently implemented in the Direct3D 9, OpenGL and OpenGL ES 2 drivers using shaders, but these currently don't work correctly when bilinear filtering is enabled. Fixing it would require using texture targets, but I'm not sure on the best way to implement this. A software-based fallback is also available, which doesn't have this problem.

testoverlay2 has been modified to make use of this feature, and another test program has been added to test palette rotation effects that may occur without updating the image data. I've also opened sergiou87/open-supaplex#41 to demonstrate this in a real-world application.

[icculus]

I really don't want to get into paletted textures in the 2D API, fwiw; if there isn't a trivial way to upload them to a non-paletted format, we should add that as a generic thing, and let people that need this use a shader directly in the upcoming GPU API.

[ihhub]

I really don't want to get into paletted textures in the 2D API, fwiw; if there isn't a trivial way to upload them to a non-paletted format, we should add that as a generic thing, and let people that need this use a shader directly in the upcoming GPU API.

Hi @icculus , SDL 1 supports paletted textures and for some projects like fheroes2 it is important to have such ability. Right now we do a hack to convert from 8-bit images to 32-bit images which leads to 20% performance drop. This is a very significant reduction which affects not only the gameplay but eats more resources and battery on portable devices.

If you think about an easier way to add support for such feature then please explain it.

[ihhub]

Hi @ccawley2011, I roughly took a look at your changes and put several general comments here. Could you please take a look when you have time?

[ihhub]

We should update this comment due to new releases being online.

[ihhub]

Do we need this commented code to be present? If not then we can do:

if (format != SDL_PIXELFORMAT_INDEX8) {
    SDL_SetError("Unsupported palette format");
    return NULL;
}

[ccawley2011]

I would like to support 1-bit and 4-bit formats as well, I'm just not sure how locking should behave in these cases.

[ihhub]

Since this pull request is stuck for multiple months a good approach is to implement only for 1 format and add TODO comments in relevant places for other format. On top of this after this pull request is merged it would be wise to create an issue asking to add these missing formats.

[ihhub]

I'm not sure about this project's way of coding but a good practice is to use variable declaration with its initialization:

SDL_SW_PaletteTexture *swdata = (SDL_SW_PaletteTexture *) SDL_calloc(1, sizeof(*swdata));

This way we reduce the chance of uninitialized pointers being used in the code.

[ccawley2011]

Mixed declarations and code is only possible in C99, while SDL uses C89.

[ihhub]

We should initialize row within the loop as it would be easier to read and at the same time would be the same performance since this variable is still on stack.

[ccawley2011]

This is only possible in C99.

[ihhub]

Interesting. Is there a restriction not to use C99?

[ihhub]

This place and other places in the code: please align pointers to have the same code style. For example, here we align them in the middle while in the include/SDL_render.h we align to left.

[ihhub]

All these functions miss Doxygen comments and also please add empty lines between them.

[ccawley2011]

Doxygen comments don't generally seem to be used in internal headers such as this one.

[ihhub]

Despite of fact this line is present in other files this is an editor specific thing which shouldn't exist in the generic source code. This is a developer responsibility to adjust editors per their needs.

[ihhub]

Also it would be important to know the target version of this change, whether this PR going to be a part of the source code or there are concerns about integrating it. If there are concerns what are they besides the fact of possible complexity of the code?

[ccawley2011]

The main blocker for this PR is the fact that bilinear filtering still doesn't work with backends that implement this using shaders. Once that's done it should be ready, but I'm not sure that I'm capable of doing this myself. If anyone else is interested, then feel free to take over this PR.

If this is ever finished, I would quite like to get this into SDL2 as well as SDL3, but I don't know if new API calls are still being accepted there.

[ihhub]

Hi @ccawley2011 , bilinear interpolation cannot be done on indexed images. The only way to do it is to get values in RGB space and then find the nearest value in the palette to the result. If we think that RGB is a 3D space then it's a task of finding the smallest distance between 2 dots: the result value and existing palette value. The complexity of this algorithm is huge and bigger than to O(M*N). It would definitely be very slow.

[glebm]

I've implemented bilinear downscaling for indexed images in the past here. That file has 2 functions, one does 32-bit scaling to arbitrary size, the other one downscales 8-bit indexed images by 50%.

Downscaling of 8-bit indexed images is achieved by passing a Uint8 paletteBlendingTable[256][256], created on palette changes, which is very slow (but we do it infrequently in DevilutionX). The table is created here.

[nlapinski]

You can handle the filtering in the shader method with some number of weighted lookups on the texture then blend. (I'm not sure this works for scaling down?).

#define SHADER_FRAGMENT_CUBIC                                   \
"varying vec4 v_color;\n"                                       \
"varying vec2 v_texCoord;\n"                                    \
"uniform sampler2D tex0;\n"                                     \
"\n"                                                            \
"const float pal[768] = float[768](\n"                          \
PAL_0                                                           \
");\n"                                                          \
"float cubic(float x) {\n"                                                                  \
"    x = abs(x);\n"                                                                         \
"    float x2 = x * x;\n"                                                                   \
"    float x3 = x2 * x;\n"                                                                  \
"    if (x <= 1.0) {\n"                                                                     \
"        return 1.0 - 2.0 * x2 + x3;\n"                                                     \
"    } else if (x < 2.0) {\n"                                                               \
"        return 4.0 - 8.0 * x + 5.0 * x2 - x3;\n"                                           \
"    } else {\n"                                                                            \
"        return 0.0;\n"                                                                     \
"    }\n"                                                                                   \
"}\n"                                                                                       \
"vec4 textureBicubic(sampler2D sampler, vec2 uv) {\n"                                       \
"    vec2 texSize = textureSize(sampler, 0);\n"                                             \
"    vec2 invTexSize = 1.0 / texSize;\n"                                                    \
"    vec2 f = fract(uv * texSize - 0.5);\n"                                                 \
"    vec2 p = floor(uv * texSize - 0.5);\n"                                                 \
"    vec4 sum = vec4(0.0);\n"                                                               \
"    for (int i = -1; i <= 2; ++i) {\n"                                                     \
"        for (int j = -1; j <= 2; ++j) {\n"                                                 \
"            vec2 index = p + vec2(i, j);\n"                                                \
"            float weight = cubic(f.x - float(i)) * cubic(f.y - float(j));\n"               \
"            float idx = texture2D(sampler, (index + 0.5) * invTexSize).r * 255.0;\n"       \
"            int colorIdx = int(idx) * 3;\n"                                                \
"            float alpha = colorIdx <= 2.0 ? 0.0 : 1.0;\n"                                  \
"            vec4 color = vec4(pal[colorIdx], pal[colorIdx+1], pal[colorIdx+2], alpha);\n"  \
"            sum += color * weight;\n"                                                      \
"        }\n"                                                                               \
"    }\n"                                                                                   \
"    return sum;\n"                                                                         \
"}\n"                                                                                       \
"void main() {\n"                                                                           \
"    vec4 color = textureBicubic(tex0, v_texCoord);\n"                                      \
"    gl_FragColor = color * v_color;\n"                                                     \
"}\n"

For sdl2 I think this is maybe too much to add since it adds a lot to each rendering backend implementation, new indexed texture formats and specific shaders. Any use of this would likely need some shader customization anyway ontop - which SDL2 is not built for.

I would rather see a generic data/byte type of image that can bind a generic/custom/utility shader to , something that could also be useful for tonemapping, luts etc. (thinking sdl3)

[slouken]

This hasn't been updated in a while and we're cleaning house for SDL 3.0. Please feel free to reopen this if you'd like to clean it up and get it in!