rsx: Improved 24-bit format handling and shader refactoring

[kd-11]

Changes

1. Do not write to host channels that are disabled. This largely applies to RGBX formats where on PS3 the alpha channel is disabled. On PC we were writing to this non-existent channel which is wrong.
2. Refactor shader ROP stuff in preparation for programmable blending down the line. The full implementation will come in a later PR.
3. Correctly perform float->fixed conversion for 8-bit formats. During the vanillaware investigation it became clear that PS3 wraps negatives before clamping.

[JimScript]

Hey the hollowed out part of issue #10754 has been fixed by this:

It still overdraws both opaque and translucent parts like hundreds of times each, and there's still only one plant(several in some R:FOM maps) visible at a time, but this is still progress.

[JimScript]

Also this almost fixes my new #15063 issue, but on Windows it now displays only one plant at a time like TOD does on every platform.
Before:

After:

Edit: Wait I just noticed the stray pixels in the first image, so I guess #10754 and #15063 are related, but why does only ACIT work differently on Linux/Steam Deck? But since this fixes what was distictly broken I can close #15063 with this PR.

[JimScript]

Well I might as well show how this PR fixes ACIT on Steam Deck too.
Before:

After:

[kd-11]

Hey the hollowed out part of issue #10754 has been fixed by this: It still overdraws both opaque and translucent parts like hundreds of times each, and there's still only one plant(several in some R:FOM maps) visible at a time, but this is still progress.

I think I know what this issue is. I'll revisit some time in the near future.

[JimScript]

This PR is making every game with Shader Quality on "Low" crash on startup with it stating:
F {RSX Worker 5} SIG: Thread terminated due to fatal error: Failed to compile fragment shader (in file D:\a\1\s\rpcs3\Emu\RSX\VK\VKProgramPipeline.cpp:29[:25], in function compile)
Although it's :29[:5] on Linux.

[Megamouse]

Suggested change

	// Everything is fp32 on ouput channels
	// Everything is fp32 on output channels

[mwsimmonite]

Hi - I’ve just finished Resistance 2 (another Insomniac game) on the steam deck and had the exact same issue with a lot of the grass there being bright green and luminescent (lots of it in the prologue level).

By eye, it looks like it could be the same asset. I tried various settings that were suggested for other Insomniac games in case that may do something but didn’t have any luck.