A potential fix to mp3 and shader caching bug. Needs engine changes.

[Lizard-Of-Oz]

Description

There is a way to avoid mp3 caching bug and custom shaders caching bug: just don't compress mp3s, wavs and shaders specifically.
Bsp-bzip is compressed on per-file basis, meaning you can have everything except mp3s, wavs and shaders being compressed. It's the compression that causes the caching issues.

While this is a "dirty" workaround, it works and it's trivial to implement in the bsp packing routine:
If the file is located in `shaders/` folder OR has `*.mp3` extension OR has `*.wav` extension, don't compress the file.

Since mp3s and wavs don't compress well anyway, and shaders are tiny, the map file size won't meaningfully increase as the result of this change. It's better to avoid compressing wavs too because ADPCM wavs are useful, but are affected by the mp3 caching bug too.

To clarify potential confusion: it's okay to pack mp3s, wavs and shaders into a bsp (as in, to have them be inside a bsp), but it's not okay to apply bzip compression to those files.

Note that TF2 has two separate places responsible for compressing maps, and the file filter needs to be applied to both:

• bspzip.exe - used when compiling a map. Its code is not included in the SDK.
• bsppack.dll - used when uploading to the workshop (is also used by bsp_repack command).
  • It appears that this line is the corresponding code location. You can simply change packfileCompression value depending on the file path.
     

The reason it's important to implement in Live TF2 is mapping accessibility.

People want to make maps for custom gamemodes, such as VSH, ZI, VIP, etc, but the process of avoiding the bug is extremely convoluted and manual, thus, prone to user error. As the result, community-made maps for those gamemodes are prone to the caching issue, which is semi-permanent and "infects" other maps of their respective gamemode, even maps with the fix applied.
If this filter would be built into the stock tools, the problem will be solved.

In other words, "just don't compress (just don't repack)" works only as long as everyone follows it, and the only realistic way to achieve this is to make the change to the official tools. (Or to fix the actual underlying causes, but note that shaders are corrupted for a different reason than mp3s and wavs, and both sources of corruption are located in parts of the engine not available in the SDK code)

This workaround has been tested on a few workshop maps and it appears to work, but, as mentioned, without the filter being built into the stock tools it can't be used by the wider mapping community.

While modifying the tools won't retroactively fix pre-existing maps, I trust the community developers to release an update with changed file paths to mp3s and wavs to not collide with pre-existing maps. Any map released after will no longer be affected by the caching bugs.

Why not have the exceptions list as a compile parameter?

As mentioned above, it's important to have these exceptions applied when you upload a map on the workshop or compile a map with default settings. Having to modify compile parameters on a per-map basis will defeat the point, as people will simply fail to do it, and there is no fail check to stop or correct them.
Also, it's more moving parts when compared to a few simple checks proposed above.

Wait, Shaders?

It has recently been discovered that the Source Engine supports loading custom pixel shaders, via a completely undocumented shader named screenspace_general.

These custom shaders work even when packed into the BSP, including the Workshop. Here's a full explanation and tutorial by Ficool2.

This feature has already found use on several Workshop maps to great success, such as TF2Ware and Mobster VIP gamemode.

However, if a shader is compressed, it might fail to load with the following error:
BZIP Error (-5) decompressing shaderfailed to create shader
but only if the shader was already loaded into memory, aka, when you're revisiting the map for the second time.

Fortunately, if we "simply don't compress the shader", then it works just fine.

If you'd like to fix the shader bug directly, you can use this older version of a Mobster VIP map to test the bug with: https://tf2maps.net/downloads/brooklynn-mobster-vip.19180/version/57402/download
To reproduce the bug, you need to load the map, change to a different map and load this map again. If you see a black square on your screen once you've spawned, the bug has been reproduced.

[Lizard-Of-Oz]

Added some clarifying notes:

1. To clarify potential confusion: it's okay to pack mp3s and shaders into a bsp (as in, to have them be inside a bsp), but it's not okay to apply bzip compression to those files.

2. "Just don't compress (just don't repack)" works only as long as everyone follows it, and the only realistic way to achieve this is to make the change to the official tools. (Or to fix the actual underlying causes, but note that shaders are corrupted for a different reason than mp3s, and both sources of corruption are located in parts of the engine not available in the SDK code)

3. While modifying the tools won't retroactively fix pre-existing maps, I trust the community developers to release an update with changed file paths to mp3s to not collide with pre-existing maps. Any map released after will no longer be affected by the caching bugs.

4. Do not compress *.wav files either - they are also hardly compressible, but ADPCM wavs are also affected by the mp3 caching bug. Mappers use ADPCM wavs to have looped music or voice lines with lipsync (was attempted with VSH but then rolled back) while having lower size.

5. Why not have the exceptions list as a compile parameter? - as mentioned above, it's important to have these exceptions applied when you upload a map on the workshop or compile a map with default settings. Having to modify compile parameters on a per-map basis will defeat the point, as people will simply fail to do it, and there is no fail check to stop or correct them.
   Also, it's more moving parts when compared to a few simple checks proposed above.

6. If you'd like to fix the shader bug directly, you can use this older version of a Mobster VIP map to test the bug with: https://tf2maps.net/downloads/brooklynn-mobster-vip.19180/version/57402/download
   To reproduce the bug, you need to load the map, change to a different map and load this map again. If you see a black square on your screen once you've spawned, the bug has been reproduced.

[Lizard-Of-Oz]

And while we're at it, *.sw.vtx model files are not used by at least TF2, and probably by any current Source 1 game.
If either studiomdl.exe would simply not compile them or if the packing process wouldn't pack them at all, the collective storage space saved from this wouldn't be nothing.