Compressing with small window size

[nlimper]

For a zlib compressed file that needs to be decompressed by an MCU with very limited memory, I want to use a window size of only 4096. I started by defining TDEFL_LESS_MEMORY, and changing TDEFL_LZ_DICT_SIZE to 4096, which leaded to corrupted zlib files. After some more investigation, it appeared I could lower TDEFL_LZ_DICT_SIZE to 16384 and 8192, but anything below would corrupt the output file.

Eventually I managed to fix it, because I found some hardcoded values in the source of tdefl_compress_normal:
8U * 1024U in https://github.com/richgel999/miniz/blob/master/miniz_tdef.c#L1185
and
31 * 1024 in https://github.com/richgel999/miniz/blob/master/miniz_tdef.c#L1234
Changing both of the hardcoded values to 4096 solved the problem, and, in combination with the right TDEFL_LZ_DICT_SIZE, leaded to valid zlib files with window size 4096.

As I'm not exactly know what I'm messing with ;-) , I didn't want to make this in to a PR. Ideally, it probably should use the window_bits parameter in int mz_inflateInit2(mz_streamp pStream, int window_bits) (which is restricted to 15 and -15 at the moment, and only used to check if the value is negative or positive).

[KHigsUni]

Did this end up working well for you? I have max 64kb of ram to use and currently the tdefl takes about 350kb which I definitely don't have. I am new to compression, what does "window size" mean?

[nlimper]

It's working well with the changes I proposed, but you have to know what you're doing. To know what window size means in respect to gzip, I'm sure chatGPT or google can tell you, as they are far better in explaining than me ;-)

[KHigsUni]

Oh I see. What it can look at to compress it. I am looking to compress 1400 bytes to about 500 so 4095 would be perfect.

I will mess around with it and see if I can figure it out. How much memory did you need for tdefl_compressor in the end?

Thanks for the quick answer!

[KHigsUni]

Managed to get the compressor struct down to only 41kb. Good enough for my micro controller's RAM.

Just had to modify the enums. I was able to successfully compress a 1400 byte array to about 500. Not sure what the limit is with this small but still very good compression.

#if TDEFL_LESS_MEMORY
enum
{
	TDEFL_MAX_HUFF_TABLES = 3,
	TDEFL_MAX_HUFF_SYMBOLS_0 = 288,
	TDEFL_MAX_HUFF_SYMBOLS_1 = 32,
	TDEFL_MAX_HUFF_SYMBOLS_2 = 19,
	TDEFL_LZ_DICT_SIZE = 4096,
	TDEFL_LZ_DICT_SIZE_MASK = TDEFL_LZ_DICT_SIZE - 1,
	TDEFL_MIN_MATCH_LEN = 3,
	TDEFL_MAX_MATCH_LEN = 258
};

enum
{
	TDEFL_LZ_CODE_BUF_SIZE = 7 * 1024,
	TDEFL_OUT_BUF_SIZE = (TDEFL_LZ_CODE_BUF_SIZE * 13) / 10,
	TDEFL_MAX_HUFF_SYMBOLS = 288,
	TDEFL_LZ_HASH_BITS = 12,
	TDEFL_LEVEL1_HASH_SIZE_MASK = 4095,
	TDEFL_LZ_HASH_SHIFT = (TDEFL_LZ_HASH_BITS + 2) / 3,
	TDEFL_LZ_HASH_SIZE = 1 << TDEFL_LZ_HASH_BITS
};
#else