Is it still make sense to align structs for AVX ?

[gotzmann]

It seems, Q4 / Q8 weights do not aligned within memory bounds / cache line size.

Like here 4bytes + 16bytes:

#define QK4_0 32
typedef struct {
    float   d;          // delta
    uint8_t qs[QK4_0 / 2];  // nibbles / quants
} block_q4_0;

I used to think that it better to align for 32/64 bytes for faster AVX2 / AVX512 (and there special ops to work with aligned vectors).

So not sure maybe modern CPUs do handle mis-aligned data easily or maybe we loose some performance here?

[prusnak]

The struct is not packed, so its size is 20, not 18.
In other words, each field that has a length not divisible by 4 will get padded so that it becomes aligned to 4 bytes.
As a result, each struct is located at an address aligned to 4-byte boundaries.

Having said all that, I don't know much about AVX, so I'm not sure how aligning to 32/64 bytes would help.
However, if you were to align a 20-byte struct to 32 bytes, wouldn't the memory requirements increase to 160%?

[SlyEcho]

float is 4 bytes.

[gotzmann]

Having said all that, I don't know much about AVX, so I'm not sure how aligning to 32/64 bytes would help. However, if you were to align a 20-byte struct to 32 bytes, wouldn't the memory requirements increase to 160%?

I mean, we can store deltas in some other memory buffer to allow faster aligned AVX operations on the quants.

I'm not an expert in AVX too, so maybe if there pattern of sequential access (like matmul pipeline getting one struct after another), there no cache penalty at all for modern CPUs.

There definitely was a penalty for older CPUs, thus different AVX instructions for aligned / unaligned vectors.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.