10.0.x: OpenReadHead failed for cachekey <key> : vector inconsistency

[bneradt]

When updating one of our internal 9.2.x ATS instances to 10.0.x, I observe the following cachekey notice messages frequently:

[Jun  9 19:12:43.041] [ET_NET 25] NOTE: OpenReadHead failed for cachekey D0211905 : vector inconsistency - unmarshalled -1 expecting 4536 in 22712 (base=80, ver=24:2) - vector n=0 size=0first alt=-1[bogus]
[Jun  9 19:13:29.716] [ET_NET 26] NOTE: OpenReadHead failed for cachekey B3C93263 : vector inconsistency - unmarshalled -1 expecting 24096 in 24172 (base=80, ver=24:2) - vector n=0 size=0first alt=-1[bogus]
[Jun  9 19:14:49.779] [ET_NET 29] NOTE: OpenReadHead failed for cachekey 87F9D92A : vector inconsistency - unmarshalled -1 expecting 4224 in 4307 (base=80, ver=24:2) - vector n=0 size=0first alt=-1[bogus]

I'm filing an issue for this because I do not see these messages in our 9.2.x instances but I do after upgrading to 10.0.x. As I understand it, there should be no cache compatibility issues with the cache when crossing from 9.x to 10.x.

[cheluskin]

The same problem. Clearing the cache doesn't fix it.

[cmcfarlen]

The -1 return is likely coming from HTTPInfo::get_handle(char *buf, int len)

[bneradt]

Bisecting commits on our prod system, I found the commit that introduces this vector inconsistency behavior:
#9556

[bneradt]

@masaori335 : I wonder whether the issue comes from swapping the order of pinned element here:
https://github.com/apache/trafficserver/pull/9556/files#diff-16cd59c47246f90109883f36526055d937b92630ed6e0cdba4664f2328bf1909L301

Do you recall why we switched the order of those elements?

[bneradt]

@masaori335 : I wonder whether the issue comes from swapping the order of pinned element here: https://github.com/apache/trafficserver/pull/9556/files#diff-16cd59c47246f90109883f36526055d937b92630ed6e0cdba4664f2328bf1909L301

Do you recall why we switched the order of those elements?

I tested with that order restored, and the box still has cache inconsistency issues. I suppose the issue is rather the change of the type.

[bneradt]

The type change from uint32_t to time_t changes the type size from 32 bits to 64 bits on 64 bit systems. It makes sense that would change the stored doc structure.

[bneradt]

I'm not sure what the right direction is here. Thinking about @masaori335 's change further, it is reasonable to avoid downcasting the time_t to uint32_t. Maybe we should just document that the cache is inconsistent in the upgrade from ATS 9 to ATS 10? Clearing the cache resolves the issue. I'll solicit feedback about this so we can figure out whether a cache inconsistency across the major versions is acceptible.

[masaori335]

Let's revert the change made by #9556. I didn't noticed it breaks cache compatibility when I fixed the coverity issue. IMO, using uint32_t is enough for cache and cache compatibility is more important.