Supporting max item size for the memcached client (query-frontend)

[craigfurman]

Is your proposal related to a problem?

I'm seeing error messages from the query-frontend relating to memcached insertions failing:

{
  "caller": "memcached.go:235",
  "err": "server=10.252.1.8:11211: memcache: unexpected response line from \"set\": \"SERVER_ERROR object too large for cache\r\n\"",
  "level": "error",
  "msg": "failed to put to memcached",
  "name": "",
  "ts": "2021-03-03T06:16:55.594447923Z"
}

We configure max item sizes of 16MB for both the query-range and labels caches, and set the corresponding max item size on the memcached servers.

This appears to be due to a known incomplete feature, as I also see log messages "MaxItemSize is not yet supported by the memcached client" that appear to originate from #3114. The code has a TODO above it to add this feature:

thanos/pkg/queryfrontend/config.go

Lines 101 to 104 in 513707d

	// TODO(krasi) Add support for it in the cortex module.
	if config.Memcached.MaxItemSize > 0 {
	level.Warn(logger).Log("message", "MaxItemSize is not yet supported by the memcached client")
	}

Describe the solution you'd like

For the memcached clients used by query-frontend to support max item size.

Describe alternatives you've considered

n/a

Additional context

I don't see similar error messages from the store, for which we've deployed index and chunk memcached caches. I've only glanced at the code related to this, but the store's client doesn't appear to have this problem. We're using the default 1MB max item size there.

[craigfurman]

I'm happy to try my hand at a PR for this, but figured it was worth raising first, since this appears to be an artifact of a code consolidation effort between thanos and cortex? I'm a total outsider though, and might have got the wrong end of the stick 😉

[yeya24]

Just opened a pr on Cortex side cortexproject/cortex#3929.
If that is merged we can support it in Thanos easily.