Remove queryset caching support

[jeremystretch]

Proposed Changes

Remove the support for query caching that was introduced in v2.6 under #2647. This would remove django-cacheops as a dependency and eliminate the REDS['caching'] and CACHE_TIMEOUT configuration parameters.

Justification

Since its implementation, we have been constantly chasing numerous bugs attributable to limitations in the caching engine. I have addressed some of these issues within django-cacheops itself, however we continue to run into problems, particularly where complex workflows are involved.

In my opinion, it does not make sense to continue burning time chasing these bugs to support a feature with relatively little benefit. I would prefer to optimize NetBox itself where necessary to meet performance expectations rather than trying to keep the caching layer intact. I've opened this issue for discussion to see what the community thinks.

[jhujhiti]

Are there any quantitative numbers on how much the removal of caching would hurt specific API endpoints, or a way to disable the cache in an existing 2.11 environment to evaluate what the impact of this change would be? As a user, I'm certainly in favor of reducing the complexity of the stack, but my deployment has a number of identical queries run by external monitoring repeatedly and losing the cache there could put a lot more easily-cached load on the database. (Although I do suspect Redis isn't making much difference, I don't have a way to prove it.)

[jeremystretch]

@jhujhiti You can test on your own deployment by toggling the caching feature on and off. Setting CACHE_TIMEOUT = 0 in configuration.py and restarting the NetBox service will disable caching.

[jhujhiti]

Great! No discernible impact on the 95th or 50th percentile latency on the API endpoints I've been monitoring. Consider this a vote in favor.

[dreng]

Very low impact in our environment. Not noticeable in every day use. I'd say: kick it!

[maximumG]

As mentioned in issue #6651. The check_release process will be impacted by cache removal.

However check_release can easily rely on RQ result caching (result_ttl) and RQ registries to achieve the same result as the current implementation.

When browsing the home page it is possible to check in the RQ registries for the latest check_release job and its corresponding result. If the result has expired (> result_ttl time), we can queue a new check_release job.

I think that setting the check_release job with a result_ttl to 86400s is acceptable to check for an update every day.

[candlerb]

Just to add a +1 here. Databases like postgres have very good internal cache mechanisms, and I never saw any benefit in adding a layer on top.

Caching of HTML rendering might have some justification, but only for pages which are viewed by multiple people and fetched repeatedly - which would probably only be the front page, and even then it varies by the user's permissions.

[jeremystretch]

I should clarify that this is just removing queryset caching. We will still maintain a Redis-based cache for generic caching functions (e.g. checking for the latest NetBox release). We will likely replace django-cacheops with django-redis, which includes support for Redis Sentinel.