API Caching

[adriancollier]

We are currently utilising the API resource for generating pins for the Global Project maps and project page maps.

The process to collect all the data required for the maps take some time and has caused server issues during the previous release.

We have implemented a part-solution to collect the points on the map in 100 intervals, allowing the server to respond more effectively. But the process still takes too long to completely populate.

We need to investigate and implement a Caching solution for the API data returned for the maps. This solution should collect and store the data set being called for, to allow faster access to the web server when requested.

• The API Call being used for the maps should be run regularly
• The cached data should be stored and made available to Apache.
• Apache should retrieve the cached data on request
• In the event that the data is already stale, a new call should be made to the API
• The new data should replace the existing cached resource
• Subsequent calls should collect the latest cached data

In the event that this solution is effective then it should be ported also to the homepage to replace the image that is currently present.

Any solution that has been found that covers these requirements should be presented as following:

• Outline of solution
• Process flow for map users
• Additional libraries/resources to be used
• Notable dependencies
• Drawbacks or issues
• Estimation of implementation time
• Test case

[adriancollier]

API Caching solution for Map calls - to be Code-Checked by @peeb

[adriancollier]

@zzgvh - did we implement the Caching solution for the API?

If so, can you confirm it's working correctly?

[adriancollier]

@zzgvh am I correct that what was implemented for this previously has been removed due to issues?

[zzgvh]

Caching is currently running on live using a customized version of django-cacheback. However even if it's very elegant in a way, the coding needed for it to integrate into tastypie is ugly and brittle since I had to rewrite a couple of the internal methods. It has already broken once due to internal changes in tastypie. So I'd like to back out of this experiment and go back to a memcached-based solution for the short term, and look at using varnish (for the whole site) in the longer term.

[adriancollier]

@zzgvh could you give a brief update on what we still need to do, and at that point we need to decide which Milestone/Release to put this into.

[zzgvh]

Having dug deeper in the Tastypie code I find that the caching for the maps resources works as intended. Unfortunately that means that only individual objets are cached, that is all Tastypie provides, so the calls we're making to populate the global maps are never cached 😞

I suggest we close this for now and implement Varnish ASAP. The live server seems to hold up fairly well as it is.

[adriancollier]

Agreed - I'll create a new Varnish issue based on the above requirements.