Implement InMemoryCache garbage collection and eviction. #5310

benjamn · 2019-09-12T15:26:14Z

This PR adds a handful of powerful new methods to the official InMemoryCache class:

The cache.gc() method removes any unreachable objects from the normalized cache, using a tracing garbage collection strategy (as opposed to reference counting). Reachability is determined starting from a set of root entity IDs, which includes any IDs that have been directly written to the cache—including the synthetic ROOT_QUERY ID, which is the default ID used when no specific ID is provided. From this root set of reachable IDs, garbage collection traces all child references (computed lazily and cached aggressively) until the entire reachable reference graph has been visited. Any remaining unvisited IDs are then removed from the cache, and the cache.gc() function returns a list of those removed ID strings.

As of this writing, the cache.gc() method is never called automatically, but must be invoked by the application developer. Although garbage collection becomes more and more efficient over time, the first garbage collection takes time proportional to the number of entities in your cache, so you might want to run it during an idle period after page load.

The cache.retain(id) method can be called to treat specific entities as immediately reachable "root" entities, which protects them (and all their transitive child references) from garbage collection. Note that this method is called automatically for the IDs written using writeQuery or writeFragment. Our hope is that this method will not be needed very often, though it can be essential when you don't want certain data to be removed.
The cache.release(id) method undoes the effect of cache.retain(id), allowing the given ID to be garbage collected once it becomes otherwise unreachable. Note that cache.release(id) must be called at least as many times as cache.retain(id) was called for a given id. Both methods return the new retainment count. This retainment count is not a "reference count" in the usual sense, since most entities will never need to be retained or released, and we are not using a reference counting garbage collection strategy. In particular, cache.release(id) does not automatically trigger garbage collection when the retainment count falls back to zero.
The cache.evict(id) method forcibly removes the given rootId from the cache, but does not immediately trigger garbage collection, since you might want to evict multiple IDs before triggering a single garbage collection. With that said, removing an entity from the cache can have significant consequences for the reachability of other entities, so eviction should usually be followed by garbage collection.

Closes #3965, #4681, #4917, #4916

Part of apollographql/apollo-feature-requests#4

benjamn · 2019-09-12T16:33:48Z

src/cache/inmemory/entityCache.ts


  // Although the EntityCache class is abstract, it contains concrete
  // implementations of the various NormalizedCache interface methods that
  // are inherited by the Root and Layer subclasses.

  public toObject(): NormalizedCacheObject {
-    return this.data;
+    return { ...this.data };


Because the StoreWriter class updates entities non-destructively, this shallow copy of this.data is sufficient to provide a complete, immutable snapshot of the cache.

jbaxleyiii · 2019-09-12T17:33:51Z

src/cache/inmemory/__tests__/entityCache.ts

+
+    // If you're ever tempted to do this, you probably want to use cache.clear()
+    // instead, but evicting the ROOT_QUERY should work at least.
+    expect(cache.evict({


Also called "Welcome to the fragment zone"

benjamn · 2019-09-12T17:38:53Z

src/cache/inmemory/entityCache.ts

+    if (idsToRemove.length) {
+      let root: EntityCache = this;
+      while (root instanceof Layer) root = root.parent;
+      idsToRemove.forEach(root.delete, root);


By the way, this idiom is equivalent to

idsToRemove.forEach(id => root.delete(id))

except that it does not allocate a new function object.

This implementation currently requires calling cache.gc() manually, since the timing of garbage collection is subject to developer taste.

jbaxleyiii

@benjamn as always, this is wonderful work filled with helpful comments and the occasional chuckle when reading the tests. I really appreciate your styling in the commits and actual implementation here.

I've left one comment that I'd love answered before moving this to an approve. Namely, do we still need the query parameter to the eviction option?

src/cache/inmemory/entityCache.ts

jbaxleyiii · 2019-09-12T17:38:37Z

src/cache/inmemory/entityCache.ts

+  public gc() {
+    const ids = this.getRootIdSet();
+    const snapshot = this.toObject();
+    ids.forEach(id => {


This method for creating a stack is just so cool

jbaxleyiii · 2019-09-12T17:39:32Z

src/cache/inmemory/entityCache.ts

+    const idsToRemove = Object.keys(snapshot);
+    if (idsToRemove.length) {
+      let root: EntityCache = this;
+      while (root instanceof Layer) root = root.parent;


Ah, so this works up the layers to delete ids

The way I think about it:

Optimistic layers are temporary, so we don't really need to worry about garbage collecting them, since they should be totally removed sometime soon.

The goal of garbage collection, then, is to remove unreachable IDs from the Root layer of the cache.

The gc() method might have been called against a Layer object, which is important because optimistic Layers can retain otherwise unreachable entities.

If the layers didn't matter for garbage collection, we could just skip to the root at the beginning of the gc() method.

Instead, we skip to the root at the end of the gc() method to perform the final deletions of unreachable entities. That's what this code does.

src/cache/inmemory/entityCache.ts

jbaxleyiii · 2019-09-12T17:47:18Z

src/cache/inmemory/entityCache.ts

@@ -125,27 +211,27 @@ export namespace EntityCache {
 // of the EntityCache.Root class.
 class Layer extends EntityCache {


This is probably out of scope for this PR, but I think it would be quite helpful to write a set of comments on the architecture of the cache and its layers.

src/cache/inmemory/entityCache.ts

src/cache/inmemory/helpers.ts

src/cache/inmemory/inMemoryCache.ts

In practice (by which I mean while writing tests), I found it way too difficult to keep track of owner objects effectively. If an application developer wishes to implement a more sophisticated root ID tracking system, they have all the tools they need. In theory, the developer should balance retain and release calls, though one can always call release repeatedly until it returns 0 to release the ID completely.

Eviction always succeeds if the given options.rootId is contained by the cache, but it does not automatically trigger garbage collection, since the developer might want to perform serveral evictions before triggering a single garbage collection. Resolves apollographql/apollo-feature-requests#4. Supersedes #4681.

@jbaxleyiii

Per this discussion with @jbaxleyiii: #5310 (comment) tl;dr: This could be a breaking change for other Apollo Cache implementations, but InMemoryCache and the Hermes cache are the only two cache implementations we know about, and neither of them have implemented the evict method so far.

@jbaxleyiii

Per this discussion with @jbaxleyiii: #5310 (comment) tl;dr: This could be a breaking change for other Apollo Cache implementations, but InMemoryCache and the Hermes cache are the only two cache implementations we know about, and neither of them have implemented the evict method so far.

hwillson

This is AMAZING @benjamn! Thanks very much for working on this, and integrating these changes in such a well thought out and easy to grok manner. I have a few comments below (mostly thoughts or questions), and an additional one here - it looks like we have a circular dependency:

Circular dependency: src/cache/inmemory/inMemoryCache.ts -> src/cache/inmemory/writeToStore.ts -> src/cache/inmemory/inMemoryCache.ts

It isn't caused by the changes in this PR, but just a heads up in-case you want to address it here.

Thanks again!

src/cache/inmemory/entityCache.ts

src/cache/inmemory/inMemoryCache.ts

src/cache/inmemory/entityCache.ts

@hwillson

Based on discussion with @hwillson here: #5310 (comment)

@jbaxleyiii

Per this discussion with @jbaxleyiii: #5310 (comment) tl;dr: This could be a breaking change for other Apollo Cache implementations, but InMemoryCache and the Hermes cache are the only two cache implementations we know about, and neither of them have implemented the evict method so far.

@hwillson

Based on discussion with @hwillson here: #5310 (comment)

dminkovsky · 2020-11-15T21:01:17Z

Thank you for this.

As of this writing, the cache.gc() method is never called automatically

Is this still the case?

benjamn added 🛫 in-progress 🧞‍♂️ enhancement labels Sep 12, 2019

benjamn added this to the Release 3.0 milestone Sep 12, 2019

benjamn self-assigned this Sep 12, 2019

benjamn requested review from hwillson and jbaxleyiii September 12, 2019 15:26

benjamn added a commit that referenced this pull request Sep 12, 2019

Mention PR #5310 in CHANGELOG.md.

a1fd24c

This was referenced Sep 12, 2019

Feature/garbage collect cache #4681

Closed

Release 3.0 #5116

Merged

benjamn commented Sep 12, 2019

View reviewed changes

jbaxleyiii reviewed Sep 12, 2019

View reviewed changes

benjamn commented Sep 12, 2019

View reviewed changes

benjamn added 2 commits September 12, 2019 13:44

Make StoreWriter#writeSelectionSetToStore private.

a869634

Implement mark-and-sweep garbage collection for EntityCache.

a5ee594

This implementation currently requires calling cache.gc() manually, since the timing of garbage collection is subject to developer taste.

jbaxleyiii reviewed Sep 12, 2019

View reviewed changes

benjamn added 6 commits September 12, 2019 16:43

Reimplement findChildRefIds without using JSON.stringify.

7d335ba

Add basic tests of EntityCache garbage collection.

79125bb

Temporarily increase bundlesize limit to 16.9kB.

3f4ff08

Mention PR #5310 in CHANGELOG.md.

d24485f

benjamn force-pushed the cache-garbage-collection-and-eviction branch from a1fd24c to 7c5b4e5 Compare September 12, 2019 21:03

benjamn force-pushed the cache-garbage-collection-and-eviction branch from 7c5b4e5 to 3b659f8 Compare September 12, 2019 21:04

Mention new cache.evict(id) signature in CHANGELOG.md.

1ae4757

benjamn requested a review from jbaxleyiii September 12, 2019 21:11

hwillson reviewed Sep 13, 2019

View reviewed changes

src/cache/inmemory/entityCache.ts Show resolved Hide resolved

src/cache/inmemory/inMemoryCache.ts Show resolved Hide resolved

src/cache/inmemory/entityCache.ts Show resolved Hide resolved

jbaxleyiii approved these changes Sep 13, 2019

View reviewed changes

Stop automatically retaining entities when reading.

4b3d78c

Based on discussion with @hwillson here: #5310 (comment)

hwillson approved these changes Sep 13, 2019

View reviewed changes

benjamn merged commit 219ddaa into release-3.0 Sep 13, 2019

dylanwulf mentioned this pull request Sep 20, 2019

Partial cache reset apollographql/apollo-feature-requests#29

Closed

morrys mentioned this pull request Oct 1, 2019

Cache is not updating and new queries are not being displayed morrys/wora#17

Closed

StephenBarlow pushed a commit that referenced this pull request Oct 1, 2019

Mention PR #5310 in CHANGELOG.md.

72ea0e5

StephenBarlow pushed a commit that referenced this pull request Oct 1, 2019

Stop automatically retaining entities when reading.

ce8617d

Based on discussion with @hwillson here: #5310 (comment)

This was referenced Oct 31, 2019

Implement missing evict method for InMemoryCache #4917

Closed

Expose data for InMemoryCache #4916

Closed

benjamn mentioned this pull request Jun 22, 2021

Add options to cache.gc to enable deleting nonessential/recomputable result caching data #8421

Merged

github-actions bot locked as resolved and limited conversation to collaborators Feb 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement InMemoryCache garbage collection and eviction. #5310

Implement InMemoryCache garbage collection and eviction. #5310

benjamn commented Sep 12, 2019 •

edited

Loading

benjamn Sep 12, 2019 •

edited

Loading

jbaxleyiii Sep 12, 2019

benjamn Sep 12, 2019 •

edited

Loading

jbaxleyiii left a comment

jbaxleyiii Sep 12, 2019

benjamn Sep 12, 2019

jbaxleyiii Sep 12, 2019

benjamn Sep 12, 2019 •

edited

Loading

jbaxleyiii Sep 12, 2019

hwillson left a comment •

edited

Loading

dminkovsky commented Nov 15, 2020

		@@ -125,27 +211,27 @@ export namespace EntityCache {
		// of the EntityCache.Root class.
		class Layer extends EntityCache {

Implement InMemoryCache garbage collection and eviction. #5310

Implement InMemoryCache garbage collection and eviction. #5310

Conversation

benjamn commented Sep 12, 2019 • edited Loading

benjamn Sep 12, 2019 • edited Loading

Choose a reason for hiding this comment

jbaxleyiii Sep 12, 2019

Choose a reason for hiding this comment

benjamn Sep 12, 2019 • edited Loading

Choose a reason for hiding this comment

jbaxleyiii left a comment

Choose a reason for hiding this comment

jbaxleyiii Sep 12, 2019

Choose a reason for hiding this comment

benjamn Sep 12, 2019

Choose a reason for hiding this comment

jbaxleyiii Sep 12, 2019

Choose a reason for hiding this comment

benjamn Sep 12, 2019 • edited Loading

Choose a reason for hiding this comment

jbaxleyiii Sep 12, 2019

Choose a reason for hiding this comment

hwillson left a comment • edited Loading

Choose a reason for hiding this comment

dminkovsky commented Nov 15, 2020

benjamn commented Sep 12, 2019 •

edited

Loading

benjamn Sep 12, 2019 •

edited

Loading

benjamn Sep 12, 2019 •

edited

Loading

benjamn Sep 12, 2019 •

edited

Loading

hwillson left a comment •

edited

Loading