cache instance for subplugins

[pieh]

After implementing separate cache for each plugin, there is issue with cache instance for subplugins:

• subplugins can implement their own API hooks and there they get cache instance for that subplugin
• way parent plugins interact with subplugins is most often implementing most of subplugin functionality in something like index.js in this is called by parent plugin and often we pass cache from parent plugin to subplugin - for example in gatsby-transformer-remark:

  gatsby/packages/gatsby-transformer-remark/src/extend-node-type.js

  Lines 186 to 203 in 37489fa

  	Promise.each(pluginOptions.plugins, plugin => {
  	const requiredPlugin = require(plugin.resolve)
  	if (_.isFunction(requiredPlugin)) {
  	return requiredPlugin(
  	{
  	markdownAST,
  	markdownNode,
  	getNode,
  	files: fileNodes,
  	pathPrefix,
  	reporter,
  	cache,
  	},
  	plugin.pluginOptions
  	)
  	} else {
  	return Promise.resolve()
  	}

Because of that different cache instances are used in subplugins gatsby-node.js and index.js files.

You can check https://github.com/pieh/gatsby-subplugins-cache reproduction (most important thing is in https://github.com/pieh/gatsby-subplugins-cache/tree/master/plugins/gatsby-remark-test ).

I think we would need to pass cache manager to plugins as well so parent plugins can pass correct cache instances to subplugins

/cc @DSchau @raae

[KyleAMathews]

Or do we really need different instances per plugin? My thought was that we'd track cache IDs by plugin so could purge cache items that way. Having plugins manage cache instances sounds like a confusing and error prone API.

[DSchau]

The plugin name is effectively the id, and that's what sets everything up!

Having plugins manage cache instances

I don't think we want that either! The cache process should be transparent to plugins, the API hasn't/won't change!

[pieh]

Or do we really need different instances per plugin? My thought was that we'd track cache IDs by plugin so could purge cache items that way.

Tracking cache keys by plugin will have similar issues (at least for this issue) as we are passing cache object (which would be what ultimately would have to track what cache keys belong to what plugin). With separate cache instances we get this tracking for free as every plugin just have its own cache, so we can just purge entire plugin cache and not filter out cache keys belonging to plugin we want to purge.

Having plugins manage cache instances sounds like a confusing and error prone API.

I do agree on that. This is only needed for some subplugins (when we pass cache instance to them from parent plugin and when they implement their own gatsby node API hooks).

Maybe alternative solution would be to handle subplugins caches in gatsby core and pass them as part of subplugins array - but it would ultimately be up to parent plugins to make use of that. Unless we abstract subplugins even more so parent plugins don't call them directly, but that would be breaking change at this point.

[pieh]

Also just to be clear - right now this change should only be used by gatsby-transformer-remark plugin when calling subplugins - other plugins should use cache objects they get from gatsby core

[chmac]

@pieh @KyleAMathews @DSchau Any update on this one? It seems like the implementation of the gatsby-remark-oembed plugin (which looks awesome) is blocked by this. Anything I can do to help out? It seems like we're blocked waiting for a decision on the best approach.

[amykapernick]

Any update on this? This bug also means I'm stuck on 2.0.17 because I'm using gatsby-remark-oembed but now I'm getting a different error error Plugin gatsby-transformer-sharp returned an error TypeError: getNodesByType is not a function which according to #9533 needs to use 2.0.33 or higher to resolve

[DSchau]

@amykapernick a few of us are out with the holiday until early next week, so I can prioritize this as something to take a look at first thing next week. Hopefully that's OK! IF not PRs are more than welcome in the interim!

[amykapernick]

@DSchau I would love to help out, but unfortunately I'm not quite at a point where I can contribute to this kind of bug. When I do get there though, I'm keen to give back to Gatsby because it's amazing and I love the work you're all doing!

[raae]

What is the status of this issue? I see there is a pull request, but do not understand if it's being approved or not.

If this will take a long time I will have to start looking into a workaround for my plugin.