Provide a way to discover a template's dependencies

[chocolateboy]

nunjucks doesn't provide a way to discover a template's direct or transitive dependencies i.e. the child and grandchild templates that are loaded (via extends, import or include) when a parent template is rendered. This is needed to integrate (efficiently) with bundlers such as Parcel, which track changes in dependencies and use them to trigger rebuilds.

(Note: this is orthogonal to nunjucks' caching support via chokidar, though the implementations may overlap.)

Ideally, this would be scoped to a particular Environment#render call, but there doesn't appear to be any space in the render method's parameters to squeeze in additional options, or to return an additional value. Maybe a sister method with a more flexible interface?

// result is an object rather than a string
const result = env.parse(templatePath, { context: ..., trackDependencies: true })
console.log('dependencies:', result.dependencies)
console.log('source:', result.source)

This makes dependency tracking optional, to err on the side of efficiency, but they could be included by default as it's a new API, and the overhead is unlikely to be high e.g.:

const result = env.parse(templatePath, { context: ... })

Another way to do it is via an EventEmitter API (which is already used, lightly, in loaders) e.g.:

env.on('dependency', ({ name, path, parent }) => { ... })

This has several advantages:

• it doesn't clutter or crowd any existing APIs
• it's easier to implement
• it's easy to extend
• if there's a performance concern, it could be toggled on/off with a constructor option e.g. new Environment(loaders, { trackDependencies: true })

The main disadvantage is that it doesn't directly answer the question "What are this template's dependencies?" i.e. it requires extra work to scope the results to a particular template/render. (This also implies the use of a full EventEmitter implementation rather than the cut-down version that's currently used for loaders, and suggests the addition of some new events e.g. render:start and render:end.)

---

I'm envisaging each dependency as an object with the following core fields:

{
    name: "../macros/util.html.njk",
    path: "/home/foo/dev/example/src/html/macros/util.html.njk",
    parent: "/home/foo/dev/example/src/html/screens/layout.html.njk",
}

Optional fields could include cached (a boolean indicating that the template was retrieved from its loader's cache) and maybe async (the template was retrieved by an async loader).

Example

layout.html

{% include "../components/header.html" %}
<h1>Body</h1>
{% include "../components/footer.html" %}

header.html

<h1>Header</h1>

footer.html

<h1>Footer</h1>
{% include "./copyright.html" %}

copyright.html

Copyright ⓒ example.com 2018

render

    const result = env.parse("src/html/screens/layout.html")
    console.log(result.dependencies)

result

[
    {
        name: "src/html/screens/layout.html",
        path: "/home/foo/example/src/html/screens/layout.html",
        parent: null,
    },
    {
        name: "../components/header.html",
        path: "/home/foo/example/src/html/components/header.html",
        parent: "/home/foo/example/src/html/screens/layout.html",
    },
    {
        name: "../components/footer.html",
        path: "/home/foo/example/src/html/components/footer.html",
        parent: "/home/foo/example/src/html/screens/layout.html",
    },
    {
        name: "./copyright.html",
        path: "/home/foo/example/src/html/components/copyright.html",
        parent: "/home/foo/example/src/html/components/footer.html",
    },
]

[chocolateboy]

Related: #1129

[andreyvolokitin]

So, for now, we can use nunjucks-parser to get dependencies from entry file?

[chocolateboy]

@andreyvolokitin Yes, but the implementation is hacky as it's tightly coupled to the internals of Environment#getTemplate. The EventEmitter solution mentioned above would make it non-hacky.

[andreyvolokitin]

@chocolateboy I use only FileSystemLoader in my use case, and internally it gets every file path needed (though every other loader should also do it). If there would be a way to utilize this, I thought that this may work similar to nunjucks-parser where it collects every dependency with Environment#getTemplate?

[chocolateboy]

@andreyvolokitin I'm not sure what you're asking, but if you have a question about nunjucks-parser feel free to raise an issue.

[andreyvolokitin]

I was wondering if it possible/viable to solve this issue (getting template dependencies) by using the internals of a loader (FileSystemLoader specifically), which gets all related file paths

[chocolateboy]

Not without changing Environment#getTemplate. Environment#getTemplate doesn't call Loader#getSource if the template is cached (the noCache option doesn't affect this).

[fdintino]

I think hooking into Environment.getTemplate() would be the correct way to approach this, no? It's perfectly legal nunjucks / jinja2 syntax to write:

{% include template_name %}

In the simple case, where includes and extends are only strings, it is possible to parse the dependencies. But in the general case I think you would only be able to determine dependencies at runtime.

[fdintino]

Do you have a proposal for what sort of option would make it easiest to get at runtime template dependencies for parcel? I'd be partial to a callback or an event emitter, so that it would work for async templates as well as synchronous rendering.

[chocolateboy]

@fdintino Yes, I'd prefer the EventEmitter API.

[fdintino]

@chocolateboy does something like 35e3ad7 work for you?

[chocolateboy]

@fdintino I haven't had a chance to play with it, but yes, thanks, that looks useful 👍