Can you create collections from tags added dynamically via global data

[kilmc]

Hey folks,

I've been struggling to find an answer to this one in issues or discussions so I'm hoping this post is different enough to make this discussion worthwhile.

Problem: Collections with tags I have added dynamically to pages when generating them from JSON data are empty.

Details

I make a lot of end of year lists. I've decided to store them as JSON files and I currently have a bunch of files that looks something like this:

src/lists/annual/2019.music.albums.json

{
  "type": "annual",
  "year": 2019,
  "category": "music",
  "subcategory": "albums",
  "list": []
}

Based off some solutions I found in issues on how to create pages from these files I'm doing the following to create a "paginatable" data structure out of these files

src/_data/lists/annual.js

const fs = require("fs");
const path = require("path");

const annualFolder = path.resolve(__dirname, "../../lists/annual/");

const annualLists = fs
  .readdirSync(annualFolder)
  .filter((name) => path.extname(name) === ".json")
  .map((name) => ({
    key: path.parse(name).name,
    ...require(path.join(annualFolder, name)),
  }));

module.exports = annualLists;

I'm then paginating with the following code.

src/lists/annual/annual.njk

---
pagination:
    data: "lists.annual"
    size: 1
    alias: "list"
permalink: "lists/{{ list.category }}/{{ list.year }}/{{ list.subcategory }}/"
tags: 
  - {{ list.category}}
  - {{ list.year }}
  - {{ list.subcategory }}
---
<h1>{{ list.title }}</h1>
<ol>
  {% for item in list.list %}
    <li>{{item.title}}</li>
  {% endfor %}
</ol>

If you notice in the tags section I'm adding tags for the bits of meta data I want to create indexes for (category, year, subcategory). My issue is that when I try to create an index via a collection there are no pages with those tags.

src/indexes/music.md

---
title: Music Lists
permalink: music/
layout: base.njk
---

## {{title}}

{%- for post in collections.music %}

- [{{post.data.title}}]({{post.url}})

{% endfor %}

When I have tried to write in a hard-coded tag in the annual.njk file I get results from that collection/tag combo but for all of the data in the curly braces, I get empty collections.

Is this some kind of order of operations issue where the data isn't resolved by the time the collections are generated?

Any help at all would be really appreciated.

Thanks

[pdehaan]

You can’t set dynamic tags that way. permalink is the only front matter key that can be set dynamically (I think). You’ll need to look into “eleventyComputed” in the docs. I can try helping debug when I get to a laptop.

[kilmc]

I think I'm following what you're saying here. I'll do a little bit of playing around with that last idea. Is there anything I can do to kinda +1 this as a bug somewhere? Should I link this discussion to either of those other issues or is it just something that may get dealt with over time?

[kilmc]

@pdehaan I'm making some progress but I'm stuck a little. Here's what I've done so far

annual.njk

---js
{
  pagination: {
    data: "lists.annual",
    size: 1,
    alias: "list",
    addAllPagesToCollections: true
  },
  permalink: "lists/{{ list.category }}/{{ list.year }}/{{ list.subcategory }}/",
  eleventyComputed: {
    title: (data) => {
      const { year, category, subcategory } = data.list
      return `${category}: ${year} ${subcategory}`
    },
    tags: (data) => {
      const { year, category, subcategory } = data.list
      return [year, category, subcategory, 'dynamic']
    }
  }
}
---

<h2>{{ list.title }}</h2>
<ol>
  {% for item in list.list %}
    <li>{{item.title}}</li>
  {% endfor %}
</ol>

.eleventy.js

eleventyConfig.addCollection("dynamic", (collectionApi) => {
  return collectionApi.getFilteredByTag("dynamic");
});

annualLists.md

---
title: Annual Lists
permalink: lists/annual/
layout: base.njk
eleventyExcludeFromCollections: true
---

<main>

## {{title}}

{%- for post in collections.dynamic %}

- [{{post.data.title}}]({{post.url}}) ({{post.data.tags}})

{% endfor %}

</main>

What I want to do next is build up that set of tags and then generate collections but in the code example below, what data do I have access to in eleventy.js that I could use to generate that set? In essence, how do a populate the dynamicTags array below.

---

module.exports = function (eleventyConfig) {
  // [other eleventy config stuff]

  dynamicTags.forEach((tag) => {
    eleventyConfig.addCollection(tag, (collectionApi) => {
      return collectionApi.getFilteredByTag(tag);
    });
  });

  // [other eleventy config stuff]
}

Again, thanks so much for your help on this!

[pdehaan]

Note that I've never actually attempted this yet, so I can only give you my rough hypothesis and some pseudo code and cross our fingers it's remotely correct/intelligent/coherent.

I don't think you can tell what your dynamicTags are in eleventyConfig because I don't know if everything is built yet. Here's where we get weird. We might need to build the dynamic collections inside of another .addCollection() call since I think all the pages are probably generated by the time .addCollection() is invoked and all the pagination has paginated and dynamic bits have been fetched and processed.

I suspect we have to do something like this:

module.exports = function (eleventyConfig) {
  // [other eleventy config stuff]

  eleventyConfig.addCollection("dynamic", (collectionApi) => {
    const dynamicTags = new Set();

    // Get all posts currently tagged with "dynamic" tag.
    const dynamicPosts = collectionApi.getFilteredByTag("dynamic");

    // For each post in the collection, see if we can add each tag in `tags` array into our Set.
    dynamicPosts.forEach(post => post.data.tags.forEach(tag => dynamicTags.add(tag));

    // For each unique tag in our Set...
    [...dynamicTags].forEach(tag => {
      // ... create a new collection based on the current tag name and populate it with any posts with that dynamically set tag.
      eleventyConfig.addCollection(tag, collectionApi => collectionApi.getFilteredByTag(tag));
    });

    // Return our original array of dynamic tagged items.
    return dynamicPosts;
  });

  // [other eleventy config stuff]
};

DISCLAIMER: I typed that in GitHub, so might be littered with typos and faulty logic.

So if I'm right (happens very rarely, don't get too excited), this would create a collections.2019, collections.music, and collections.albums collection with the posts from our dynamically tagged posts.

Probably a lot of stuff can go wrong there... What if there isn't a .tags property in the post.data object? The .forEach() will probably throw some errors unless you add some additional checks.

What if one of your categories or subcategories is something like "all"? It'd probably overwrite the internal collections.all and unleash some form of chaos. Not sure if thats a legit concern, or just hypothetical. Possibly prefixing with year:2019, cat:music, and subcat:albums is silly overkill.

Not sure if you can do something like collections.2019. Seems like possibly a JavaScript syntax error, but I'm sure it might be fine if you did collections[2019] or possibly collections["2019"]. I don't know and would need to experiment. I guess it all depends on how you consume the collections elsewhere. 🤷

[kilmc]

I tried this out and sadly it's not working. I have a feeling that it's to do with these collections being registered inside of the closure for the addCollection('dynamic') because I tried a forEach to register these collections using a manually generated array in the main block of the eleventy.js and it worked perfectly. I also toyed with this approach…

// [other eleventy config stuff]
  const dynamicTags = new Set();
  
  eleventyConfig.addCollection("dynamic", (collectionApi) => {
    const dynamicPosts = collectionApi.getFilteredByTag("dynamic");

    dynamicPosts.forEach((post) => {
      post.data.tags.forEach((tag) => dynamicTags.add(tag));
    });

    dynamicTags.delete("dynamic");

    return dynamicPosts;
  });

  [...dynamicTags].forEach((tag) => {
    eleventyConfig.addCollection(tag, (collectionApi) =>
      collectionApi.getFilteredByTag(tag)
    );
  });

// [other eleventy config stuff]

Sadly the order these are run in means that the dynamicTags set is empty when the forEach is run. To check this I logged at the forEach and within the addCollection('dynamic') and it logged at the forEach first and then the inner part of addCollection('dynamic') after that.

I think that while this whole thing isn't fully automated, I've definitely found a solution that gets me 90% of the way there. My next plan is to try to write a little script to generate a JSON file with the tags in it and import that into eleventy.js to then do the forEach at that main block level. That feels like an acceptable compromise and it's just adding a little extra build step to my process.

Thanks @pdehaan for all of the help with this. Great first experience for me asking for help in this community!

[pdehaan]

Yeah, I couldn't really get that nested .addCollection() solution to work because I'm not very clever.
But on the bright side, I'm also incredibly lazy... Since I'm operating on a static ./src/_data/books.js file which returns an array of books:

// ./src/_data/books.js
module.exports = [
  {title: "Cat in a Hat", tags: ["cat", "hat", "dynamic"]},
  {title: "Dog in a Coat", tags: ["dog", "coat", "dynamic"]},
];

Now I can create another simple ./src/_data/dynamic-tags.js data file which adds all the book's tags into a Set (so I don't get duplicates) and then exports a sorted array of tags:

// ./src/_data/dynamic-tags.js
const books = require("./books");

const tagSet = new Set();

for (const book of books) {
  for (const tag of book.tags) {
    tagSet.add(tag);
  }
}

module.exports = [...tagSet].sort();

Now this simplifies my .eleventy.js logic greatly since I can precalculate the unique dynamic tags from my data file and operate on a simple for..of loop:

// .eleventy.js
const dynamicTags = require("./src/_data/dynamic-tags");

module.exports = eleventyConfig => {
  for (const tag of dynamicTags) {
    console.log(`Adding dynamic collection for "${tag}"`);
    eleventyConfig.addCollection(tag, collectionApi => collectionApi.getFilteredByTag(tag));
  }
  // ...
};

You'll still need to do a bit of tweaking on your side to get it to work with your "src/_data/lists/annual.js" file, but hopefully it works better than my previous idea of nested .addCollection().