addPreprocessor Configuration API method to Ignore Files and simplify "Draft" templates

[khrome83]

Trying to identify if in .eleventy.js if there is a way to determine if I want to process a file or not.

I would love it if I can do the following for example -

Frontmatter

---
status: draft
---

# Test

.eleventy.js

module.exports = function(eleventyConfig) {
  eleventyConfig.ignoreFiles(function(file) {
    return (process.env.DRAFT === 'true') ? false : file.data.status === 'draft';
  });
};

NPM Scripts

"scripts": {
  "build": "DRAFT='true' eleventy --input=src --output=dist",
  "dev": "DRAFT='true' eleventy --input=src --output=dist --serve",
  "publish": "eleventy --input=src --output=dist"
}

This would allow the ability to not only setup draft or other conditions for publications, but even future date publications (check timestamp), ignore files to specific output targets - Example - api docs & frontend component docs could share a repo, but have two different build targets? Maybe they end up on different domains.

I tried looking through the code to see if I could add support for this, but I am a little unclear what determine what should render outside of the ignoreFiles which takes a full file path. So you would need to scan all files in a prestep to add it too that list running the function passed into the config.

[zachleat]

Oh, I do like this. The ability to run a filter callback on the found files array before processing would have a lot of utility, I think.

[zachleat]

Can you upvote your original post? This is going into the new feature queue

[zachleat]

This repository is now using lodash style issue management for enhancements. This means enhancement issues will now be closed instead of leaving them open.

The enhancement backlog can be found here: https://github.com/11ty/eleventy/issues?utf8=%E2%9C%93&q=label%3Aneeds-votes+sort%3Areactions-%2B1-desc+

Don’t forget to upvote the top comment with 👍!

[khrome83]

@zachleat - love that you adopted the lodash format. First time I am seeing the explanation. Makes perfect sense.

Sort of feel like this should be a GitHub plugin that handles this. Can't wait to see someone monetize the concept.

Really happy you like the concept. Is there anything short term that would make sense to do besides having a custom collection to build links that are not in draft, and crawling the output and remove pages that were listed in draft? Seems like more work than adding the initial feature, but I am struggling to wrap my head around how the code is organized, no offense.

[edwardhorsford]

I have a related need that this would help with...

I'd like to be able to have 'template' files in each of my directories that I can duplicate when I want to create a new post. They'd have standard front-matter applied so my posts are more consistent.

My current strategy is just to give them an extension eleventy doesn't recognise. But being able to mark them as drafts would be nicer.

[khrome83]

@zachleat - what is the target vote count. We are currently sitting at 12...

Would love to see this feature make it into the next branch. For production we are currently taking the output, and then removing the files individually, and ensuring draft files are not listed as part of a collection to not show in the menu. Not ideal. =)

[edwardhorsford]

This is also high on my wish list. I've currently got a load of files sat uncommitted because I don't want them to get processed on production.

I've got a secondary use for this - I've made a stats page that lists every page that eleventy is processing and links to it. Also gives summaries of how many pages, how many tags, etc. Already it's helped me catch Eleventy outputting a load of files I wasn't expecting.

This is useful for dev, but obviously I don't want it rendered in prod.

[khrome83]

@edwardhorsford - Stats page sounds really cool, would that be a good feature to "bake" into Eleventy? Would be nice to see a visual output for sure.

[zachleat]

Patience, @khrome83 😀 It’s coming. Until then you can use permalink: false #61 in 0.5.4

[adamsilver]

Very excited for this feature.

[SeanMcP]

@edwardhorsford It isn't a JavaScript solution, but to achieve something similar I added this line to my .eleventyignore file:

**/_*.md

Now all drafts or templates prepended with an underscore (e.g. articles/_template.md) are ignored. To publish (or preview) them, remove the underscore.

EDIT: I ended up moving templates to _includes/templates/. The underscore still works well for drafts.

[khrome83]

@SeanMcP - I just manually added tags of the existing status. Draft, Staged, Relesed. Then I setup custom collections to pull the right data depending on the enviroment.

[SeanMcP]

@khrome83 That's definitely a more powerful solution to the problem. Can you share those custom collections in a gist or something?

[Ryuno-Ki]

Since I'm doing something similiar with the toolbox on my website...

// .eleventy.js
function toolbox (collection) {
  const toolPages = collection.getFilteredByTag('tool');
  toolPages.sort((a, b) => {
    const titleA = a.data.title.toLowerCase();
    const titleB = b.data.title.toLowerCase();
    if (titleA < titleB) {
      return -1;
    }

    if (titleA > titleB) {
      return 1;
    }

    return 0;
  });
  return toolPages.filter((item) => item.data.tags.filter((tag) => tag !== 'tool').length > 0);
}

and on /public/projects/toolbox/tenon.md:

---
tags:
  - tool
  - accessibility
  - a11y
  - validator
  - service
layout: tool
title: 'tenon.io'
---
[Tenon.io](https://tenon.io/)

Finally, the template:

---
layout: h1
title: 'Toolbox'
---
<p>
Over the time, different handy websites and programs crossed my way.
</p>
<p>
In order to not to have to look them up again and again, I'm listing them here.
</p>
<div data-js="tag-filter"></div>
<ul class="toolbox">
  {% for tool in collections.toolbox %}
    <li class="tool" data-tags="{{ tool.data.tags | excludeItem('tool') | join(' ') }}">
      <p>
        <a href="{{ tool.url }}">
          {{ tool.data.title }}
        </a>
      </p>
      <p>
        Tagged with {{ tool.data.tags | excludeItem('tool') | sort | join(', ') }}
      </p>
    </li>
  {% endfor %}
</ul>

<script type="text/javascript" src="/js/toolbox.js"></script>

[zachleat]

Related from @remy: https://remysharp.com/2019/06/26/scheduled-and-draft-11ty-posts

[edwardhorsford]

Is there any progress on this?

I'd really like to be able to tell Eleventy not to process a file on the basis of frontmatter and maybe a callback / function. If it's possible, please let me know how.

I've just discovered this is a significant cause in build times for my site.

A file that I thought I was ignoring because of permalink:false was in fact still getting processed, even if it didn't output anything.

Temporarily deleting the file just now, my build time went from 15 seconds all the way down to 8 seconds

[infovore]

Yep, similar.

I am investigating building a fairly chunky site (with a variety of collections) with Eleventy. Given there are two modes of use: writing content + checking it, vs production build - I've used an environment variable to determine whether to do a "full" or "light" build. That works well for most of my custom collections - I can shave nearly 90s off the build time if I don't even generate those collections.

However, I can't shave down the 'default' collection, nor the 'all tags' collection. Actually deleting my "all posts with a tag" paginated template, tag.njk, knocks 9s off my build time, down from 29 to 20; at this point, the only thing left being built is ~3000 individual pages. Of course, ideally, being able to programmatically ignore certain files - say, everything but the last year of content - would bring that up a lot, which is really handy for a) development and b) content-writing.

It looks like the bottleneck is at the point of the 'default' collections, and what gets passed into them; a code-based way of throttling that would be useful.

[zachleat]

Cross posted from #2060

See also this approach using the Configuration API for ignores. I like that folks can decide what convention they want to use for drafts (the example below uses a _ prefix and a .md suffix).

module.exports = function(eleventyConfig) {
  if(process.env.CI) {
    eleventyConfig.ignores.add("**/_*.md");
  }
};

In this example process.env.CI is set by your CI server, e.g. https://docs.netlify.com/configure-builds/environment-variables/#netlify-configuration-variables

[vrilcode]

@zachleat Is there any solution to make "ignores" more specific? For instance iterating over all pages in Eleventy config and add them to ignore list if they fulfill certain criteria? Something like this:

for (const item of items) {
  if (item.data.draft) {
    eleventyConfig.ignores.add(item.data.page.inputPath)
  }
}

[pdehaan]

@cvh23 I don't think you could. In order for Eleventy to know "items", it'd have to do a full pass of all the pages in order to build the data cascade (and know the front matter for each file, as well as merge w/ directory data files and global data files, etc. to determine what item.data.* is).

You'd have to find a way to globally map draft to permalink:false and eleventyIncludeInCollections:false to prevent drafts from being built, otherwise you could possibly do some two stage build to generate a list of draft files which get programmatically ignored.

  eleventyConfig.addCollection("drafts", collectionApi => {
    const $drafts = [...collectionApi.getAll()]
      .reduce((pages = [], page = {}) => {
        if (!!page.data.draft) {
          pages.push(page.data.page.inputPath);
        }
        return pages;
      // Note that the initial `drafts` seed data here is via the `require()`d "./drafts.json" file above.
      }, drafts);
    // Write the "drafts.json" to disk so we can use for future builds.
    fs.writeFile("drafts.json", JSON.stringify($drafts));
    return drafts;
  });

Then elsewhere in .eleventy.js config:

  let drafts = [];
  try {
    drafts = require("./drafts.json");
    for (const p of drafts) {
      eleventyConfig.ignores.add(p);
    }
  } catch (err) {
    // No "drafts.json" file found. Ignore.
  }

If I don't have an initial "drafts.json" file, my local test repo will generate 14 pages (including my 4 drafts) but after that first build and the drafts.json file is created via my fake collection, it will generate the 10 non-draft pages. But it's a bit awkward. Because if you end up removing the draft: true front matter from a file, you'd have to also remove it from the cached drafts.json file before rebuilding, or else delete the drafts.json file and rebuild it (and have it probably generate too many pages the first time).

And you can't call eleventyConfig.ignores.add() from within the eleventyConfig.addCollection() since it the ignored files are read before collection creation.

[freyquency]

I'm a novice at Eleventy but couldn't YAML's built-in variable, published: false be used??

[zachleat]

I don’t think YAML provides any specific data variables to Eleventy.

I would note for future visitors that this has been packaged up in the eleventy-base-blog project—there is a small bit of explainer on the README! https://github.com/11ty/eleventy-base-blog#readme

[zachleat]

Eleventy v3.0.0-alpha.17 ships with a very powerful preprocessor API (be careful with this one!)

export default function(eleventyConfig) {
	// function(name, file extensions, callback)
	eleventyConfig.addProcessor("drafts", "njk,md,liquid", (data, content) => {
		if(data.draft) {
			return false; // Explicit `false` ignores this template the same as `eleventyConfig.ignores` or `.eleventyignore`
		}

		// You can also modify the rawInput of the template here, be careful!
		return `Butts${content}`;

		// But unlike transforms: if you return nothing or `undefined`, nothing is changed
	});
};

Additional example in the tests here:

eleventy/test/EleventyTest.js

Lines 1531 to 1549 in f19ef55

	test("#188: Content preprocessing", async (t) => {
	let elev = new Eleventy("./test/stubs-virtual/", undefined, {
	config: eleventyConfig => {
	eleventyConfig.addPreprocessor("drafts", ".njk", (data, content) => {
	if(data.draft) {
	return false;
	}
	return `Hello ${content}`;
	});
	eleventyConfig.addTemplate("index.njk", "Before");
	eleventyConfig.addTemplate("draft.njk", "Before", { draft: true });
	}
	});
	let results = await elev.toJSON();
	t.is(results.length, 1);
	t.is(results[0].content, `Hello Before`);
	});

This will drastically simplify the drafts use case.

[zachleat]

Additional documentation note: Eleventy Layout files are not subject to preprocessors

[uncenter]

Why are the file extensions prefixed with .? I believe other configuration options dealing with file extensions / template languages don't (e.g. templateFormats).

[zachleat]

@uncenter Good point! I did do that intentionally as I plan to add support for globs eventually but I think that was overly cautious. I changed it to support both but we’ll probably roll with "njk" and ["njk"] for the docs for consistency!

[murtuzaalisurti]

woah, this is so powerful, the ability to modify content programmatically before build is awesome, like I could a banner to posts published before a certain date or the other way around:

eleventyConfig.addPreprocessor("outdated", "md", (data, content) => {
        if (new Date(data.date).getTime() < new Date("2023-03-12").getTime()) {
            return content + `> this is old content`;
        }
    })

[openmindculture]

What is the recommended way to use drafts / unpublished page fragments in Eleventy 2.0 ? Asking because all related issues have been closed but 3.0 is still marked as beta. permalink: false throws build time errors in 2.0

[zachleat]

@openmindculture https://github.com/11ty/eleventy-base-blog/blob/main/eleventy.config.drafts.js has an example.

Regardless, permalink: false should work in 2.0 either way

[zachleat]

Docs deploying to https://v3.11ty.dev/docs/config-preprocessors/

[Ryuno-Ki]

Ah, it's in Alpha 17. That means I should be able to test it already.
(Helpful as I don't have to risk loosing my drafts again over hardware failure 😿 )

Thank you!