Persist in-memory cache

[kamituel]

In-memory cache is empty whenever Eleventy build starts from scratch. It's not an issue for sites with small number of images. But for sites with large number of large images, compressing them takes quite a lot of time.

In my case (https://kamituel.pl) I've over 300 high resolution source images that I'd ideally serve in 2-3 formats (avif, jpg, maybe webp) in multiple sizes.

I propose a new config option that would, if enabled, persist the in-memory cache to disk and reuse it the next time the build starts.

If maintainers consider this idea to be a good candidate to exist in eleventy-img itself, I'd be happy to formulate this proposal more concretely and later help / submit a PR.

[dimitrinicolas]

For thoses wanting a quick solution for caching optimized images whenever build starts from scratch, here is a solution.

Warning, it checks for already existing file outputs by looking if their output path file is existing, it may or may not be ideal depending on your filenameFormat configuration.

const fs = require('fs');
const Image = require('@11ty/eleventy-img');
const path = require('path');
const shortHash = require('shorthash2');

/** ... */

const options = {
  /** ... */
  filenameFormat: (id, src, width, format) => {
    const extension = path.extname(src);
    const name = path.basename(src, extension);

    const stats = fs.statSync(src);

    const hash = shortHash(`${src}|${stats.size}`);

    return `${name}-${hash}-${width}w.${format}`;
  },
};

const stats = Image.statsSync(src, options);

/** Creating a flat array of all the output paths from the stats object. */
const outputPaths = Object.keys(stats).reduce((acc, key) => {
  return [
    ...acc,
    ...stats[key].map((resource) => {
      return resource.outputPath;
    }),
  ];
}, []);

/** Checking if all output files exists. */
let hasImageBeenOptimized = true;
for (const outputPath of outputPaths) {
  /** Edit the output file path resolving, depending of this file */
  if (!fs.existsSync(path.resolve(__dirname, '..', '..', outputPath))) {
    hasImageBeenOptimized = false;
  }
}

if (!hasImageBeenOptimized) {
  Image(src, options);
}

/** Do whatever you want with the `stats` object */
const imageAttributes = {
  // [...]
};

const html = Image.generateHTML(stats, imageAttributes, {
  whitespaceMode: 'inline',
});

If you are deploying on Netlify you can cache your eleventy-img output directory using netlify-plugin-cache.

npm install netlify-plugin-cache

In your netlify.toml:

[[plugins]]
  package = "netlify-plugin-cache"
    [plugins.inputs]
      paths = ["<YOUR OPTIMIZED FILES OUTPUT DIR PATH>"]

[marcamos]

I would love for something like this to exist in the official plugin. I thought I was doing something wrong when I saw that builds took three minutes (converting 22 images into three formats and ten sizes each [though, maybe that's overkill?]).

[snibbo71]

Just adding my tuppence here, that I'd like this too as I'm using the Image.generateHTML code to pull images remotely. The caching side of it works, but 11ty still chucks those cached images through the resizing process which annoys my laptop and takes a lot longer than is ideal :)

I'm really not clear on how to get Dimitri's work-around to work with the generateHTML method - if indeed I can

[dimitrinicolas]

@snibbo71 My piece of code does not take into account whether the resource is remote or local. I use it with the html generation function. After my shortcode, you can generate your html code with the stats object returned by Image.statsSync , e.g.:

const imageAttributes = {
  // [...]
};

const html = Image.generateHTML(stats, imageAttributes, {
  whitespaceMode: 'inline',
});

(I've just added it to my previous comment, to be sure that every future reader will understand)

Both Image.statsSync and Image.generateHTML are not generating resized/optimized images output files. The Image.statsSync is just returning the output file paths that the async Image.stats would "normally" export.

Then in my shortcode, we check if those output files are existing (that's why we rely on a hashed src+file size string in output file names), if not, we call the Image.stats async function, but we do not have to wait for it to resolves to generate the html. The whole html code generation is async.

[snibbo71]

Heya, thanks for the quick reply. I must be doing something entirely wrong because nothing is cached in my setup - even local stuff. Remote images just blow your code up unfortunately :) (Actually, it's the fs.statSync and Image.statSync that dies with ENOENT - which isn't entirely surprising)

I shall keep playing. I have too many images to deal with to have to redo them every time I change a template :) If I come up with anything I'll post back

[snibbo71]

Having dug into this, FileSizeCache isn't persistent, which explains why it doesn't work across restarts.

At the moment I can't dig much further to see whether there's any reason not to make that another AssetCache subclass. There may be good reasons not to persist this information. And that's largely down to the fact that I'm somewhat over my head as this is my first look at anything serious with node.js anyway :)

[dimitrinicolas]

I'm sorry, I haven't been into 11ty image for a moment, and I never used remote images fetching. Good luck in your research! 😄

[zachleat]

Since it came up in the Eleventy Meetup today, here’s a quick config-based solution that won’t build images that haven’t changed (after the initial build):

const eleventyImage = require("@11ty/eleventy-img");

module.exports = function(eleventyConfig) {
  let modifiedFiles = [];

  eleventyConfig.on("beforeWatch", files => {
    modifiedFiles = files || [];
  });

  // This function will likely not match your project and will require you to customize to your build.
  eleventyConfig.addJavaScriptFunction("eleventyImageHtml", async function(src, options, htmlAttributes) {
    let stats;
    if(modifiedFiles.length === 0 || modifiedFiles.includes(src)) {
      // During watch/serve, only generate images that have been modified.
      stats = await eleventyImage(src, options);
    } else {
      // Return the metadata only without generating the Image.
      stats = eleventyImage.statsSync(src, options);
    }

    return eleventyImage.generateHTML(stats, htmlAttributes);
  });
}

Read more about Events: https://www.11ty.dev/docs/events/ This solution will work in Eleventy 0.11.1+

Sample usage in an 11ty.js template:

class Test {
  async render() {
    return this.eleventyImageHtml("./nebula.jpg", {
      widths: [300],
    }, {
      alt: "Nebula"
    });
  }
}

module.exports = Test;

[brycewray]

Thanks very much, @zachleat — but guess I’ll have to wait until it’s incorporated in the regular eleventy-img examples (https://www.11ty.dev/docs/plugins/image/#use-this-in-your-templates) since I can’t seem to translate either of these to shortcode creation as opposed to addJavaScriptFunction. That’s my fault, not yours, of course! Much obliged for your answer today in the Eleventy Meetup as well as these code samples.

[zachleat]

Oh, I’ve made a mistake in my code, sorry about that! I’ll fix it

[zachleat]

So I went down the rabbit hole here, perhaps unnecessarily. Just to rewind a bit—we’re all aware of the existing In-Memory Cache feature right? https://www.11ty.dev/docs/plugins/image/#in-memory-cache-new-in-image-0.7.0

(this larger issue #51 is specific to persisting the in-memory cache to the file system)

The approach documented at #51 (comment) is unnecessary when useCache: true (though I did update it to work correctly)

Also note that you can use DEBUG=EleventyImg to check for in-memory cache hits and misses.

[marcamos]

Hi! Sorry, complete amateur chiming in here as I've tried interpreting the latest comments—and implementing the code provided in the most recent instance—and, I think, it's not working as desired … at least for me. No matter what I try, I can't get the cache to persist when I come back to my project a 2nd, 3rd, etc. time.

Each time I open the project and run (in my case) npm run dev, it churns through all of the images again and finally has the local environment ready to view about 100 seconds later, despite those images not changing in any way, shape, or form since the last time I opened and worked on the project.

…or, is this to be expected as eleventy-img is not setup for this particular scenario?

…double-or, am I a dope and completely missing something here?

♥️

[zeroby0]

Heyo @marcamos :)

Try using this PR #116

npm r @11ty/eleventy-img 
npm i zeroby0/eleventy-img#cache

If you do rimraf on the output folder, make sure you exclude the image output folder.
Examples here:

• https://github.com/zeroby0/netlify-plugin-11ty/#i-rimraf-my-_site-on-every-build
• Cache eleventy_img generated images AleksandrHovhannisyan/aleksandrhovhannisyan.com#99

[brycewray]

@marcamos It works for only remote images, not local images — the latter of which is what I asked @zachleat about during an Eleventy Meetup session, which I think is part of why he entered the conversation here recently. Unfortunately, doesn’t appear that caching local images to streamline the dev process is in the cards for the plugin, at least for the time being. Perhaps there are programmatic reasons why; I’m not qualified to guess.

[tobystokes]

fwiw, I have local images caching, i.e. not rebuilt on initial run based on @dimitrinicolas' previous answer

If a nunjucks shortcode is of any use to you, see https://gist.github.com/tobystokes/b4b8bb743ec4c57317f6ef043093ff95

[zachleat]

@brycewray @marcamos #116 @zeroby0 has a PR open to add this, pending review!

[zeroby0]

@zachleat Thanks :D ! Do you think there is someone I can request to review it? For some reason I have received no feedback or suggestions on it. I'm new to the repos, I don't know how things work here

[zachleat]

@zeroby0 I added a few comments—it’s looking super solid!

[zachleat]

Shipping this with Eleventy Image 1.0—thank you!