homepage NixOS package does not clear cache on upgrade leaving broken state

[sedlund]

Description

After upgrading from homepage-dashboard-0.9.13 to homepage-dashboard-0.10.6 the product did not clear its cache leaving a broken cache and the homepage output broken.

homepage version

homepage-dashboard-0.10.6

Installation method

Source

Configuration

# /etc/systemd/system/homepage-dashboard.service
[Unit]
After=network.target
Description=Homepage Dashboard

[Service]
Environment="HOMEPAGE_CACHE_DIR=/var/cache/homepage-dashboard"
Environment="HOMEPAGE_CONFIG_DIR=/etc/homepage-dashboard"
Environment="LOCALE_ARCHIVE=/nix/store/xsajjdk5fchhv42hhawvnk6a85h8q0q2-glibc-locales-2.40-36/lib/locale/locale-archive"
Environment="LOG_TARGETS=stdout"
Environment="PATH=/nix/store/4s9rah4cwaxflicsk5cndnknqlk9n4p3-coreutils-9.5/bin:/nix/store/jqrz1vq5nz4lnv9pqzydj0ir58wbjfy>
Environment="PORT=8082"
Environment="TZDIR=/nix/store/gfkyf4hw4riqwgf5scfk8r8g69crbn4x-tzdata-2024b/share/zoneinfo"
CacheDirectory=homepage-dashboard
DynamicUser=true
EnvironmentFile=/run/secrets/rendered/arrEnvironmentFile
ExecStart=/nix/store/n6imaq8va4q0a1v117rry1gg56kpjhnh-homepage-dashboard-0.10.6/bin/homepage
Group=homepage-dashboard
Restart=on-failure
Type=simple
User=homepage-dashboard

[Install]
WantedBy=multi-user.target

Container Logs

nothing related in systemd- journal

Browser Logs

No response

Troubleshooting

removing the cache (sudo rm -rf /var/cache/homepage-dashboard) and restarting the service (sudo systemctl restart homepage-dashboard) resolves.

when starting a new version it should take care of its cache appropriately.

also see: NixOS/nixpkgs#346016

[shamoon]

Yea sorry I think this is a Nixos thing, as I frequently see with their packages. Homepage is a relatively straightforward nextjs application so we do not directly manage the cache in any way. I don’t know what all of the config vars above are for but I suspect they are modifying that behavior.

I see in that thread they are somehow saying this is an upstream issue but I don’t see how, again nothing has changed about the app’s core nextjs setup between these versions (checking the diff or changelog will corroborate this)

[bjornfor]

@sedlund: Do you have a copies of the cache directories when the issue exists, and after re-creating the directory? It'd be interesting to see in what way they differ. I bet nextjs caches part of its env (a Nix store path) that it probably shouldn't.

[mainrs]

I updated from nixos-24.05 to nixos-24.11 and therefore upgraded homepage from 0.8.12 to 0.9.10. That was the change that started the caching issue.

The only thing present inside my /var/cache/homepage-dashboard directory was this:

$ tree /var/cache/homepage-dashboard
/var/cache/homepage-dashboard
└── homepage
    └── pages
        ├── en.html
        └── en.json

3 directories, 2 files

I could fix the issue by using

systemd.services.homepage-dashboard.environment.HOMEPAGE_CACHE_DIR = lib.mkForce "/tmp/homepage-dashboard";

However, that had not the effect one would suspect. I assumed the directory would contain some files. However, the directory is empty. I think what makes this work is that homepage just doesn't use any cache at all, circumventing all the issues.

[sedlund]

thanks for looking

Do you have a copies of the cache directories when the issue exists, and after re-creating the directory?

attached

homepage-caches.tar.gz

[bjornfor]

@shamoon: Thanks! What do you think happens if the app always runs with empty ...env.HOMEPAGE_BUILDTIME? Can this be the root cause?

[shamoon]

That should not cause an issue and that is presumably how most 'source' installs run. It just defaults to the empty string, revalidation should still happen for other reasons.

[sedlund]

@bjornfor

@sedlund: Have you tried rolling back the homepage version, and give it the old cache dir? Does it work again then

yes if i roll back, put the old cache dir back and restart the service it works.

[bjornfor]

It just defaults to the empty string, revalidation should still happen for other reasons.

What other reasons? I saw config file content(?), but that doesn't necessarily change. (Nix(OS) tries its best to give programs and services a reproducible environment, so there might be less things that "change" there.)

The comment in hash.js that was linked above says that in docker the service uses HOMEPAGE_BUILDTIME to force re-evaluation of the cache on every service start (or docker build?). That's interesting!

@sedlund: I think testing with setting HOMEPAGE_BUILDTIME environment variable to something would be valuable. Normally we'd use SOURCE_DATE_EPOCH, but since that isn't always reliable (can be 1970-01-01), I think maybe the homepage version string is the better option here. This would be something like systemd.services.homepage-dashboard.serviceConfig.Environment.HOMEPAGE_BUILDTIME = pkgs.homepage-dashboard.version in your NixOS config (untested).

[shamoon]

You are mixing the revalidate that happens here with static generation, which happens at build time. The content that gets hashed here only refers to the users content e.g. configs.

[sedlund]

it works fine in docker as well.

the container image puts the cache in /app/.next/server/pages - which gets thrown out when the container upgrades/restarts so it is regenerated like a next build redeploy. which afaict is one of the two options to revalidate the cache officially.

@bjornfor I think your suggestion to just delete the cache with ExecStartPre would mimic the container behaviour and be good enough. i will try the HOMEPAGE_BUILDTIME and report back later

[bjornfor]

Yes, we can do such things in ExecStartPre (adding the same workaround logic that the docker setup has), but I'm not convinced yet that there isn't a bug in homepage for not handling cache invalidation itself.

[shamoon]

I'm not convinced yet that there isn't a bug in homepage for not handling cache invalidation itself.

What exactly is that based on? nixOS is notorious for tinkering with apps and frequently breaking them, I think that is much more likely to be the issue.

Anyway, there seems to be confusion of two ideas here: the static site generation (SSG) which is done at build time (e.g. during CI for our docker image and required by running pnpm build with new releases) vs. homepage's mechanisms for detection of updated config files. The screenshot above is clearly not just an issue with an outdated config, something is messing with nextjs' SSG, either not properly re-building it or somehow messing with its own internal file management. Since the software works fine on other platforms, and is largely unchanged for literally a couple of years in how it runs, its hard for me to believe this is suddenly a "homepage bug".

I think thats about it for me with this issue, and as an aside, unfortunately this only furthers my perception that nixOS is frequently a burden to FOSS developers which I find frustrating.

I would be happy to be proven wrong, of course.

[bjornfor]

I was just saying I want to find the bug, and that externally cleaning the cache is a workaround, not a real fix. So I didn't want us to stop looking. I'm a bit disappointed that you seem so sure that NixOS is at fault (marked as the answer), without further evidence.

[shamoon]

Huh? This entire conversation and the very fact that it is only an issue on nixOS is evidence of that assertion.

But as I said, feel free to demonstrate otherwise.

[benphelps]

Thank you all, this is simply an issue with nixOS packaging.

[bjornfor]

I made a NixOS VM test and reproduced the issue and after a while I noticed that HOMEPAGE_CACHE_DIR doesn't exist upstream but is something introduced by nixpkgs 🤦. So that "cache dir" seems more like a build artifact than a runtime cache, and the issue only exist in nixpkgs.