WebP links being generated, no webp in content

[elijahsgh]

This may be something very weird with my storage adapter, but when using source I have my images generated with a formats/webp path but my content directory has no formats directory and no .webp

The articles (and images) were created while using source. The srcsets for the other formats are generated.

[minimaluminium]

Hey @elijahsgh, do you have a live link to your website?

[elijahsgh]

https://resolvereptiles.com

The webp links are removed from the template as mentioned above.

[minimaluminium]

Would you be able to provide some steps to reproduce the issue?

1. Create a post using Source theme
2. Upload a webp image as a feature image and/or a content image in the editor
3. Check the content/images directory, but there's no format directory and .webp images in there

Is that the correct way to test?

[elijahsgh]

Would you be able to provide some steps to reproduce the issue?

1. Create a post using Source theme
2. Upload a webp image as a feature image and/or a content image in the editor
3. Check the content/images directory, but there's no format directory and .webp images in there

Is that the correct way to test?

That's exactly what I did.
There are a couple of items. I use a modified version of the Google Cloud Storage adapter (GCP buckets). In that bucket there is no format directory nor any .webp, etc.

If you want me to try creating the directories on the volume I can. I don't see any errors or anything and I assume I would if it was just a missing directory.

[ngeorger]

I think

Would you be able to provide some steps to reproduce the issue?

1. Create a post using Source theme
2. Upload a webp image as a feature image and/or a content image in the editor
3. Check the content/images directory, but there's no format directory and .webp images in there

Is that the correct way to test?

That's exactly what I did. There are a couple of items. I use a modified version of the Google Cloud Storage adapter (GCP buckets). In that bucket there is no format directory nor any .webp, etc.

If you want me to try creating the directories on the volume I can. I don't see any errors or anything and I assume I would if it was just a missing directory.

I'm pretty sure this is unrelated to any theme at all, it could be related to cache adapter, storage adapter or file permissions depending on how are you actually running Ghost (and obviously the version you are running).

Regards!

[elijahsgh]

I think what we learned here is that the webp stuff is generated on access by ghost. It doesn't matter if you upload in JPEG or webp format or what - it tries to create a formats/webp/<FILENAME>.jpeg and the templates serve those as webp.

When you're using a storage adapter this process doesn't happen. It would make more sense for ghost to automatically make webps as part of the resizing process but it seems like it doesn't do this. The resized images go to the storage adapter (GCP in my case) but the webps are never generated since Ghost isn't serving them.

Here's the line in the Source theme where the webp links are generated:
https://github.com/TryGhost/Source/blob/main/partials/post-card.hbs#L6

This will come back with a JPEG extension.

This is a very weird process.

My interim solution is to just run a python script to iterate through the storage adapter (GCP bucket) items, resize, reencode, upload to formats/webp, etc. It's very silly. Alternatively if you just remove instances of "format=webp" then Ghost works normally with the storage adapters.

The storage adapters are all 6-8 years out of date and unmaintained. I'm not sure if the webp functionality was intended to be added to them but never was or...? There's definitely a disconnect here on how the webp stuff is supposed to work.

[ngeorger]

I think what we learned here is that the webp stuff is generated on access by ghost. It doesn't matter if you upload in JPEG or webp format or what - it tries to create a formats/webp/<FILENAME>.jpeg and the templates serve those as webp.

When you're using a storage adapter this process doesn't happen. It would make more sense for ghost to automatically make webps as part of the resizing process but it seems like it doesn't do this. The resized images go to the storage adapter (GCP in my case) but the webps are never generated since Ghost isn't serving them.

Here's the line in the Source theme where the webp links are generated: main/partials/post-card.hbs#L6

This will come back with a JPEG extension.

This is a very weird process.

My interim solution is to just run a python script to iterate through the storage adapter (GCP bucket) items, resize, reencode, upload to formats/webp, etc. It's very silly. Alternatively if you just remove instances of "format=webp" then Ghost works normally with the storage adapters.

The storage adapters are all 6-8 years out of date and unmaintained. I'm not sure if the webp functionality was intended to be added to them but never was or...? There's definitely a disconnect here on how the webp stuff is supposed to work.

I'm running Ghost with Kubernetes and PersistentVolumeClaims, but I remember tried some storage adapters noticing exactly what you said so I've desisted to use any kind of adapters for the same reason. Thanks for your insights!

[elijahsgh]

I'm running Ghost with Kubernetes and PersistentVolumeClaims, but I remember tried some storage adapters noticing exactly what you said so I've desisted to use any kind of adapters for the same reason. Thanks for your insights!

Yeah honestly at this point I've heard people have just left Ghost local storage and then just turned on a CDN feature and that seems fine.

I was extremely confused and a little disappointed to find out that Ghost worked this way. Even with the local storage adapter you have to wait until someone accesses the resource to get the webp generated - which means even if you're cloning a local storage backend to something you still have to wait. All the webp stuff was implemented long after the storage adapters (which are unmaintained) were written.

CDN features solve these problems and accomplish roughly the same end result. 🤷 I'm guessing this is what they do with Ghost Pro.

[ngeorger]

@elijahsgh Have you tried it combined with the native Redis cache adapter feature? I guess it worth the try: Ghost Docs Adapters

Cache adapters
The cache layer is used for storing data that needs to be quickly accessible in a format requiring no additional processing. For example, the “imageSizes” cache stores images generated at different sizes based on the fetched URL. This request is a relatively expensive operation, which would otherwise slow down the response time of the Ghost server. Having calculated image sizes cached per image URL makes the image size lookup almost instant with only a little overhead on the initial image fetch.

By default, Ghost keeps caches in memory. The upsides of this approach are:

no need for external dependencies
very fast access to data
The downsides are:

Having no persistence between Ghost restarts — cache has to be repopulated on every restart
RAM is a limited resource that can be depleted by too many cached values
With custom cache adapters, like Redis storage, the cache can expand its size independently of the server’s system memory and persist its values between Ghost restarts.

[elijahsgh]

@elijahsgh Have you tried it combined with the native Redis cache adapter feature? I guess it worth the try: Ghost Docs Adapters

No. In my opinion, this is a redundant and kind of not useful feature.

What I want to do is have uploads go to a Google Storage Bucket (which they do with the GCP adapter I had to modify since it's unmaintained but still advertised on the plugins on the site...), have them resized (which ghost does automatically - which is an expensive operation), and have Ghost generate the webps while it's resizing in the previous step (which it doesn't - instead it does that expensive operation during request).

Adding redis causes you to have to repeat all of the above on restart or using redis LFU. It (in my opinion) makes way more sense to generate the resources on publish, have them go to the backend storage (bucket in my case), and if possible add a CDN feature.

The way it works doesn't make sense. :(

[ngeorger]

You are right again, thanks! BTW, if you are interested, take a look at my project to deploy ghost on kubernetes: https://github.com/sredevopsorg/ghost-on-kubernetes

[elijahsgh]

Very nice!
Here's mine in terraform:
https://github.com/elijahsgh/terraform-gke-ghost

When I get a chance I'll see what we did that was different.

Off the bat I think your Dockerfile is far cleaner than mine.