Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨Image Backups API endpoint #14659

Closed
wants to merge 5 commits into from
Closed

✨Image Backups API endpoint #14659

wants to merge 5 commits into from

Conversation

ronaldlangeveld
Copy link
Member

@ronaldlangeveld ronaldlangeveld commented May 3, 2022

👋 Hi, first time contributor here.

I've been using Ghost for years and one small feature I've always felt was missing, is an image exporter on self-hosted installations, that could potentially compliment the JSON Content exporter. Of course, one could just ssh onto a server and manually zip + download via sftp or something, but it feels like extra steps to achieve something that one would expect to be in the settings.

I challenged myself to build it, as it would be a great way to familiarise myself with the Ghost codebase and as a gateway to perhaps get more involved in open-source.

Last week I dived right in and have a working alpha prototype up and running, that's ready for PR.

  • There's a clear use-case for this code change, explained below
  • Commit message has a short title & references relevant issues (I hope I did)
  • The build will pass (run yarn test:all and yarn lint)

I added an Export button in the Admin client as well under Labs -> Alpha features.
Added PR for that here TryGhost/Admin#2362

Screenshot 2022-05-03 at 16 53 50

A next iteration of this would be perhaps to make it a multithreaded task, to avoid it timing out when it's a big archive.
I only tested it with around 50mb worths photo content, on an M1 Pro. Most cloud servers aren't that powerful and production entities will have much more content to compress.
And then finally, perhaps an uploader that does the reverse.

Create backups.js (controller?) that would handle the request.
Import backups.js in index.js.
Index provides the exporter module, that would be initiated by the backups API controller.

BackupsExporter is the main function with the logic. For now it creates a new folder inside content called backups and generates a file called images.zip and then serves it via the http request.

Added /backups as a path / folder inside the shared helpers config file.
Pointed API controller to backups service.
@ErisDS
Copy link
Member

ErisDS commented May 3, 2022

Hey there @ronaldlangeveld 👋 I really appreciate your time, effort & enterprise with getting stuck in here.

I think you've touched on the main reason why this feature doesn't exist already: the image folder could be infinite in size and we have to make absolutely certain that a backups endpoint couldn't take a site offline as that rather defeats the object 😬

This means that the code to do this needs to be infallible and non-blocking. We'd probably need to generate the backup in a job & provide a download link in an email, rather than doing this inside an HTTP request. There are talks of restructuring the main importer this way too & it is also how the members importer was built.

There was an extensive discussion of implementing a backup feature into Ghost-CLI here: TryGhost/Ghost-CLI#468 although it died. This discussed the performance benefits and complexities of shelling out to tar on the server.

As a last note, there are additional files that would be needed inside of a backup or archive as Ghost now has folders with media & files as well as a few json/yaml files dotted around, although it depends slightly on what the definition of the backup is.

I don't say any of this to discourage you or dismiss your work and I completely agree it'd be great to have an official and single-step method for generating a ghost backup, however, there's a lot to think about here and I don't want you to spend more time going in a direction that we wouldn't merge.

Hope this gives you some food for thought 😄

@ronaldlangeveld
Copy link
Member Author

Hi @ErisDS

Thank you so much for the detailed feedback, much appreciated. 😊

While building it, I did have a gut feeling (that my curiosity ignored 😅) that there's probably informed reasons by the team for not having a feature like this already - and you've summed it up perfectly.

Was fun building a prototype regardless. 😊

@ErisDS
Copy link
Member

ErisDS commented May 4, 2022

If this was something we could do in a blocking HTTP request then this implementation would be spot on 👌 would just be asking for a test 😬

I'll close this for now. If you get inspiration to build something non-blocking by all means open a fresh PR and I'll take a 👀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants