Rebuild underlying image + Preserve Workspace files

[eliezedeck]

Is your feature request related to a problem? Please describe

Currently, when I update the Docker image for a workspace (for example, add a new tool as I progress in the project), I'm always forced to create a new workspace, thereby loosing all the things I have done in that workspace, especially the Docker containers. Needless to say, it's pretty annoying.

Describe the behaviour you'd like

I have an old workspace that depends on a gitpod/workspace-*:latest, after some time away and I come back, I suppose it got updated and Gitpod rebuilds the image, but does not wipe away the workspace: that is exactly what I'm looking for, but kind of on demand.

So, may be, have a menu that says "Rebuild image + Preserve Workspace", something like that. This is much better when it's done at Gitpod's level, not when the workspace has already started (see why just below).

One of the nice thing about this also, is that I retain the URLs to the workspace. For example, I develop webhooks, and I register the URL to our partner, I'm progressing in the development, it's ideal that the URL remains the same or else I'll have to ask the partner again to use a different URL, that's very frustrating indeed.

Describe alternatives you've considered

While this is something that can be done manually, like for example creating a new workspace and then syncing the whole /workspace folder from the old to the new one, it's far from an ideal situation of it just works.

One of the problem with this approach also, is the fact that usually you want Docker to be up and running for the new workspace. So, I have to manually stop everything first to ensure the files in /workspace/* are not currently used, then I do the sync, then I manually restart everything back manually, or stop the workspace, and restart after a few seconds once it's fully stopped. As one can tell, it's very frustrating to do that manually.
So, again, a process that is much better managed during Gitpod's workspace preparation phase.

Additional context

Like I said earlier, it is already happening, but not on demand. So, I'm guessing it should be possible with some not-too-heavy works.

I had started this discussion on Discord: https://discord.com/channels/816244985187008514/1004317083087159346

[geropl]

Hey @eliezedeck , can you describe exactly what steps you're doing now, and how your're using your workspace? And what you expect differently? I'm asking because I feel there is a misunderstanding of concepts somewhere, or at least I have a hard time wrapping my head around it.

Especially this part is sth I don't understand:

like for example creating a new workspace and then syncing the whole /workspace folder from the old to the new one

This should never necessary, and I 'd love to understand why you think that's the only way out.

BTW It will not be possible to update the workspace image of a running workspace due to kubernetes constraints, but I'm unsure if your asking for this or not.

[eliezedeck]

Hey @geropl, so imagine this situation:

• Start to migrate a large project to Gitpod, one that has been created long before Gitpod
• You build your initial gitpod.yml file with what you think is going to be necessary for the projet
• It's a projet that contains a bunch of containers during the dev, like MySQL, MongoDB, Kafka, ETCD, etc... via Docker Compose
• You progressively add more data into MongoDB, MySQL, etc... while these are test data, they are somehow valuable as they are closely related to the current task, so these are not data that you just want to be wiped-out, much like if you're working directly on your computer, you don't want to wipe-out your containers because you build your app upon these data

Now, you feel you're getting productive and progressing:

• You realize that you needed the ping tool to test something, for some reason (or any other system tools)
• So, you basically have 3 options:
  • Add it manually for the current workspace; but then every time you restart that workspace, you'll have to do those steps again; needless to say it's frustrating
  • You can rebuild the workspace, right now, it's basically starting a brand new workspace, and loose all your MongoDB and workspace/Docker stuffs; now, you have to somehow reconstruct all previous data, may be thru backups or manually; it's far from straightforward
  • You can start another workspace, ensure you stopped all Docker containers, sync the workspace files from the old to the new, then try to start the containers again hoping they'll be safely integrated in the new workspace

Now, the most pain part is this: what if you had several tools you realize you need to add into the workspace and not just ping. Obviously, you won't know everything you'll need initially, and progressively you'll need more. You can imagine the amount of manual work that will need to be done everytime you restart the workspace given the fact that you want to keep the workspace as it was last time.

So, that's basically the use case. The emphasis here is then about productivity, and it's a legitimate use case to have long-lived workspace.

Like I said, this is already happening sometimes when the upstream image is updated. So, why not make it explicit?

One this I haven't yet tested is: what if I build my own image with the tools I need, and if I update that image, would Gitpod pick that up at workspace startup and rebuild the base image?

[jldec]

We recommend automating as much as possible, all the steps needed to rebuild your workspaces.

That said, this does sound like the concept of rebasing for buildpacks (cc: @csweichel)

[eliezedeck]

We recommend automating as much as possible, all the steps needed to rebuild your workspaces.

OK. One requirements for me to do that is trying to get the hostname + username that is required to SSH into the old worspace (still running). This is still a manual step for me, @jldec I wonder if there is an API that I can use.

[axonasif]

Hey @geropl and @jldec, I think you could get some additional context if you also check the discord thread he linked in his main-post 😊

I'll try to say the same thing again with what I understood from the discord conversation.

• Let's say gitpod/workspace-full image latest tag gets updated because of some new change at the workspace-images repo.
• I have an old workspace pinned on my dashboard that uses gitpod/workspace-full:latest on either it's .gitpod.yml:image or the .gitpod.Dockerfile. (or doesn't, in that case it's[workspace-full:latest] the default image anyway, isn't it 👀?)
• Now if I re-open that old workspace, I would see an imagebuild happening as the latest tag was updated.
• In this automatic case, my old /workspace dir is still there and I still get the same workspace URL. But my container was updated.

So what he's asking is a manual way to trigger this thing I believe.

Right now we can trigger imagebuild (i.e gitpod.io/#imagebuild/<repo>) but that requires creating a new workspace. What if we could trigger it like gitpod.io/#imagebuild/<ws-id> as well? (I'm not suggesting this however, see below 👇 )

Maybe #7671 also relates with this somehow.

[eliezedeck]

Thanks @axonasif. That's exactly what I'm looking for. It's already happening, but not yet on demand.

It would be very cool and will save lots of time if it can be triggered on demand.

[atduarte]

Hey @eliezedeck! Thanks for the suggestion and thanks everyone for the additional clarifications.

Workspace images are published quite infrequently, and workspaces are supposed to be ephemeral and used for a single task.

The root need seems that you require stable urls, which is a long-standing request: #898. Would invite you to read some of the suggestion in that issue and then, if possible, invite you to share more details about why you need to use a specific URL, without wildcards. Hope it is helpful 🙏

In the meantime, I will close the request as it doesn't align with the vision for the product regarding ephemerality of workspaces. Feel free to reopen if you disagree with anything I said 💭

[eliezedeck]

Thanks for your input @atduarte.

While it is true that the workspaces was designed to be ephemeral, for certain projects (for example for trunk-based ones) you'll be working mostly on the main branch. So, you don't really need to create that much workspace to work on feature a, b or c. So, the ephemeral-ness of the workspace can go like many many days or weeks as there's just no need to branch out to a new workspace.

Another thing: it is possible to pin the workspace. Now, let's say it could go as long as 6 or a full year if the user needs it to be, right? When the user resumes work on it, he may still very much want to keep that workspace for a reason (hence the pin) but after one year, the underlying image will very likely need to be updated, at least for security purposes, while still maintaining the workspace data.

Last, but no least, if I'm not mistaken, Gitpod's vision is also about Better Developer Experience and Productivity. The workspace's vision is to be ephemeral, which is under the general Gitpod's vision. So, in that sense, I'd say the Gitpod's main vision should take precedence over the workspace's vision.

After all, it's already partially happening, but just not on demand to satisfy the Productivity and Better DevX vision. I mean, it's so close!

[eliezedeck]

@atduarte Stable URL isn't probably the root need for it, however, it is indeed a very nice side effect.