URGENT - Don't update to latest version unless you have time and nothing is critical

[andyburgessmd]

Very annoying - the latest update requires a couchdb update and will take down your apps and ability to do anything until you do the update.
budibase folks - perhaps you could have put a warning that this was needed before people press the button! NOT COOL!

[samwho]

Hey @andyburgessmd, you're absolutely right. I've gone and noted this in the release notes here https://github.com/Budibase/budibase/releases/tag/2.33.0, and we're adding it to our Changelog on the Budibase website as well.

Sorry we tripped you up here, thank you for letting us know and helping others avoid the surprise 🙏

[mike12345567]

Hi @andyburgessmd - just to check, did you perform this update using the button within Budibase which calls out to watchtower? We're sorry this occurred, we will likely be removing watchtower in the future as it is a very inconsistent way to update, we recommend updating using the Budibase CLI as it will automatically perform any infrastructure migration required (assuming your installation is relatively default).

[andyburgessmd]

Yes it was through the /builder/portal/settings/version page.
I've had to restore from a backup to get things working again.

[mike12345567]

Hi @andyburgessmd - sorry about that - we will likely be removing that in a very soon update because it is a very inconsistent upgrade mechanism which simply isn't fit for purpose.

For future we recommend upgrading using the Budibase CLI as it can correctly handle the update process (in a way that watchtower has no ability to).

[arminus]

Just to let you know, the upgrade 2.33.x did not work for me at all either. (I did not update with the CLI but with dockcheck).

After having changed the couchdb image to budibase/couchdb and having restarted the stack everything was gone. No apps, nothing. And no automigration running either. Reverting back to 2.32.17 didn't work either because the old ibmcom/couchdb3 images container wouldn't start anymore, even after I temporarily mounted a new/clean volume into it:

couchdb-service-1  | [error] 2024-10-27T18:12:09.479477Z couchdb@127.0.0.1 <0.316.0> -------- CRASH REPORT Process  (<0.316.0>) with 0 neighbors crashed with reason: no case clause matching {error,eacces} at mem3_util:ensure_exists/1(line:298) <= mem3_nodes:initialize_nodelist/0(line:94) <= mem3_nodes:init/1(line:45) <= gen_server:init_it/2(line:417) <= gen_server:init_it/6(line:385) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {mem3_nodes,init,['Argument__1']}, ancestors: [mem3_sup,<0.313.0>], message_queue_len: 0, links: [<0.314.0>], dictionary: [], trap_exit: false, status: running, heap_size: 610, stack_size: 28, reductions: 48407

I only managed to get out of this mess by restoring last nights backup (budi backups --export) into completely new minio and couchdb volumes. I guess I'll have to switch to https://docs.budibase.com/docs/budibase-cli-setup - first for any of my docker based services :-/ I usually stay away from "managed" docker configurations because the typically make assumptions which conflict with existing services on the same box etc., we'll see...

[samwho]

Hey @arminus, would you be willing to share your docker-compose.yaml file before and after the update that wiped out your data? We tested a wide variety of update scenarios when making this change but it sounds like we've missed one. Really sorry about that.

[arminus]

Here's the docker file from the last months:
docker-compose.yaml.pre-upgrade.txt - with this file, I followed the instructions as given in https://docs.budibase.com/docs/migrations#budibase-db-sql-support and ended up with wiped data.

I now have a file which I mixed together from what budi hosting --init generates and from my original file - I've also pinned the current versions for now just to avoid any surprises. docker-compose.yaml.txt

Note that contrary to what's stated here https://docs.budibase.com/docs/migrations#budibase-db-sql-support budi hosting --init does not generate COUCH_DB_SQL_URL - so is that needed now or not?

Also, one more thing - which wasn't a problem so far, keeping this in:

      BB_ADMIN_USER_EMAIL: ${BB_ADMIN_USER_EMAIL}
      BB_ADMIN_USER_PASSWORD: ${BB_ADMIN_USER_PASSWORD}

with having those values still set in the .env file did re-generate an admin user on first start from scratch, then, after having loaded the backup with budi backups --import ... apparently created a 2nd admin, preventing me from login in.

So - what if I need to modify the compose as generated by budi hosting --init (e.g. my policy is to never named volumes but bind mounts), can I still run budi hosting --update ?

[samwho]

Having had a look at the two docker-compose.yaml files you supplied (thank you!), I believe the reason your installation got reset is the missing TARGETBUILD environment variable. The data wasn't lost, but CouchDB was pointing at the wrong directory to try and find it. It's not entirely clear to me why going back to the old image didn't work, the error message makes it look like a filesystem permissions issue but beyond that I'm unsure.

I've updated the documentation at https://docs.budibase.com/docs/migrations#budibase-db-sql-support to mention this environment variable, and given the whole section a once-over. You're correct in that COUCH_DB_SQL_URL is no longer required, we made it default to the right value in the end and did not update these docs to reflect that.

The BB_ADMIN_USER_{EMAIL,PASSWORD} environment variables are intended as a "break glass" feature in case you lock yourself out of your installation and need a way back in. We don't recommend leaving them set all the time, and have updated the docs at https://docs.budibase.com/docs/hosting-settings to reflect this.

If you're making modifications to the docker-compose.yaml file then using budi hosting --init is going to either overwrite or conflict with your changes. You're welcome to make the sorts of modifications you're talking about, but you will need to check https://github.com/Budibase/budibase/blob/master/hosting/docker-compose.yaml each time you need to update to make sure you're getting the latest configuration changes.

Sorry again for the hassle. We try to support a range of deployment methods, and understand that some folks have need to do custom things. It was a miss from us to forget to mention the TARGETBUILD environment variables in our migration guide.