Update 2.6.5 -» 3 : lost all folder pairs but one, comming back one by one restarting client

[tomdereub]

Issue not serious because I finally got back my config, just for information.

Expected behaviour

After update, I should keep all folder pairs that were configured.

Actual behaviour

My nextcloud client was configured with 3 different servers, the first one of with 9 different pairs of folders synchronized.
After updating, I got only one folder pair for each server.
After closing nextcloud client and starting it again, I got 2 pairs for folders for the first server. Then restarting again 3 pairs, etc... So I got again all folders pairs after restarting 9 times nextcloud client.

Steps to reproduce

1. Configure nextcloud client 2.6.5 with many folder pairs
2. Update to nextcloud client 3
3. If all folder pairs don't appear, restart nextcloud client.

Client configuration

Client version: nextcloud client 2.6.5 to version 3

Operating system: Windows 10

OS language: French

Installation path of client: C:\Program Files (x86)\Nextcloud

Logs

Sorry, I didn't find logs just after the update, logs I get with nextcloud --logfile log.txt were just 2 minutes before entering that command.

[Xansala]

Had the same issue with 2.7.0-rc1 when updating from 2.6.5. Only have one server but connections only came back the same way tomdereup is describing it.

[er-vin]

@Aldaris1985 this would seem linked to having several sync folders then? Do you have such a setup as well?

If someone manage to reproduce that upgrade path and provide us debug logs on the first start of 3.0.0 after moving from 2.6.5 I'm interested.

[Emporea]

I was also facing this issue. After upgrading from 2.6.5 to 3.0.0 Nextcloud didnt recognize my existing data > 500 MB and started to download them again ( 80 GB ... )

On another test system I installed 3.0.0 without any existing version. In the setup I ticked the "sync all folders". After the setup when the first sync started it asked me again and unticked every folder > 500. That seems to be the bug in the first place for me.

[er-vin]

@DominiqueFuchs do you think that could be related to the journal db move somehow? Didn't have time to investigate it much yet, but that came to mind...

[CoMPaTech]

Looks primarily because the ._sync(...).db moved from the local storage (i.e. ~/NextCloud) to (at least for us on macOs) to ~/Library/Application\ Support/NextCloud (the generated id-naming resembles the old db name).

Didn't test what would happen if you move the ._sync*db file yourself though

[DominiqueFuchs]

Looks primarily because the ._sync(...).db moved from the local storage (i.e. ~/NextCloud) to (at least for us on macOs) to ~/Library/Application\ Support/NextCloud (the generated id-naming resembles the old db name).

Indeed, this is what @er-vin was referring to. We had to implement a upgrade mechanism for that and I think this was in the 2.6.5 release. IIRC we did that to exactly prevent happening stuff like this here, but maybe multiple folders are not habdled correctly. Will try to dig that out on the weekend if no one does it earlier.

[CoMPaTech]

For what it's worth, my laptop and one from a colleague had multiple accounts - so that could be it. We'll test on a single-connected laptop and let you know

[er-vin]

Looks primarily because the ._sync(...).db moved from the local storage (i.e. ~/NextCloud) to (at least for us on macOs) to ~/Library/Application\ Support/NextCloud (the generated id-naming resembles the old db name).

Indeed, this is what @er-vin was referring to. We had to implement a upgrade mechanism for that and I think this was in the 2.6.5 release.

AFAIR 2.6.5 doesn't move the database, only started doing that since 2.7.0-beta2 (I might be a bit off there though).

IIRC we did that to exactly prevent happening stuff like this here, but maybe multiple folders are not habdled correctly. Will try to dig that out on the weekend if no one does it earlier.

That would be appreciated!

[DominiqueFuchs]

FTR: @er-vin is right, it was 2.7.0-beta2: #2188

[CoMPaTech]

FWIW - Testing laptop with single account 2.6.5 => 3.0.0 no db migration (macOS 10.15.6)

or do you mean the path should be 2.6.5->2.7.0->3.0.0?

[DominiqueFuchs]

@CoMPaTech Thanks for testing. You mean single account but multiple sync folder connections, right?

And no, 3.0.0 includes the same code, no insertion of the beta version necessary

[CoMPaTech]

Yes, my own laptop was multiple users and multiple (>512mb) folders (connected to the same nextcloud server) - the testlaptop was a single user but with multiple (amongst others >512mb) folders.

[CoMPaTech]

Other admin at our side tried by manually moving the DB and it started a full sync but his 'selective synced' folders remained intact and he is operational again.

[Xansala]

@Aldaris1985 this would seem linked to having several sync folders then? Do you have such a setup as well?

Yes, several sync folders but only one server source.

[DPTJKKVH]

Ubuntu 20.04
Nextcloud Desktop 3.0.0 via Nextcloud Devs PPA / Upgraded from 2.6.5

I had the same issue with most of my sync folders being removed from the client after upgrade so I had to add them again. HOWEVER I just noticed that they continue to get lost again an again after reboots. So I am now in the situation that I have to re add the same sync folders again and again with every reboot.

[DPTJKKVH]

I just noticed: After restarting the client now the lost sync entries reappeared (this didn't happen before I manually readded after the upgrade) but I now have duplicate sync entries. Some sync entries are two times present, some are three times present. The duplicate sync entries are only from folders I had to readd after the upgrade however not every readded folder suffers from that duplication issue.

EDIT: I now manually removed the duplicate entries and restarted the client again. This seem to have worked but I will still monitor it over the next days.

[raoulbhatia]

I am facing a similar issue. After upgrade on MacOS, the folders >500MB were not marked for synchronization.
After checking them, my client re-downloaded >50GB of data.

Is there any work around when upgrading my other systems? I read something about manually moving the DB. Is there additional information available beyond what is mentioned in #2279 (comment)?

Thanks,
Raoul

[DPTJKKVH]

It seems to have fixed itself now.

Here is exactly what I did:

Ubuntu 20.04 using the Nextclous Devs Stable PPA on Launchpad.

No Advanced Settings in General Tab.

1. Upgrade from 2.6.5 to 3.0.0

2. After some days notice that most of my sync folders are gone.

3. Manually readd missing Sync folders.

4. Next day after a reboot notice that most if not all readded sync folders are missing again.

5. Closing and restarting the sync client.

6. Noticing that missing sync folders are suddenly back but some of my manually readded sync folders now have duplicate entries.

7. Manually removing duplicates.

8. Restarting Sync client. All Folders are present and no duplicates.

9. Next days after some reboots: All is still well.

10. Update to 3.0.1 and reboot. Still all okay.

[vasyugan]

Exactly the same here.

[brtptrs]

After upgrading, all but one sync entries were missing. Stopping and restarting the client added one more sync entry on each restart. After x restarts all entries were restored and everything works great.

I did not have to add the missing entries manually and no duplicates were created.

V3.0.1 on Ubuntu 20.04

[MorphBonehunter]

I am facing a similar issue. After upgrade on MacOS, the folders >500MB were not marked for synchronization.
After checking them, my client re-downloaded >50GB of data.

Is there any work around when upgrading my other systems? I read something about manually moving the DB. Is there additional information available beyond what is mentioned in #2279 (comment)?

Thanks,
Raoul

Same here but on Windows.
All Folders >500MB where removed on local system and must resynced afterwards.
There where a short warning about findings of directories >500MB and one could press "sync all" but this does nothing and the folders got removed.

[pstimpel]

Same basic issue, but was able to get away without new syncing, just by enabling all those "do you really want to sync this big folder" folders again for sync. Then close nextcloud client, restart it. Took two runs of that procedure. Windows 10, Multiuser machine but only one user is using nextcloud. The data are all there, as far as I can tell

[bugzy]

The following is an attempt to describe exactly where the points failure occur for the new client that created the problem described by OP:

1. The latest client seems to be moving the sync journal out of the "sync path" into the "config path." (Note: each sync folder has a journal.)

2. For some reason, the first launch of the updated client succeeds only at moving the journal of the last alphabetically configured sync folder.

3. Subsequent restarts behave similarly to 2. above in that they only move the journal for the next to last alphabetically configured sync folder (that was not previously moved).

4. Caveat: If multiple configured sync folders begin with the same alphabet, the client sometimes successfully moves all their journals upon the same restart.

5. Issue: If a user recreates a "lost" sync folder at step 1. A new journal is created in the config path for that sync folder while the old journal remains in the sync path. (Note: a new sync journal means freshly downloading the content of the sync path)
   Unfortunately, the client is unable to detect multiple journals for same sync folder. Hence, when a restart finally triggers the move of the sync journal out of the sync path in a re-added sync folder, multiple entries for the same sync folder will exist in the client because multiple journals now exist in the config path.

6. The root problem is the failure described in 2. The next failure is the inability to detect multiple journals for the same sync folder (in config path and sync path, and after both are moved to sync path).

[747]

This does not seem a new problem in v3. I encountered it several times after update, and each time I did something similar to @pstimpel 's. This time to 3.0.1 too, unfortunately.

[er-vin]

This does not seem a new problem in v3. I encountered it several times after update, and each time I did something similar to @pstimpel 's. This time to 3.0.1 too, unfortunately.

Ah so might not be related to the database move after all... I'd be interested in you or someone else running the client once with --logdebug and --logfile just after an upgrade. If it doesn't show all the sync folders you expect then quit and send us the log file.

An "only on upgrade" event is not that easy to reproduce and having a log at that time might help.

[patricksebastien]

All my users lost their sync folders, now my server needs to resync a huge amount of files. Is there a way to disable the update notification on nextcloud side so that other users will not see the update notification?

[Gizmokid2005]

This does not seem a new problem in v3. I encountered it several times after update, and each time I did something similar to @pstimpel 's. This time to 3.0.1 too, unfortunately.

Ah so might not be related to the database move after all... I'd be interested in you or someone else running the client once with --logdebug and --logfile just after an upgrade. If it doesn't show all the sync folders you expect then quit and send us the log file.

An "only on upgrade" event is not that easy to reproduce and having a log at that time might help.

I have a log that isn't the first time after the upgrade, but it's still adding folders back on each launch. It looks like there's probably a bit of sensitive information in this log cookies and paths, is there someone I can send this to?

[Birkenstab]

I had the same behaviour described in the issue directly upgrading to 3.0.1

[jokuha]

2. For some reason, the first launch of the updated client succeeds only at moving the journal of the last alphabetically configured sync folder.

I have seen such behavior on both my MacBooks when upgrading to 3.0.1, but there does not seem to be any alphabetical sorting in the addition process.
Neither ascending nor descending, nor was it the same on both systems, which have the very same sync-folder settings. Thus, the order seems to be different, at least on my macOS systems.

[er-vin]

All my users lost their sync folders, now my server needs to resync a huge amount of files. Is there a way to disable the update notification on nextcloud side so that other users will not see the update notification?

Having autoUpdateCheck=false in the General group of the user's nextcloud.cfg should do the trick.

[er-vin]

I have a log that isn't the first time after the upgrade, but it's still adding folders back on each launch. It looks like there's probably a bit of sensitive information in this log cookies and paths, is there someone I can send this to?

You could send it to me I guess. You should be able to drop it there: https://cloud.nextcloud.com/s/SD2cDdr5rKAPxG5
This file drop folder will be unshared tomorrow.

Thanks in advance.

[Gizmokid2005]

@er-vin I have uploaded the log file for this.

Hope this helps.

[er-vin]

Thanks

[er-vin]

One more thing, could you let me know which of the sync folders reappeared during that sync? (complete path to the folder would be preferred)

[Gizmokid2005]

@er-vin I figured that might come up. I'm not positive, but I think I can check again. Give me a few minutes and I'll see if I can pull that together quickly for you.

[Gizmokid2005]

@er-vin Can you give me one more share link? I have a new log file with a folder I know was added this launch.

[er-vin]

@er-vin Can you give me one more share link? I have a new log file with a folder I know was added this launch.

There you go: https://cloud.nextcloud.com/s/ZQ3sKQ2pfHAqKc2

Please remember to indicate me somewhere the path to said folder (both on the client side and server side). Thanks you.

[Gizmokid2005]

@er-vin Thanks! I've uploaded the file, the folder was "Introversion" but the first few lines in the log (surrounded with "=") give the actual paths (In the V2 file, the original is just the pure log). Please let me know if you need anything else!