Virtual File Sync no longer working properly

[efrenbg1]

Since the beginning of this week all clients updated automatically to client version 2.7.2. This made Virtual File Sync to stop working properly. I have observed two behaviors:

1. With previous files synced -> New files are immediately deleted by clients with Virtual File Sync.
2. On a clean sync -> Folders are perfectly created, but files are not, causing the client to delete every single file on the server.

Expected behaviour

Working with Virtual File Sync as expected.

Actual behaviour

All files are missing from the client folder, causing a complete wipe of data on the server.

Steps to reproduce

1. Share folder with user x
2. Add Virtual File Sync with user x
3. Wait for second sync caused by missing files

Server configuration

Operating system: debian 10 (Linux 4.19.0 - amd64)

Web server: nginx/1.14.2

Database: 10.3.25-MariaDB

PHP version: 7.2.34

ownCloud version: 10.5.0

Storage backend (external storage): none

Client configuration

Client version: 2.7.2

Operating system: Windows 10 Pro 20H2

OS language: Spanish

Qt version used by client package (Linux only, see also Settings dialog): none

Client package (From ownCloud or distro) (Linux only): none

Installation path of client: C:\Program Files (x86)\ownCloud

Logs

1. Client logfile: (Nothing on the log file)

2. Web server error log: (Nothing on the log file)

3. Server logfile: (Nothing on the log file)

What I've tried

1. Full clean install of the client removing all %appdata% files

2. Dowgrading to previous client versions (only 2.5.4 seems to work as of right now)

3. Clean install of ownCloud server

4. Update Windows 10 to latest version

~ Thanks in advanced for your hard work ~

[voerg]

I can confirm the behavior of file deletions on the server if the virtual file sync is used.
Please provide an update.

[efrenbg1]

I have done more testing. The problem seems to be related to a Windows 10 update.

1. All clients (including version 2.7.2) work perfectly under compilation version:
   19042.662
2. Not working under compilation version:
   19042.685

Both Windows are on the 20H2 version. As a result of Windows update policy, all computers at my office updated at the beginning of this week causing also the update of the client (due to the reboot of the OS).

[Tenivar]

I have done more testing. The problem seems to be related to a Windows 10 update.

1. All clients (including version 2.7.2) work perfectly under compilation version:
   19042.662

2. Not working under compilation version:
   19042.685

Both Windows are on the 20H2 version. As a result of Windows update policy, all computers at my office updated at the beginning of this week causing also the update of the client (due to the reboot of the OS).

I can confirm that, we are trying to find what changed in the "cloud files" windows API. But if there was a major change in how windows 10 handle files, they would have announced it...
I wonder if that's just a parameter/function name that changed between versions

[hodyroff]

Thanks! We have identified the issue and are working on a fix.
Indeed the Windows 10 API changed and the error was not handled correctly resulting in the above described issue.
2.7.3 client is upcoming ...

[TheOneRing]

Fixed in 2.7.3 https://github.com/owncloud/client/releases/tag/v2.7.3

[butonic]

Is there a script to restore all files since a given date? Should be doable using bash & curl...

An moving files on the server CLI and then running occ file:scan would assign new file ids to the deleted files and all shares and metadata would be lost...

[welkineins]

@hodyroff Sorry for comment on a closed issue.

Can you share which API is changed in Windows 2020H2? It might affect other software use the same API.

Thanks.

[TheOneRing]

@hodyroff Sorry for comment on a closed issue.

Can you share which API is changed in Windows 2020H2? It might affect other software use the same API.

Thanks.

microsoft/Windows-classic-samples#164

[AlexAndBear]

@efrenbg1 @welkineins @voerg @Tenivar @butonic

Restore script is here
https://github.com/JanAckermann/owncloud-restore-trash

[stylefieber]

I'm a bit scared now to use ownCloud in production at my company, where I am responsible for the infrastructure. Could that happen again?

[dragotin]

Well. You might want to refer to microsoft/Windows-classic-samples#164 to understand how that could happen at all and what levels are involved.

As much as I can't assure you that nifty things will never happy again, I can assure you that we do everything for our part to avoid that. This has been the most severe bug in the desktop syncing since years. I am grateful that we have mechanisms in place to help (trashbin).

[efrenbg1]

Well. You might want to refer to microsoft/Windows-classic-samples#164 to understand how that could happen at all and what levels are involved.

@dragotin Just out of curiosity. Don’t changes to the cloud file API show first in the Insiders program?

All changes made to the cloud file API (or others) are first tested and published to the Insiders channel, but considering they are making constant changes to that API I'm not really surprised that has passed under the radar.

[TheOneRing]

Well. You might want to refer to microsoft/Windows-classic-samples#164 to understand how that could happen at all and what levels are involved.

@dragotin Just out of curiosity. Don’t changes to the cloud file API show first in the Insiders program?

All changes made to the cloud file API (or others) are first tested and published to the Insiders channel, but considering they are making constant changes to that API I'm not really surprised that has passed under the radar.

I'm running the beta insiders preview and the problematic change 19041.685 was released at the same time as for the other Windows users.

[EdouardRt]

@efrenbg1 @welkineins @voerg @Tenivar @butonic

Could you please download this zip file, unpack in a folder,
read the read.me file

Please try to restore the files and provide feedback.
(comment also shared in issue #38208.)

Thanks, i did try and run the restore.php,

Having a warning on composer install, and an error on php restore.php execution.
Tryied both from the owncloud directory and from /home, with admin and user rights. won't work.
Got an idea ? Am i doing something wrong ? is it related to the https ?

Warning on installation :
Cannot create cache directory /home/desktop/.composer/cache/repo/https---packagist.org/, or directory is not writable. Proceeding without cache
Cannot create cache directory /home/desktop/.composer/cache/files/, or directory is not writable. Proceeding without cache
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Package operations: 2 installs, 0 updates, 0 removals

Installing sabre/uri (2.2.1): Downloading (100%)
Installing sabre/xml (2.2.3): Downloading (100%)
Generating autoload files
Error on execution :

php restore.php
Collection files to restore
PHP Fatal error: Uncaught Sabre\Xml\ParseException: The input element to parse is empty. Do not attempt to parse in /home/desktop/restore/vendor/sabre/xml/lib/Service.php:122
Stack trace:
#0 /home/desktop/restore/lib/RestoreTrash.php(59): Sabre\Xml\Service->parse(false)
#1 /home/desktop/restore/lib/RestoreTrash.php(25): RestoreTrash->collectTrashbinData()
#2 /home/desktop/restore/restore.php(11): RestoreTrash->run()
#3 {main}
thrown in /home/desktop/restore/vendor/sabre/xml/lib/Service.php on line 122

[AlexAndBear]

@EdouardRt some permission errors, please try to follow the steps here:
https://askubuntu.com/questions/1077879/cannot-create-cache-directory-home-user-composer-cache-repo-https-packagi

[EdouardRt]

thank you for your help @janackermann ! it does go through the cache warning, but the error while restoring remains :

sudo -u www-data php restore.php
Collection files to restore
PHP Fatal error:  Uncaught Sabre\Xml\ParseException: The input element to parse is empty. Do not attempt to parse in /home/desktop/restore/vendor/sabre/xml/lib/Service.php:122
Stack trace:
#0 /home/desktop/restore/lib/RestoreTrash.php(59): Sabre\Xml\Service->parse(false)
#1 /home/desktop/restore/lib/RestoreTrash.php(25): RestoreTrash->collectTrashbinData()
#2 /home/desktop/restore/restore.php(10): RestoreTrash->run()
#3 {main}
  thrown in /home/desktop/restore/vendor/sabre/xml/lib/Service.php on line 122

My detailed configuration is available here.
Meanwhile, i wander if the naming of elements in my trashbin is "normal" ie. IMG_0550.JPG.d1607618465 ?

[AlexAndBear]

@EdouardRt
Did you changed the variables in the restore.php as written in documentation?
https://github.com/JanAckermann/owncloud-restore-trash

This error usually occures if the script can't connect to the server

Meanwhile, i wander if the naming of elements in my trashbin is "normal" ie. IMG_0550.JPG.d1607618465 ?

Totaly "normal", please don't try to rename them

[EdouardRt]

@janackermann Yes, variables are defined, all 4 between single quotes : 'name' ; 'https://website.xt' ; 'p@sword' ; 'yyyy-mm-dd' .
I know i have a problem with the https certificate which needs browsing through chrome to hit the "proceed anyway" button.
the windows client connects though, but maybe a php command could be weaker ?

[AlexAndBear]

@EdouardRt please try the following:

1. Get the new version of the restore script here
2. In restore.php set ignore SSL to true (without quotes)
3. Run the script again

[EdouardRt]

Thanks for the updated script !
Feeling sorry @janackermann : getting an error (404 !?) :-(
and can't find any log that would help /var/log/syslog or /php7.0-fpm.log or /apache2/log ...

sudo -u www-data php restore.php
Collection files to restore
ERROR: The requested URL returned error: 404 Not FoundPHP Fatal error:  Uncaught Sabre\Xml\ParseException: The input element to parse is empty. Do not attempt to parse in /home/desktop/restore/owncloud-restore-trash-master/vendor/sabre/xml/lib/Service.php:122
Stack trace:
#0 /home/desktop/restore/owncloud-restore-trash-master/lib/RestoreTrash.php(74): Sabre\Xml\Service->parse(false)
#1 /home/desktop/restore/owncloud-restore-trash-master/lib/RestoreTrash.php(27): RestoreTrash->collectTrashbinData()
#2 /home/desktop/restore/owncloud-restore-trash-master/restore.php(11): RestoreTrash->run()
#3 {main}
  thrown in /home/desktop/restore/owncloud-restore-trash-master/vendor/sabre/xml/lib/Service.php on line 122

[AlexAndBear]

@EdouardRt you are welcome!

404 usually means that the requested resource does not exist.
<your-server>/remote.php/dav/trash-bin
should be callable via browser with the output:

This is the WebDAV interface. It can only be accessed by WebDAV clients such as the ownCloud desktop sync client.

Is the server reachable from the client you want to execute the script?

There is a new version of the script, please download the new version, do a composer install
and try to run the command like this:

➜ php restore.php --url="https://owncloud.local" --username="admin" --password="admin" --date="2020-12-08" 

An additional / at the end of the url can lead to problems

[EdouardRt]

@janackermann
Weird :
https://server.xt/remote.php/dav/trash-bin
Sends an owncloud error page with "Not Found File not found: trash-bin in 'root'"

While :
https://server.xt/remote.php/dav/
Sends the error you mention : "This is the WebDAV interface. It can only be accessed by WebDAV clients such as the ownCloud desktop sync client."

Trying the updated script right away.

[AlexAndBear]

@EdouardRt

Sends an owncloud error page with "Not Found File not found: trash-bin in 'root'"
This must be some server configuration problem,

in order to get the script run this must be fixed

Edit

https://server.xt/remote.php/dav/trash-bin/<your_username>
should work

[EdouardRt]

@janackermann
I do not get it :-(
/dav/files/user >> works fine
but
/dav/trash-bin/user >> does not (owncloud error "file not found")

I tried an occ repair > no change
An upgrade > already latest version

Browsing through the files, i see them in data/user/files_trashbin/files
But /dav/trash-bin won't grant access to it.

Any hint on where to look ? i did not find any relevant case elsewhere...

[EdouardRt]

@janackermann
I was able to force an upgrade (sounds like my repo was outdated) and had to also force a php upgrade.
Now am able to run the script. : 37k files found... The restore process is ongoing.
Thank you very much for your help !
Edouard

[AlexAndBear]

@EdouardRt
Good to hear man!

[audouts]

The 2.7.2 client deleted hundreds of thousands of files from our server.

We could not access the "Trash" via the web interface:
This directory is unavailable, please check the logs or contact the administrator

The recovery script was unusable because it would take hundreds of days for it to recover the files.
(https://github.com/JanAckermann/owncloud-restore-trash)

I could not find a way to use propfind to get the file ID, which the ownCloud docs describe in the webDAV, TrashBin API. I made a minor change that allowed me to get the file IDs and then dumped a list with propfind. I then used the TrashBin API "restore" call and tried to restore the files.

Approximately 92% of the files succeeded. We now face other problems:

First, when I try the TrashBin API restore on one of the remaining files, it says that the file already exists. It does not. I cannot find it anywhere in the database (except the trash and the history, showing it was deleted). It's not in the filesystem.

Second, the web interface works now (presumably due to the lower number of files) but has no way of selecting files before/after a certain date. I don't want to restore files that were intentionally deleted before the client bug started deleting everything.

I decided to try using the TrashBin API to delete files from the trash, before a certain date. That brings me to the third problem. Apparently, I can only delete one file at a time via the API. I get an error about it being "locked" if I try to run concurrent calls. It's taking 70-80 seconds per delete, which means it will take days to complete this relatively small operation.

What can we do to solve this?

[dannyrobertson]

This is still an issue even with Windows client v2.7.6:
When setting up a new Windows machine, I downloaded and installed the latest client and connected to my existing ownCloud instance.
Afterwards I spent all day recovering files.

[TheOneRing]

Could you open a new issue and fill in the details?
Ideally I'd also need some logs.

[virginistechdev]

Thanks Edouard for sharing this issue and the fix.
Just a simple testimony if that can help other people : I've been through the same issue, on a Debian 9 system with PHP 7.0 . 10K files deleted by a windows client.
I had to upgrade to Debian 10, then upgrade PHP to 7.3 then upgrade owncloud.
Then I ran the script , which ran well on its side, but unfortunately on the server side, I had for each file to restore an error HTTP 405 METHOD NOT ALLOWED for the the "MOVE" operation :
10.100.1.1 - - [29/Jul/2021:07:38:33 -0400] "MOVE /owncloud//owncloud/remote.php/dav/trash-bin/myuser/16488 HTTP/1.1" 405 697 "-" "-"

at the end, the files didn't get restored. I don't know if this is because :
a) I ran the script on the server itself ?
b) I have this strange "/owncloud//owncloud" at the beginning of the URL =>>> looks like this is due to a trailing "/" at the end of the --url parameter, i did not read enough this page, it's quoted above

Anyway, I found on the new owncloud web interface a "select all" checkbox in the "Trash" page. I just clicked it and pressed Restore. The operation takes a while, but it finally works.
Files restored, situation saved.

Thanks Jan Akermann, thanks owncloud team, thanks Edouard.

[kanebrands]

Does this assume everyone saves their users passwords so they can run these scripts? I don't like having to ask users for their passwords. Is there no system wide deleted files access that an admin can access and do bulk restores from? This seems like a huge oversight!

As a side note I'm also spending today restoring files that were deleted as a result of VFS being enable on a clients system. Client is using 2.9.1 and is moving from non-VFS to VFS. It looks like the client decided that all the files the client has not previously synced didn't need to be on the ownCloud server either. User made the change last night and managed to delete well over a TB of data. Enormous pain in the ***.

[hodyroff]

No, you can also use impersonate as an admin to do what is needed here. The script also doesn't need the user PWs ... but as stated above it is slow.

If you have a reproduceable case - how it is still possible to delete by accident with 2.9.1 we would be happy to look into it and try to find a fix, please open a new issue in that case. Sorry vor any invoncinience.

If the user has pointed ownCloud to an existing sync folder where selective sync was activated, this sounds possible, but there is a clear warning. We do recommend branding and using MECM or similar for larger instances to avoid any user interaction mistakes. Ideas for better usability always welcome as well. Thanks for your feedback!

[brozkeff]

I am afraid this issue occured to us somehow again with the latest desktop client on one of our few Windows 11 computers last week. The user whose credentials were logged by ownCloud server is not aware at all of manipulating in any way with ownCloud files in the last weeks yet basically all the contents of all the folders shared to that user vanished last week from the server, residing in the recycle bins of the dozens of users owning various shared folders.

Win11 Czech Edition Pro, ownCloud desktop client 2.10.1, server 10.9.1 on Ubuntu Server 18.04.

Unfortunately I had logging limited to errors and not notices etc so no data about the DELETE commands or so from the owncloud.log are available. The only mention I see is in the Activity tab where it is saved which particular user was responsible for the mass deletion.

[dragotin]

Could you check if you have a file called .owncloudsync.log in the top of your sync directory on the Windows machine and check if files you lost are mentioned in there? Can you make that logfile or parts of it available?

[dragotin]

And, @brozkeff, just to be sure: The sync uses Virtual Files (VFS), correct?

[brozkeff]

I am sorry but it was a computer of my colleague who asked me to disable the owncloud sync completely to not mess with the files, since she did not need the files to be synced at all for her work.
Unfortunately we deleted then the entire ownCloud folder from the computer therefore including the .owncloudsync.log if it was there :-(
Yes, the sync used Virtual Files (VFS) and I have a feeling it was related to it and some regression related to Windows 11.
There are dozens of other users using ownCloud sync with VFS enabled but almost all other employees use Windows 10 Pro, not Windows 11. This was perhaps the only user on our network with W11.

At least I discovered Apache access logs and detected the very first files being deleted, however nothing fancy to see there, just the DELETE commands sent by the client (IP and username redacted):

│81.x.x.x - mxxxxxx.axxxxxxxxx [24/May/2022:06:55:55 +0000] "DELETE /remote.php/dav/files/mxxxxxx.axxxxxxxxx/GRAFIKA-ostatni/ADR%20prelepky/img/ADR_3.png HTTP/1.1" 204 5909 "-" "Mozilla/5.0 (Windows) mirall/2.10.1 (build 7187) (ownCloud, windows-10.0.22000 ClientA│
│81.x.x.x - mxxxxxx.axxxxxxxxx [24/May/2022:06:55:55 +0000] "DELETE /remote.php/dav/files/mxxxxxx.axxxxxxxxx/GRAFIKA-ostatni/ADR%20prelepky/img/adr_3-45deg.png HTTP/1.1" 204 5909 "-" "Mozilla/5.0 (Windows) mirall/2.10.1 (build 7187) (ownCloud, windows-10.0.22000 C│
│81.x.x.x - mxxxxxx.axxxxxxxxx [24/May/2022:06:55:55 +0000] "DELETE /remote.php/dav/files/mxxxxxx.axxxxxxxxx/GRAFIKA-ostatni/ADR%20prelepky/img/blackandwhite.png HTTP/1.1" 204 5909 "-" "Mozilla/5.0 (Windows) mirall/2.10.1 (build 7187) (ownCloud, windows-10.0.22000│
│81.x.x.x - mxxxxxx.axxxxxxxxx [24/May/2022:06:55:55 +0000] "DELETE /remote.php/dav/files/mxxxxxx.axxxxxxxxx/GRAFIKA-ostatni/ADR%20prelepky/img/ADR_3.svg HTTP/1.1" 204 5909 "-" "Mozilla/5.0 (Windows) mirall/2.10.1 (build 7187) (ownCloud, windows-10.0.22000 ClientA│
│81.x.x.x - mxxxxxx.axxxxxxxxx [24/May/2022:06:55:55 +0000] "DELETE /remote.php/dav/files/mxxxxxx.axxxxxxxxx/GRAFIKA-ostatni/ADR%20prelepky/img/instrukce.png HTTP/1.1" 204 5909 "-" "Mozilla/5.0 (Windows) mirall/2.10.1 (build 7187) (ownCloud, windows-10.0.22000 Cli│
│81.x.x.x - mxxxxxx.axxxxxxxxx [24/May/2022:06:55:55 +0000] "DELETE /remote.php/dav/files/mxxxxxx.axxxxxxxxx/GRAFIKA-ostatni/ADR%20prelepky/ADR-3%20lepit%20o%2045stupnu%20natocene%204x.pdf HTTP/1.1" 204 5909 "-" "Mozilla/5.0 (Windows) mirall/2.10.1 (build 7187) (o│
│81.x.x.x - mxxxxxx.axxxxxxxxx [24/May/2022:06:55:55 +0000] "DELETE /remote.php/dav/files/mxxxxxx.axxxxxxxxx/GRAFIKA-ostatni/ADR%20prelepky/ADR-3%20lepit%20o%2045stupnu%20natocene.glabels HTTP/1.1" 204 574 "-" "Mozilla/5.0 (Windows) mirall/2.10.1 (build 7187) (own│
│81.x.x.x - mxxxxxx.axxxxxxxxx [24/May/2022:06:55:55 +0000] "DELETE /remote.php/dav/files/mxxxxxx.axxxxxxxxx/GRAFIKA-ostatni/ADR%20prelepky/ADR-cernobily%20kosoctverec%20lepit%20o%2045stupnu%20natocene%204x.pdf HTTP/1.1" 204 574 "-" "Mozilla/5.0 (Windows) mirall/2│
│81.x.x.x - mxxxxxx.axxxxxxxxx [24/May/2022:06:55:55 +0000] "DELETE /remote.php/dav/files/mxxxxxx.axxxxxxxxx/GRAFIKA-ostatni/ADR%20prelepky/ADR-cernobily%20kosoctverec%20lepit%20o%2045stupnu%20natocene.glabels HTTP/1.1" 204 574 "-" "Mozilla/5.0 (Windows) mirall/2.│
│81.x.x.x - mxxxxxx.axxxxxxxxx [24/May/2022:06:55:55 +0000] "DELETE /remote.php/dav/files/mxxxxxx.axxxxxxxxx/GRAFIKA-ostatni/ADR%20prelepky/UN%201170.docx HTTP/1.1" 204 574 "-" "Mozilla/5.0 (Windows) mirall/2.10.1 (build 7187) (ownCloud, windows-10.0.22000 ClientA│
│81.x.x.x - - [24/May/2022:06:55:56 +0000] "GET /status.php HTTP/1.1" 200 6301 "-" "Mozilla/5.0 (Windows) mirall/2.10.1 (build 7187) (ownCloud, windows-10.0.19044 ClientArchitecture: x86_64 OsArchitecture: x86_64)"                                                  │

[dragotin]

@brozkeff thanks for your help. Let's keep the eyes open on this one.