Synchronize doesn't delete files and folders on Sharepoint library

[MichaKersloot]

Bug Report Details

I want to migrate to O365 and have setup some sharepoint libraries. Syncing works and everything is uploaded. But when on the source files are moved or deleted those files are not changed in SharePoint when a next synchronize is started. I'm using the --upload-only flag and when I omit this, the now extra files in SharePoint are going to be downloaded. I've tried --resync and that doesn't solve it.

Application and Operating System Details:

• OS: Debian jessie , Linux bak-files01 3.16.0-4-amd64 Fix config folder. #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux
• Without a gui
• Sharepoint
• build from source
• LDC - the LLVM D compiler (1.25.0)
• onedrive v2.4.11-5-gb7eedbd
  Client Configuration:

Configuration file successfully loaded
onedrive version                       = v2.4.11-5-gb7eedbd
Config path                            = /root/.config/onedrive/organisation/
Config file found in config path       = true
Config option 'check_nosync'           = false
Config option 'sync_dir'               = /home/shares/Organisation
Config option 'skip_dir'               =
Config option 'skip_file'              = ~*|.~*|*.tmp|*esktop.ini
Config option 'skip_dotfiles'          = false
Config option 'skip_symlinks'          = false
Config option 'monitor_interval'       = 300
Config option 'min_notify_changes'     = 5
Config option 'log_dir'                = /var/log/onedrive_organisation/
Config option 'classify_as_big_delete' = 100
Config option 'upload_only'            = false
Config option 'no_remote_delete'       = false
Config option 'remove_source_files'    = false
Config option 'drive_id'               = 
Config option 'sync_root_files'        = false
Selective sync 'sync_list' configured  = false
Business Shared Folders configured     = false

Curl Version:

curl 7.38.0 (x86_64-pc-linux-gnu) libcurl/7.38.0 OpenSSL/1.0.1k zlib/1.2.8 libidn/1.29 libssh2/1.4.3 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API SPNEGO NTLM NTLM_WB SSL libz TLS-SRP

• 'sync_dir' is a local directory
• bak--files01--vg-home ext4 8427f1d4-e74d-4b6e-b72e-dfaa5f9c5ed8 /home
• The onedrive directory is shared via samba to a windows terminal server

To Reproduce
onedrive --confdir=/root/.config/onedrive/organisation/ --synchronize --verbose --upload-only

Log file is mailed + screenshot to show de difference between local and SharePoint

[abraunegg]

@MichaKersloot
This situation sounds very odd, as, the only difference between a SharePoint site and a normal site, is the site identifier ..

Will spend some time tomorrow investigating and attempting to replicate

[MichaKersloot]

Hi @abraunegg

Thank you for your time on forehand. If you need anything, please don't hesitate to ask.

[abraunegg]

@MichaKersloot
I have attempted to replicate this using:

• Multiple SharePoint Libraries
• Document Libraries within SharePoint sites

and I cannot replicate nor find an issue.

When using --synchronize --verbose --upload-only files are:

• Correctly uploaded without issue
• Correctly deleted when deleted locally on next sync
• Local file renamed, file is correctly moved / renamed on SharePoint Library on next sync

The debug log / data you have also provided via email also does not show any deletion activity or any sort.

Please can you re-do your testing, but this time, please:

• Indicate the file that you have created locally
• Ensure that the debug log is capturing this file upload
• Delete the file / move the file locally
• Ensure that on the next sync, that you capture the application debug log output

Example for file delete:

Using 'user' Config Dir: /home/alex/.config/onedrive-sharepoint/
Using 'system' Config Dir: 
Configuration file successfully loaded
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Opening the item database ...
All operations will be performed in: /home/alex/OneDriveSharePoint
Application version: v2.4.11-5-gb7eedbd
Account Type: documentLibrary
Default Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6-dlZIIb6fSo7k4cc0dlrV
Default Root ID: 01PTBOWVV6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 27487247635200
Fetching details for OneDrive Root
OneDrive Root exists in the database
Initializing the Synchronization Engine ...
Syncing changes from local path only - NOT syncing data changes from OneDrive ...
Uploading differences of ~/OneDriveSharePoint
Processing .
The directory has not changed
Processing asdf.docx
The file has been deleted locally
Deleting item from OneDrive: asdf.docx
Uploading new items of ~/OneDriveSharePoint

Example for file move:

Using 'user' Config Dir: /home/alex/.config/onedrive-sharepoint/
Using 'system' Config Dir: 
Configuration file successfully loaded
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Opening the item database ...
All operations will be performed in: /home/alex/OneDriveSharePoint
Application version: v2.4.11-5-gb7eedbd
Account Type: documentLibrary
Default Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6-dlZIIb6fSo7k4cc0dlrV
Default Root ID: 01PTBOWVV6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 27487247636674
Fetching details for OneDrive Root
OneDrive Root exists in the database
Initializing the Synchronization Engine ...
Syncing changes from local path only - NOT syncing data changes from OneDrive ...
Uploading differences of ~/OneDriveSharePoint
Processing .
The directory has not changed
Processing asdf.docx
The file has been deleted locally
Deleting item from OneDrive: asdf.docx
Uploading new items of ~/OneDriveSharePoint
Uploading new file ./moved_file.docx ... 
Uploading 100% |oooooooooooooooooooooooooooooooooooooooo| DONE IN 00:00:00                                                                                                                    
done.
Remaining free space on OneDrive: 27487247636657

[MichaKersloot]

Hi,

thanx for you effort.

off course now it works for new files.:

1st run:
Uploading new file ./test_voor_micha.txt ...

moving the file to another folder

2nd run:
Processing test_voor_micha.txt
The file has been deleted locally
Deleting item from OneDrive: test_voor_micha.txt
...
Uploading new file ./personeel/test_voor_micha.txt ...

It looks like it also is deleting files that other users have moved. So some assumptions from me. File handling is done on basis of the local database? I know the application crashed several times in the last week, so I guess the local database is not up-to-date with the actual situation on Sharepoint. As the local database was somehow corrupted it didn't detect moved and/or deleted files and does not process them on Sharepoint. Resync picks up the new files and it looks like they are processed correctly.

How can I get Sharepoint in sync with the local system now? Seems resync doesn't take the Sharepoint side in account?

[abraunegg]

@MichaKersloot

It looks like it also is deleting files that other users have moved.

Deleting these where - locally or on OneDrive .. because they were present on your local file system and now they are not?

File handling is done on basis of the local database? I know the application crashed several times in the last week, so I guess the local database is not up-to-date with the actual situation on Sharepoint. As the local database was somehow corrupted it didn't detect moved and/or deleted files and does not process them on Sharepoint. Resync picks up the new files and it looks like they are processed correctly.

The application 100% relies on the local database. Also, if the application crashes - raise an issue, as it shouldn't crash, but at the same time the local DB cache file will not get corrupted. You only need to use --resync if the application tells you to, or you truly want to do a resync - otherwise just do not use it - stay clear way from this option.

The issue here is, in an upload only + resync scenario, the client is not fetching any remote state - as in, what files are present online, what files are not present online, what has been uploaded previously locally - the client has zero idea because --resync deletes the file that stores this data. Because of this, the client also has no idea what changes have been made locally between last sync and this sync, because the local DB is now empty. The only option here would be to perform a full local file system walk of local data to repopulate the local database, but that would be 100% time and resource intensive at application startup when using --upload-only --resync .. and is someone is always using --resync 100% of the the time against advice (and people do ...) this would negatively impact their application / system performance all the time.

How can I get Sharepoint in sync with the local system now? Seems resync doesn't take the Sharepoint side in account?

The --resync option does take the remote side into account, but you are explicitly telling the application to ignore the OneDrive state by using --upload-only

• Dont use --resync unless the application tells you to
• Dont use --upload-only ... really this should only be used when you are uploading data to a point as a backup or performing a 1 way sync of your data (no shared by other people)
• Use --monitor so that other peoples changes on SharePoint also get reflected locally on your system (add/moves/deletes/renames) and set up the init.d or systemd service.
• Use the application options to only sync the data that you are interested in, so that you are not replicating the entire SharePoint volume locally

So how can you get the state back to a true sync state in an easy way ... this 'might' work:

• Run the client in --monitor --upload-only --dry-run
• Run a script that touches all your files in your local data directory
  • This should trigger that the file has been modified locally, and update the local database
  • This will not however remove / move files that you have moved or deleted .. you will have to do that manually
• If this works in an acceptable manner, drop the --dry-run and perform the actions correctly

[abraunegg]

Closing issue as not a bug or issue with the client

[MichaKersloot]

Hi @abraunegg ,

thank you very much for you time to answer my problem! I hope I can clear things a bit up for 'documentation' reasons.

I was using the --upload-only option because we are in a migration project to Microsoft 365 and I had to make sure local data is leading. If I now remove the --upload-only flag data will duplicate because it exists on a different path in Sharepoint as it does local.

For me, the application stopped because it seemed to struggle with Desktop.ini files that somehow ended up in the dataset. I changed the config to exclude them and was then requested to apply the --resync option.

I'm in the lucky position I have enough local storage, so for now I'm starting over again by making a local snapshot and sync that to Sharepoint, run the application with --monitor and update the local snapshot. As it is part of a migration, this situation is only needed for a short time.

Thanks for the Idea of running --upload-only --dry-run as it could be used the other way around for me ;-) Maybe I can use the --download-only --dry-run to see what is currently in Sharepoint that shouldn't be there and if it is acceptable, delete those files from sharepoint and start uploading again.

For now it seems 2.4.11 seems to complete a full run where 2.4.10 did stop. But I've learned to catch the first error / problem and not to try to work around it.

Thanks again for your time and extensive explanation.

[abraunegg]

@MichaKersloot

I was using the --upload-only option because we are in a migration project to Microsoft 365 and I had to make sure local data is leading. If I now remove the --upload-only flag data will duplicate because it exists on a different path in Sharepoint as it does local.

I think this is a case of RTFM: https://github.com/abraunegg/onedrive/blob/master/docs/USAGE.md#performing-a-sync

For me, the application stopped because it seemed to struggle with Desktop.ini files that somehow ended up in the dataset. I changed the config to exclude them and was then requested to apply the --resync option.

The application would not have struggled with these files - they would have simply be excluded by default - see: #1277 (comment) .. so this would not have been your issue.

For now it seems 2.4.11 seems to complete a full run where 2.4.10 did stop. But I've learned to catch the first error / problem and not to try to work around it.

I do not support anyone running anything older than the current release of the client. v2.4.10 contains 13 bugs - some would have been impacting to SharePoint use - see: https://github.com/abraunegg/onedrive/releases/tag/v2.4.11. There are also 3 bugs fixed post this, and these also would be impacting to SharePoint use.

You are using 'master' at the moment, so at least you are using the right code for the client.

[MichaKersloot]

Hi,

I was running master all the time, but the project started some time ago. Sorry for being very cautious, but I really, really want to be sure the local files are not touched.

For I'm using a snapshot of the local data to get onedrive in sync. I find onedrive --monitor --verbose does stop after a few hours without any apparent reason. The echo $? gives me errorlevel 141. As soon as I've gathered more info I'll submit a new issue. The debug option (2x --verbose) generates too much data to have it running over a longer time, so I'm trying some options to get relevant data.

[abraunegg]

@MichaKersloot

For I'm using a snapshot of the local data to get onedrive in sync. I find onedrive --monitor --verbose does stop after a few hours without any apparent reason. The echo $? gives me errorlevel 141.

The client is most likely exiting due to this known issue:

https://github.com/abraunegg/onedrive/blob/master/docs/known-issues.md#application-stops-running-without-any-visible-reason

This is nothing I can fix, this is a network issue where the network connection between you and OneDrive fails, which the client is unable to recover from.

Please do a verbose debug log + HTTPS debugging to determine if your cause is the same using the following process: https://github.com/abraunegg/onedrive/wiki/Generate-https-debug-log-for-support

Usually unfortunately, each time this issue has been seen, most users are attempting to access a European Microsoft Data Centre. Given your location, I suspect this issue is the same one as mentioned above.

[MichaKersloot]

@abraunegg

thanks again for the prompt response. I'll try to fetch some debug logging, but as it only happens now and then, the logfiles are getting huge.

Would it be 'dangerous' to put the onedrive --monitor command in a loop with a sleep 60 to automatically restart it for the time being?

And yes, this would probably be the European Microsoft Data Centre ;-)

[abraunegg]

@MichaKersloot

Would it be 'dangerous' to put the onedrive --monitor command in a loop with a sleep 60 to automatically restart it for the time being?

I would refer to this as a work around for the client stopping without any visible reason: #753 (comment)

[MichaKersloot]

@abraunegg
The work around is what I meant, so that's implemented now ;-)

For de debug log. It ends with:

[DEBUG] Final True-Up: Do not perform a full walk of the OneDrive objects - not required
[DEBUG] Calling sync.applyDifferences(false); Applying changes of Path ID: 01AMIM3AV6Y2GOVW7725BZO354PW.......
[DEBUG] Request URL = https://graph.microsoft.com/v1.0/drives/.........n1JUe-9FThlqt_1BNpJueau3dmTs8ocJt7YVGpq........../?select=quota
* Connection 293 seems to be dead!
* Closing connection 293

As you seem to state this probably is a curl / ssl problem and this system is rather old I'm not sure you want to proceed on this one?

[abraunegg]

@MichaKersloot
Without seeing the debug log I cannot say 100%, but it is likely. What is the indicator in the log file will be something similar to this:

< X-Powered-By: ASP.NET
< MicrosoftSharePointTeamServices: 16.0.0.19722
< X-Content-Type-Options: nosniff
< X-MS-InvokeApp: 1; RequireReadOnly
< X-MSEdge-Ref: Ref A: E8EDDD652B2E49CD80F7A99572F3477A Ref B: BN3EDGE0412 Ref C: 2020-02-07T19:43:52Z
< Date: Fri, 07 Feb 2020 19:43:51 GMT
< 
* OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
* stopped the pause stream!
* Closing connection 2

Given however:

• Your geographic location
• The type of problem

It is most likely the same issue that plagues MS EU Data Centers (or there is some sort of ISP / Government MITM inspection going on ...)

If you system is also really old, it is certainly not worth pursuing this issue any further.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.