Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: cleanup_local_files also deleting files present online #2612

Closed
robertschulze opened this issue Jan 29, 2024 · 7 comments
Closed

Bug: cleanup_local_files also deleting files present online #2612

robertschulze opened this issue Jan 29, 2024 · 7 comments
Labels
Bug Something isn't working Fixed
Milestone

Comments

@robertschulze
Copy link

robertschulze commented Jan 29, 2024

Describe the bug

I use onedrive in conjunction in a download_only = "true" configuration. I want to make sure that files deleted online are also deleted locally. However, when selecting the option cleanup_local_files = "true" all files are being deleted, regardless of whether they are present online or not. This happens for normal as well as --resync type synchronizations.

Operating System Details

Linux atom 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy

Client Installation Method

From Source

OneDrive Account Type

Personal

What is your OneDrive Application Version

v2.5.0-alpha-5 GitHub version: v2.4.25-45-gc8d29c8

What is your OneDrive Application Configuration

sync_dir = "/media/3/3 - OneDrive/OneDrive_robert@guitaronline.de"                                                                                                                
monitor_interval = "300"                                                                                                                                                          
log_dir = "/var/log/onedrive/"                                                                                                                                                    
upload_only = "false"                                                                                                                                                             
check_nomount = "false"                                                                                                                                                           
check_nosync = "false"                                                                                                                                                            
download_only = "true"                                                                                                                                                            
cleanup_local_files = "true"                                                                                                                                                      
disable_notifications = "false"                                                                                                                                                   
disable_upload_validation = "false"                                                                                                                                               
enable_logging = "true"                                                                                                                                                           
force_http_11 = "false"                                                                                                                                                           
local_first = "false"                                                                                                                                                             
no_remote_delete = "false"                                                                                                                                                        
skip_symlinks = "false"                                                                                                                                                           
debug_https = "false"                                                                                                                                                             
skip_dotfiles = "false"                                                                                                                                                           
dry_run = "false"                                                                                                                                                                 
min_notify_changes = "5"                                                                                                                                                          
monitor_log_frequency = "5"                                                                                                                                                       
monitor_fullscan_frequency = "10"                                                                                                                                                 
sync_root_files = "false"                                                                                                                                                         
classify_as_big_delete = "1000"                                                                                                                                                   
user_agent = ""                                                                                                                                                                   
remove_source_files = "false"                                                                                                                                                     
skip_dir_strict_match = "false"                                                                                                                                                   
application_id = ""                                                                                                                                                               
resync = "false"                                                                                                                                                                  
bypass_data_preservation = "false"                                                                                                                                                
azure_ad_endpoint = ""                                                                                                                                                            
azure_tenant_id = "common"                                                                                                                                                        
sync_dir_permissions = "700"                                                                                                                                                      
sync_file_permissions = "600"                                                                                                                                                     
rate_limit = "125000000"                                                                                                                                                          
operation_timeout = "3600"                                                                                                                                                        
webhook_enabled = "false"                                                                                                                                                         
webhook_public_url = ""                                                                                                                                                           
webhook_listening_host = ""                                                                                                                                                       
webhook_listening_port = "8888"                                                                                                                                                   
webhook_expiration_interval = "86400"                                                                                                                                             
webhook_renewal_interval = "43200"                                                                                                                                                
skip_file = "~*|.~*|*.tmp"                                                                                                                                                        
skip_dir = ""

What is your 'curl' version

curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.16
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Network

What are all your system 'mount points'

//hyperv/3 on /media/3 type cifs (rw,relatime,vers=2.1,cache=strict,username=Administrator,uid=1000,noforceuid,gid=100,noforcegid,addr=192.168.3.11,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1)

What are all your local file system partition types

NAME                  FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
loop0                 squashfs    4.0                                                         0   100% /snap/canonical-livepatch/246
loop1                 squashfs    4.0                                                         0   100% /snap/bare/5
loop2                 squashfs    4.0                                                         0   100% /snap/canonical-livepatch/248
loop3                 squashfs    4.0                                                         0   100% /snap/core/16091
loop4                 squashfs    4.0                                                         0   100% /snap/core/16202
loop5                 squashfs    4.0                                                         0   100% /snap/core18/2796
loop6                 squashfs    4.0                                                         0   100% /snap/core18/2812
loop7                 squashfs    4.0                                                         0   100% /snap/core20/1974
loop8                 squashfs    4.0                                                         0   100% /snap/core20/2015
loop9                 squashfs    4.0                                                         0   100% /snap/core22/1033
loop10                squashfs    4.0                                                         0   100% /snap/core22/864
loop11                squashfs    4.0                                                         0   100% /snap/cups/974
loop12                squashfs    4.0                                                         0   100% /snap/cups/980
loop13                squashfs    4.0                                                         0   100% /snap/firefox/3358
loop14                squashfs    4.0                                                         0   100% /snap/firefox/3416
loop15                squashfs    4.0                                                         0   100% /snap/gnome-3-26-1604/104
loop16                squashfs    4.0                                                         0   100% /snap/gnome-3-26-1604/111
loop17                squashfs    4.0                                                         0   100% /snap/gnome-3-28-1804/194
loop18                squashfs    4.0                                                         0   100% /snap/gnome-3-28-1804/198
loop19                squashfs    4.0                                                         0   100% /snap/gnome-3-34-1804/90
loop20                squashfs    4.0                                                         0   100% /snap/gnome-3-34-1804/93
loop21                squashfs    4.0                                                         0   100% /snap/gnome-3-38-2004/140
loop22                squashfs    4.0                                                         0   100% /snap/gnome-3-38-2004/143
loop23                squashfs    4.0                                                         0   100% /snap/gnome-42-2204/132
loop24                squashfs    4.0                                                         0   100% /snap/gnome-42-2204/141
loop25                squashfs    4.0                                                         0   100% /snap/gnome-system-monitor/184
loop26                squashfs    4.0                                                         0   100% /snap/gnome-system-monitor/186
loop27                squashfs    4.0                                                         0   100% /snap/gtk-common-themes/1534
loop28                squashfs    4.0                                                         0   100% /snap/gtk-common-themes/1535
loop29                squashfs    4.0                                                         0   100% /snap/snap-store/638
loop30                squashfs    4.0                                                         0   100% /snap/snap-store/959
loop31                squashfs    4.0                                                         0   100% /snap/snapd-desktop-integration/57
loop32                squashfs    4.0                                                         0   100% /snap/snapd-desktop-integration/83
sda                                                                                                    
??sda1                vfat        FAT32          62EB-8E64                               503,3M     1% /boot/efi
??sda2                                                                                                 
??sda5                LVM2_member LVM2 001       X34xd2-HOp4-sAHk-pudA-CzVG-rENF-FOtrO2                
  ??ubuntu--vg-root   ext4        1.0            43f5fc57-8370-4315-b5ba-95912f21e113     24,7G    70% /var/snap/firefox/common/host-hunspell
  ?                                                                                                    /media/robert_home
  ?                                                                                                    /
  ??ubuntu--vg-swap_1 swap        1              ad32bbec-507e-44b0-bb5f-1df3ee647a2a                  [SWAP]

How do you use 'onedrive'

./onedrive --confdir='/home/robert/.config/onedrive/accounts/robert@guitaronline.de' --synchronize --verbose --verbose --resync > debug_output.log 2>&1

Steps to reproduce the behaviour

this was my initial test:

  1. initial sync with cleanup_local_files = true
  2. --resync with with cleanup_local_files = true --> all files deleted :-(

so I tested further:
3. initial sync with cleanup_local_files = false
4. --resync with cleanup_local_files = false --> all good

so I created two test files:

  • TestWordfile.docx is a file that is stored online (and should thus be kept)
  • test.txt is a file that is stored locally only (and should thus be deleted)
  1. --resync with cleanup_local_files = false --> all good, test.txt is not propagated to online
  2. simple sync with cleanup_local_files = false --> all good, test.txt is not propagated to online
  3. simple sync with cleanup_local_files = true --> all files being deleted :-(

In below log extract one can see that

  • TestWordfile.docx is correctly considered a Remote Object but wrongfully considered not present online (DEBUG: Selecting Record that is NOT Remote Object)
  • test.txt is wrongfully considered a Remote Object (Record is a Remote Object) but correctly considered not present online

Complete Verbose Log Output

DEBUG: ------------------------------------------------------------------
DEBUG: Record is a Remote Object: Item("9ba6f07668ba1ff7", "9BA6F07668BA1FF7!22186", "OneDrive - Constanze", "", remote, "aOUJBNkYwNzY2OEJBMUZGNyEyMjE4Ni40NA", "adDo5QkE2RjA3NjY4QkExRkY3ITIyMTg2LjYzODQxOTA4NDM4NTYwMDAwMA", 2024-Jan-14 22:00:39.74Z, "9BA6F07668BA1FF7!103", "", "", "f3028bd758552faf", "F3028BD758552FAF!4690", "Y", "")
DEBUG: Selecting Record that is NOT Remote Object: Item("f3028bd758552faf", "F3028BD758552FAF!4690", "OneDrive - Constanze", "", dir, "aRjMwMjhCRDc1ODU1MkZBRiE0NjkwLjQw", "adDpGMzAyOEJENzU4NTUyRkFGITQ2OTAuNjM4NDE5MDg0MTMxMDAwMDAw", 2024-Jan-14 22:00:39.74Z, "F3028BD758552FAF!311", "", "", "", "", "Y", "")
DEBUG: Checking file: ./OneDrive - Constanze/Datenbanken/Test.txt
DEBUG: skip_file evaluation for: /OneDrive - Constanze/Datenbanken/Test.txt
DEBUG: Evaluation against 'sync_list' for this path: OneDrive - Constanze/Datenbanken/Test.txt
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: OneDrive - Constanze/Datenbanken/*
DEBUG: Evaluation against 'sync_list' result: wildcard pattern match
DEBUG: Evaluation against 'sync_list' entry: Datenbanken/*
DEBUG: Evaluation against 'sync_list' result: wildcard pattern match
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: included for sync
DEBUG: Removing local file as --download-only & --cleanup-local-files configured
DEBUG: Removing local file: ./OneDrive - Constanze/Datenbanken/Test.txt
DEBUG: ------------------------------------------------------------------
DEBUG: Record is a Remote Object: Item("9ba6f07668ba1ff7", "9BA6F07668BA1FF7!22186", "OneDrive - Constanze", "", remote, "aOUJBNkYwNzY2OEJBMUZGNyEyMjE4Ni40NA", "adDo5QkE2RjA3NjY4QkExRkY3ITIyMTg2LjYzODQxOTA4NDM4NTYwMDAwMA", 2024-Jan-14 22:00:39.74Z, "9BA6F07668BA1FF7!103", "", "", "f3028bd758552faf", "F3028BD758552FAF!4690", "Y", "")
DEBUG: Selecting Record that is NOT Remote Object: Item("f3028bd758552faf", "F3028BD758552FAF!4690", "OneDrive - Constanze", "", dir, "aRjMwMjhCRDc1ODU1MkZBRiE0NjkwLjQw", "adDpGMzAyOEJENzU4NTUyRkFGITQ2OTAuNjM4NDE5MDg0MTMxMDAwMDAw", 2024-Jan-14 22:00:39.74Z, "F3028BD758552FAF!311", "", "", "", "", "Y", "")
DEBUG: Checking file: ./OneDrive - Constanze/Datenbanken/TestWordfile.docx
DEBUG: skip_file evaluation for: /OneDrive - Constanze/Datenbanken/TestWordfile.docx
DEBUG: Evaluation against 'sync_list' for this path: OneDrive - Constanze/Datenbanken/TestWordfile.docx
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: OneDrive - Constanze/Datenbanken/*
DEBUG: Evaluation against 'sync_list' result: wildcard pattern match
DEBUG: Evaluation against 'sync_list' entry: Datenbanken/*
DEBUG: Evaluation against 'sync_list' result: wildcard pattern match
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: included for sync
DEBUG: Removing local file as --download-only & --cleanup-local-files configured
DEBUG: Removing local file: ./OneDrive - Constanze/Datenbanken/TestWordfile.docx
DEBUG: ------------------------------------------------------------------

Screenshots

No response

Other Log Information or Details

I will send full logs for all above steps via mail

Additional context

No response

@robertschulze robertschulze added the Bug Something isn't working label Jan 29, 2024
@abraunegg
Copy link
Owner

@robertschulze

It will take me a few days to review - as per my prior email I am unable to reproduce this, so I will have to re look at your steps to see if this can be reproduced.

By still having a folder online with a misleading name, it is potentially getting tripped up with that ... ie - a Shared Folder with same name as a non-shared folder .. and potentially this is why it is being deleted.

The clue here is this statement:

is correctly considered a Remote Object but wrongfully considered not present online (DEBUG: Selecting Record that is NOT Remote Object)

This means that there is potentially 2 entries in the DB for that file - one that references your drive id, and the other referencing the shared folder drive id - which is why that message could be generated as there should only be one entry - the remote record.

Can you confirm that you have cleaned up all of your folders / shared folders / links and any other erroneous folders locally & online that were caused by the '%20' folder name problem .. as this would be a major contributor to the issue you are seeing.

@robertschulze
Copy link
Author

@abraunegg

Can you confirm that you have cleaned up all of your folders / shared folders / links and any other erroneous folders locally & online that were caused by the '%20' folder name problem .. as this would be a major contributor to the issue you are seeing.

Yes, I deleted all the old entries and created new ones already about two weeks ago. So unless they were again created wrongfully (but then sync would fail/show errors, wouldn't it?) it should be fine.

@abraunegg
Copy link
Owner

@robertschulze
No problem - please could you send me the item requested via email at your earliest convenience.

@robertschulze
Copy link
Author

@abraunegg
First delivery must have gotten stuck somewhere. Just resent. Let me know, if you got it.

@abraunegg abraunegg added the Fixed label Feb 3, 2024
@abraunegg
Copy link
Owner

@robertschulze

Many thanks for your patience and data. The issue has been found and corrected. Please refer to my email for further details.

@robertschulze
Copy link
Author

@abraunegg Perfect, I can confirm that the bug is solved. Thanks!

@abraunegg abraunegg added this to the v2.5.0 milestone Feb 4, 2024
@abraunegg
Copy link
Owner

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

Repository owner locked as resolved and limited conversation to collaborators Feb 12, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Bug Something isn't working Fixed
Projects
None yet
Development

No branches or pull requests

2 participants