Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: Not all files being downloaded #2562

Closed
robertschulze opened this issue Dec 15, 2023 · 29 comments
Closed

Bug: Not all files being downloaded #2562

robertschulze opened this issue Dec 15, 2023 · 29 comments
Labels
Bug Something isn't working Fixed OneDrive API Bug OneDrive API Bug
Milestone

Comments

@robertschulze
Copy link

robertschulze commented Dec 15, 2023

Describe the bug

I am facing the same issue as @nijatismayilzada however, very strangely - only for one of my shared folders (I have four of them in my OneDrive). So while two appear to sync fine, one (which in Windows and on the Web is full of files) is containing only empty directories and whatever I tried in terms of (re)configuration, I was not able to make it sync. What could I be missing?

References to similar issues possibly related:
#1665
#563
#2550

Operating System Details

Linux atom 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy

Client Installation Method

From Source

OneDrive Account Type

Personal

What is your OneDrive Application Version

onedrive v2.4.25-1+np1

What is your OneDrive Application Configuration

onedrive version                             = v2.4.25-1+np1
Config path                                  = /home/robert/.config/onedrive
Config file found in config path             = false
Config option 'sync_dir'                     = /home/robert/OneDrive
Config option 'enable_logging'               = false
Config option 'log_dir'                      = /var/log/onedrive/
Config option 'disable_notifications'        = false
Config option 'min_notify_changes'           = 5
Config option 'skip_dir'                     = 
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 6
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = 
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = common
Config option 'user_agent'                   = 
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 600
Config option 'ip_protocol_version'          = 0
Config option 'sync_root_files'              = false
Selective sync 'sync_list' configured        = false
Config option 'sync_business_shared_folders' = false
Business Shared Folders configured           = false
Config option 'webhook_enabled'              = false

What is your 'curl' version

curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.16
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

//hyperv/3 on /media/3 type cifs (rw,relatime,vers=2.1,cache=strict,username=Administrator,uid=1000,noforceuid,gid=100,noforcegid,addr=192.168.3.11,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1)

What are all your local file system partition types

NAME                  FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
loop0                 squashfs    4.0                                                         0   100% /snap/canonical-livepatch/246
loop1                 squashfs    4.0                                                         0   100% /snap/bare/5
loop2                 squashfs    4.0                                                         0   100% /snap/canonical-livepatch/248
loop3                 squashfs    4.0                                                         0   100% /snap/core/16091
loop4                 squashfs    4.0                                                         0   100% /snap/core/16202
loop5                 squashfs    4.0                                                         0   100% /snap/core18/2796
loop6                 squashfs    4.0                                                         0   100% /snap/core18/2812
loop7                 squashfs    4.0                                                         0   100% /snap/core20/1974
loop8                 squashfs    4.0                                                         0   100% /snap/core20/2015
loop9                 squashfs    4.0                                                         0   100% /snap/core22/1033
loop10                squashfs    4.0                                                         0   100% /snap/core22/864
loop11                squashfs    4.0                                                         0   100% /snap/cups/974
loop12                squashfs    4.0                                                         0   100% /snap/cups/980
loop13                squashfs    4.0                                                         0   100% /snap/firefox/3358
loop14                squashfs    4.0                                                         0   100% /snap/firefox/3416
loop15                squashfs    4.0                                                         0   100% /snap/gnome-3-26-1604/104
loop16                squashfs    4.0                                                         0   100% /snap/gnome-3-26-1604/111
loop17                squashfs    4.0                                                         0   100% /snap/gnome-3-28-1804/194
loop18                squashfs    4.0                                                         0   100% /snap/gnome-3-28-1804/198
loop19                squashfs    4.0                                                         0   100% /snap/gnome-3-34-1804/90
loop20                squashfs    4.0                                                         0   100% /snap/gnome-3-34-1804/93
loop21                squashfs    4.0                                                         0   100% /snap/gnome-3-38-2004/140
loop22                squashfs    4.0                                                         0   100% /snap/gnome-3-38-2004/143
loop23                squashfs    4.0                                                         0   100% /snap/gnome-42-2204/132
loop24                squashfs    4.0                                                         0   100% /snap/gnome-42-2204/141
loop25                squashfs    4.0                                                         0   100% /snap/gnome-system-monitor/184
loop26                squashfs    4.0                                                         0   100% /snap/gnome-system-monitor/186
loop27                squashfs    4.0                                                         0   100% /snap/gtk-common-themes/1534
loop28                squashfs    4.0                                                         0   100% /snap/gtk-common-themes/1535
loop29                squashfs    4.0                                                         0   100% /snap/snap-store/638
loop30                squashfs    4.0                                                         0   100% /snap/snap-store/959
loop31                squashfs    4.0                                                         0   100% /snap/snapd-desktop-integration/57
loop32                squashfs    4.0                                                         0   100% /snap/snapd-desktop-integration/83
sda                                                                                                    
??sda1                vfat        FAT32          62EB-8E64                               503,3M     1% /boot/efi
??sda2                                                                                                 
??sda5                LVM2_member LVM2 001       X34xd2-HOp4-sAHk-pudA-CzVG-rENF-FOtrO2                
  ??ubuntu--vg-root   ext4        1.0            43f5fc57-8370-4315-b5ba-95912f21e113     24,7G    70% /var/snap/firefox/common/host-hunspell
  ?                                                                                                    /media/robert_home
  ?                                                                                                    /
  ??ubuntu--vg-swap_1 swap        1              ad32bbec-507e-44b0-bb5f-1df3ee647a2a                  [SWAP]

How do you use 'onedrive'

both from the OneDriveGUI as well as directly via the console (tried both --monitor and --synchronize)

Steps to reproduce the behaviour

run onedrive either from the command line with either command line option
CONSOLE.robert.onedrive.log

Complete Verbose Log Output

too large, I added it as file above

Screenshots

No response

Other Log Information or Details

please let me know in case I can provide additional information

Additional context

No response

@robertschulze robertschulze added the Bug Something isn't working label Dec 15, 2023
@abraunegg
Copy link
Owner

abraunegg commented Dec 15, 2023

@robertschulze

There could be any number of reasons why the data is empty.

Question: Are the empty / non-download items all in shared folders that you have added to your account online via "Add to my OneDrive"? If yes, then this could be path related as to how the client is looking for the files.

Now .. v2.4.x client code, if it has a bug, it will not be fixed.

What I urge you to do is try the very mature 'alpha-3' code (yes there are logging output problems, 'alpha-4' is about a week away) - please re-test your problem with 'alpha-3'

Please read:

@robertschulze
Copy link
Author

robertschulze commented Dec 15, 2023

Ok, tried to so but using the alpha-3 build yields the error

Invalid value for key in config file: drive_id
drive_id in config file cannot be empty - this is a fatal error and must be corrected

In 2.4 I did not need to set this value and --get-sharepoint-drive-id (previously --get-O365-drive-id) appears to work only for Business OneDrive.
How can I determine the drive_id to put in the config file for a personal OneDrive?

@abraunegg
Copy link
Owner

abraunegg commented Dec 16, 2023

You need to remove it from your config file for personal accounts

@abraunegg
Copy link
Owner

@robertschulze
Any update after modifying your configuration file to remove 'drive_id' ?

@abraunegg
Copy link
Owner

@robertschulze

What is your OneDrive Application Configuration

From the above, and your logfile, your output for your application configuration is not correct - as you indicate here that you are not using a synclist file, but your console log output shows that you are using a sync_list file. For example::

2023-Dec-15 23:33:52.6944675	Remote change discarded - not in sync scope
2023-Dec-15 23:33:53.6649434	Processing 200 OneDrive items to ensure consistent local state due to sync_list being used
2023-Dec-15 23:33:53.6703335	Skipping item - excluded by sync_list config: actualDirToBeSynced/Dokumente
2023-Dec-15 23:33:53.6703986	Remote change discarded - not in sync scope
2023-Dec-15 23:33:53.6704398	Remote change discarded - not in sync scope

So is your issue that items are not being downloaded due to a mis-configured 'sync_list' file?

Regarding this item: drive_id in config file cannot be empty

This 'config' problem stems from using 'OneDriveGUI' that incorrectly writes out empty configuration items. If you are using this, and using 'alpha-3' - you need to update your GUI to this: https://github.com/bpozdena/OneDriveGUI/releases/tag/v1.1.0alpha0

Please can you:

@robertschulze
Copy link
Author

robertschulze commented Dec 17, 2023

  • Provide an update as to using the updated 'config' file post removing 'drive_id'

Indeed, this solved the issue. I will try to follow up on the logs tonight.

@robertschulze
Copy link
Author

robertschulze commented Dec 17, 2023

Hi,
sorry, yes, I use a syncfile for debugging just the actual issue (my OneDrive is somewhat huge).

Unfortunately the alpha crashes, here the end of the log file (it contains all kinds of access tokens, so not sure whether it's safe to upload it here):

Syncing this OneDrive Personal Shared Folder: OneDrive - Constanze
[DEBUG] DeltaLink Query (driveId): f3028bd758552faf
[DEBUG] DeltaLink Query (id):      F3028BD758552FAF!4690
[DEBUG] Fetching /delta response from the OneDrive API for Drive ID: f3028bd758552faf
[DEBUG] Read token from appConfig
[DEBUG] Authorised State: true
[DEBUG] ------------------------------------------------------------------
[DEBUG] selectedDriveId:   f3028bd758552faf
[DEBUG] selectedItemId:    F3028BD758552FAF!4690
[DEBUG] providedDeltaLink: 
[DEBUG] ------------------------------------------------------------------
[DEBUG] Existing Microsoft OneDrive Access Token Expires: 2023-Dec-18 00:48:02.3539537
[DEBUG] Request URL = https://graph.microsoft.com/v1.0/drives/f3028bd758552faf/items/F3028BD758552FAF!4690/delta
[DEBUG] API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 201
[DEBUG] ------------------------------------------------------------------
[DEBUG] Processing OneDrive Item 1 of 201 from API Response Bundle 1
[DEBUG] Raw JSON OneDrive Item: {"@odata.type":"#microsoft.graph.driveItem","cTag":"adDpGMzAyOEJENzU4NTUyRkFGITQ2OTAuNjM4MzgxOTM5ODcyODcwMDAw","createdBy":{"application":{"id":"481710a4"},"user":{"displayName":"Robert Schulze","id":"9ba6f07668ba1ff7"}},"createdDateTime":"2023-11-06T20:37:13.387Z","eTag":"aRjMwMjhCRDc1ODU1MkZBRiE0NjkwLjI","fileSystemInfo":{"createdDateTime":"2023-11-06T20:37:13.386Z","lastModifiedDateTime":"2023-11-06T21:09:58.813Z"},"folder":{"childCount":0,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"F3028BD758552FAF!4690","lastModifiedBy":{"application":{"id":"481710a4"},"user":{"displayName":"Robert Schulze","id":"9ba6f07668ba1ff7"}},"lastModifiedDateTime":"2023-12-14T23:39:47.287Z","name":"OneDrive - Constanze","parentReference":{"driveId":"f3028bd758552faf","driveType":"personal","id":"F3028BD758552FAF!313","name":"Dokumente","path":"\/drives\/f3028bd758552faf\/items\/F3028BD758552FAF!313:"},"shared":{"owner":{"user":{"displayName":"Constanze Schulze","id":"f3028bd758552faf"}},"scope":"users","sharedDateTime":"2023-11-06T20:38:29.8602153Z"},"size":6224178783,"webUrl":"https:\/\/1drv.ms\/f\/s!AK8vVVjXiwLzpFI"}
[DEBUG] This item is not a OneDrive deletion change
[DEBUG] defaultRootId                                        = 9BA6F07668BA1FF7!103
[DEBUG] 'search id'                                          = F3028BD758552FAF!4690
[DEBUG] id == defaultRootId                                  = false
[DEBUG] isItemRoot(onedriveJSONItem)                         = false
[DEBUG] onedriveJSONItem['name'].str == 'root'               = false
[DEBUG] itemHasParentReferenceId                             = true
[DEBUG] Failure scope was called
[DEBUG] Running performStandardExitProcess due to: failureScope
[DEBUG] Shutdown OneDrive API instance
[DEBUG] No valid Microsoft OneDrive webhook subscription to delete
[DEBUG] Shutdown Sync Engine instance
[DEBUG] Shutdown Client Side Filtering instance
[DEBUG] Shutdown Application Configuration instance
[DEBUG] Shutdown Database instance
[DEBUG] Attempting to perform a database vacuum to merge any temporary data
[DEBUG] Database vacuum is complete
[DEBUG] Setting Class Objects to null due to failure scope
[DEBUG] Exit scope was called
[DEBUG] Running performStandardExitProcess due to: exitScope
core.exception.ArrayIndexError@src/sync.d(3279): index [1] is out of bounds for array of length 1
----------------
??:? onArrayIndexError [0x561fc2c341a6]
??:? _d_arraybounds_indexp [0x561fc2bd47b3]
src/sync.d:3312 bool syncEngine.SyncEngine.checkJSONAgainstClientSideFiltering(std.json.JSONValue) [0x561fc2b8fd46]
src/sync.d:1053 void syncEngine.SyncEngine.processDeltaJSONItem(std.json.JSONValue, ulong, int, ulong, bool) [0x561fc2b877d5]
src/sync.d:767 void syncEngine.SyncEngine.fetchOneDriveDeltaAPIResponse(immutable(char)[], immutable(char)[], immutable(char)[]) [0x561fc2b86698]
src/sync.d:550 void syncEngine.SyncEngine.syncOneDriveAccountToLocalDisk() [0x561fc2b85ab6]
src/main.d:569 _Dmain [0x561fc2a272a5]

Also tried --synchronize as well as --monitor, also with --resync added, but fails in all cases.
The item it fails on is a linked OneDrive from another person. The same that the old version did not manage to sync (either, but w/o crashing).
Thanks for having a look! :-)
Let me know if I can add anything more.

@abraunegg abraunegg added this to the v2.5.0 milestone Dec 18, 2023
abraunegg added a commit that referenced this issue Dec 18, 2023
* Potentially fix crash identified by #2562
@abraunegg
Copy link
Owner

@robertschulze

The item it fails on is a linked OneDrive from another person. The same that the old version did not manage to sync (either, but w/o crashing).

Many thanks for this output that identifies a previously unknown issue - greatly appreciated.

I have updated the 'alpha-3' code with a potential fix for this crash: index [1] is out of bounds for array of length 1

If you rebuild your 'alpha-3' code (you may need to delete the local git pull directory as well) and rebuild, you should get client version v2.5.0-alpha-3 GitHub version: v2.4.25-50-gc2e98ff

If you can test this version please that would be greatly appreciated as this should fix this initial issue you are having with 'alpha-3', but the positive here is that the client has found 201+ items to process from this Shared Folder (Processing OneDrive Item 1 of 201 from API Response Bundle 1) ... so potentially unlike the v2.4.x code, the 'alpha-3' client is seeing the data for this folder, thus this should be processed as required if the above error is now resolved.

@abraunegg
Copy link
Owner

@robertschulze
Any update as to testing the updated 'alpha-3' code?

@robertschulze
Copy link
Author

Hi,
the updated code works, thanks!
Unfortunately, the sync issue still persists although in the verbose log I see the following fragment:

[DEBUG] ------------------------------------------------------------------
[DEBUG] Processing OneDrive Item 4 of 434 from API Response Bundle 1
[DEBUG] Raw JSON OneDrive Item: {"@odata.type":"#microsoft.graph.driveItem","cTag":"adDo5QkE2RjA3NjY4QkExRkY3ITIxODI5LjYzODM4NTY5Mjk1OTYzMDAwMA","createdBy":{"application":{"id":"481710a4"},"user":{"displayName":"Robert Schulze","id":"9ba6f07668ba1ff7"}},"createdDateTime":"2023-11-06T20:52:05.083Z","eTag":"aOUJBNkYwNzY2OEJBMUZGNyEyMTgyOS4xOTg","id":"9BA6F07668BA1FF7!21829","lastModifiedBy":{"user":{"displayName":"Robert Schulze","id":"9ba6f07668ba1ff7"}},"lastModifiedDateTime":"2023-12-19T07:54:55.963Z","name":"OneDrive - Constanze","parentReference":{"driveId":"9ba6f07668ba1ff7","driveType":"personal","id":"9BA6F07668BA1FF7!103","name":"root:","path":"\/drive\/root:"},"remoteItem":{"fileSystemInfo":{"createdDateTime":"2023-11-06T20:37:13.3866667Z","lastModifiedDateTime":"2023-11-06T21:09:58.8133333Z"},"folder":{"childCount":13,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"F3028BD758552FAF!4690","parentReference":{"driveId":"f3028bd758552faf","driveType":"personal"},"shared":{"sharedDateTime":"2023-11-06T20:38:29.8602153Z"},"size":6224611033,"webUrl":"https:\/\/1drv.ms\/u\/s!AK8vVVjXiwLzpFI"},"webUrl":"https:\/\/1drv.ms\/u\/s!AK8vVVjXiwLzpFI"}
[DEBUG] This item is not a OneDrive deletion change
[DEBUG] defaultRootId                                        = 9BA6F07668BA1FF7!103
[DEBUG] 'search id'                                          = 9BA6F07668BA1FF7!21829
[DEBUG] id == defaultRootId                                  = false
[DEBUG] isItemRoot(onedriveJSONItem)                         = false
[DEBUG] onedriveJSONItem['name'].str == 'root'               = false
[DEBUG] itemHasParentReferenceId                             = true
[DEBUG] Attempting to calculate local filesystem path for 9ba6f07668ba1ff7 and 9BA6F07668BA1FF7!103
[DEBUG] sync_list item to check: ./OneDrive - Constanze
[DEBUG] Evaluation against 'sync_list' for this path: OneDrive - Constanze
[DEBUG] [S]exclude           = false
[DEBUG] [S]exludeDirectMatch = false
[DEBUG] [S]excludeMatched    = false
[DEBUG] Evaluation against 'sync_list' entry: OneDrive - Constanze
[DEBUG] exact path match
[DEBUG] Evaluation against 'sync_list' result: direct match
[DEBUG] [F]exclude           = false
[DEBUG] [F]exludeDirectMatch = false
[DEBUG] [F]excludeMatched    = false
[DEBUG] Evaluation against 'sync_list' final result: included for sync
[DEBUG] Adding this Raw JSON OneDrive Item to jsonItemsToProcess array for further processing
[DEBUG] ------------------------------------------------------------------

in the end, nothing is copied and the log ends:

[DEBUG] ------------------------------------------------------------------
[DEBUG] No valid Microsoft OneDrive webhook subscription to delete
[DEBUG] Finished processing /delta JSON response from the OneDrive API
[DEBUG] Number of JSON Objects received from OneDrive API:                 2836
[DEBUG] Number of JSON Objects already processed (root and deleted items): 2836
[DEBUG] Number of JSON items to process is: 0
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
[DEBUG] Updating completed deltaLink in DB to: https://graph.microsoft.com/v1.0/drives/f3028bd758552faf/items/F3028BD758552FAF!4690/delta?token=aTE09NjM4Mzg1NjkyMTA1MTc7SUQ9RjMwMjhCRDc1ODU1MkZBRiE0NjkwO0xSPTYzODM4NTgxNTM2MDUwO0VQPTIwO1NJPTUxO0RMRVA9MDtTRz0xO1NPPTI7UEk9Mw
Performing a database consistency and integrity check on locally stored data ... 
Processing DB entries for this Drive ID: 9ba6f07668ba1ff7
[DEBUG] Selecting DB items via itemDB.selectByDriveId(driveId)
[DEBUG] Database items to process for this driveId: 1
[DEBUG] Attempting to calculate local filesystem path for 9ba6f07668ba1ff7 and 9BA6F07668BA1FF7!103
Processing /media/3/3 - OneDrive/OneDrive_robert@guitaronline.de
The directory has not changed
[DEBUG] Attempting to calculate local filesystem path for 9ba6f07668ba1ff7 and 9BA6F07668BA1FF7!21829
Processing OneDrive - Constanze
Processing DB entries for this Drive ID: f3028bd758552faf
[DEBUG] Selecting DB items via itemDB.selectByDriveId(driveId)
[DEBUG] Database items to process for this driveId: 0

My command was ./onedrive --synchronize --verbose --verbose --resync > debug_output.log 2>&1

@abraunegg
Copy link
Owner

@robertschulze
I am going to need to see the entire debug log.

The items are being added to be processed, something is happening - but its not in your output there.

Please can you archive and email the entire debug log to the support email address.

@abraunegg
Copy link
Owner

@robertschulze
Regarding the question you ask, if you can send me a second email message to firstname dot lastname at the same domain. Once I receive that we can communicate via email to close this out.

@abraunegg abraunegg added OneDrive API Bug OneDrive API Bug and removed Bug Something isn't working labels Dec 20, 2023
@abraunegg
Copy link
Owner

After analysis of the debug log, this appears to be an API bug.

The JSON response from the OneDrive API includes erroneus HTML characters in JSON elements (%20) thus this is why when attempting to match the path for download, the download does not occur.

Raised an API Bug as the API specification does not articulate that it will send HTML characters embedded in JSON elements.

API Bug Reference: OneDrive/onedrive-api-docs#1765

There are 2 ways to solve the actual issue:

  • Microsoft fix the API Bug
  • Perform a folder rename on Microsoft OneDrive, and reshare between users

@robertschulze
Copy link
Author

robertschulze commented Dec 20, 2023

Hi,
I renamed the original folder from "OneDrive - Constanze" to "OneDrive-Constanze" to get rid of the spaces in a straightforward manner and better be able to debug the remaining issue. The name of the link to the original folder I left as "OneDrive - Constanze" to better pinpoint whether the rename of just the original folder solves the issue.
Unfortunately, it still does not work (I guess for another reason) even if I put both

OneDrive - Constanze/*
OneDrive-Constanze

into the sync_list.

For one, I still see %20 in the file paths below the main dir (now with the new WARNING you implemented):

[DEBUG] ------------------------------------------------------------------
[DEBUG] Processing OneDrive Item 33 of 201 from API Response Bundle 1
[DEBUG] Raw JSON OneDrive Item: {"@microsoft.graph.downloadUrl":"https:\/\/public.dm.files.1drv.com\/y4m7ea1rzglhOyaVF01layFpuiwzFqXbEKNubu_swp0joPIOXhBdbe5fNs6gn1VraY4nJufI5F5wmNlAakaDLoWF1IFYXu-1V305tfdto3uMcKl8LuxHPUI4vJHvdwNZbH1CyhEQeA88kFdv-GvUVTCm1C0MFznQxszKdWD9SmsYf3zAhHdSKotzfoks5EfHvwFWLD_xJv4w6tGYlDnBzdtjtpNyGI5dEEv_ErE83gvdUeaxKEAr_4QGR1IDSH3DOnj","@odata.type":"#microsoft.graph.driveItem","cTag":"aYzpGMzAyOEJENzU4NTUyRkFGITI4NDguMjU3","createdBy":{"application":{"id":"480728c5"},"device":{"id":"188006267c0904"},"oneDriveSync":{"@odata.type":"#microsoft.graph.identity","id":"0cf2e7c6-beb9-4dd9-aab2-ffff8d090000"},"user":{"displayName":"Robert Schulze","id":"9ba6f07668ba1ff7"}},"createdDateTime":"2022-11-13T23:14:13.923Z","eTag":"aRjMwMjhCRDc1ODU1MkZBRiEyODQ4Ljk","file":{"hashes":{"quickXorHash":"q+y+NODV3txUaKxep47mn8uzq7Q=","sha1Hash":"FF6D8542833DA5BAA3CE8ADA2AC6C4DFC249ED42","sha256Hash":"BA26279996C9F595F8EC160288AF07FCE2A990CF396FA20D850AC68D5C9722E1"},"mimeType":"image\/jpeg"},"fileSystemInfo":{"createdDateTime":"2022-11-13T23:14:13.923Z","lastModifiedDateTime":"2015-01-07T18:37:16Z"},"id":"F3028BD758552FAF!2848","image":{"height":3456,"width":5184},"lastModifiedBy":{"application":{"id":"4035b959"},"user":{"displayName":"Robert Schulze","id":"9ba6f07668ba1ff7"}},"lastModifiedDateTime":"2023-09-13T00:38:08.11Z","name":"IMG_4647.JPG","parentReference":{"driveId":"f3028bd758552faf","driveType":"personal","id":"F3028BD758552FAF!2733","name":"20150107 - Passbild","path":"\/drives\/f3028bd758552faf\/items\/F3028BD758552FAF!313:\/OneDrive-Constanze\/Fotos\/20150107%20-%20Passbild"},"pendingOperations":{"pendingContentUpdate":{"queuedDateTime":"2023-05-08T22:46:03.0314796Z"}},"photo":{"cameraMake":"Canon","cameraModel":"Canon EOS 550D","exposureDenominator":20.0,"exposureNumerator":1.0,"fNumber":5.59999999999999964,"focalLength":55.0,"iso":100,"orientation":1,"takenDateTime":"2015-01-07T19:37:15Z"},"reactions":{"commentCount":0},"size":3440417,"webUrl":"https:\/\/1drv.ms\/i\/s!AK8vVVjXiwLzliA"}
[DEBUG] This item is not a OneDrive deletion change
[DEBUG] defaultRootId                                        = 9BA6F07668BA1FF7!103
[DEBUG] 'search id'                                          = F3028BD758552FAF!2848
[DEBUG] id == defaultRootId                                  = false
[DEBUG] isItemRoot(onedriveJSONItem)                         = false
[DEBUG] onedriveJSONItem['name'].str == 'root'               = false
[DEBUG] itemHasParentReferenceId                             = true
[DEBUG] skip_file item to check (file name only - parent path not in database): /IMG_4647.JPG
[DEBUG] skip_file evaluation for: /IMG_4647.JPG
[DEBUG] skip_file evaluation for: /IMG_4647.JPG
[DEBUG] Result: false
CAUTION: Microsoft OneDrive API sent a JSON element containing HTML entities. This will cause issues performing pattern matching and potentially cause this path not to sync.
See: https://github.com/OneDrive/onedrive-api-docs/issues/1765 for further details
[DEBUG] sync_list item to check: OneDrive-Constanze/Fotos/20150107%20-%20Passbild/IMG_4647.JPG
[DEBUG] Evaluation against 'sync_list' for this path: OneDrive-Constanze/Fotos/20150107%20-%20Passbild/IMG_4647.JPG
[DEBUG] [S]exclude           = false
[DEBUG] [S]exludeDirectMatch = false
[DEBUG] [S]excludeMatched    = false
[DEBUG] Evaluation against 'sync_list' entry: OneDrive-Constanze
[DEBUG] Evaluation against 'sync_list' result: parental path match
[DEBUG] [F]exclude           = false
[DEBUG] [F]exludeDirectMatch = false
[DEBUG] [F]excludeMatched    = false
[DEBUG] Evaluation against 'sync_list' final result: included for sync
[DEBUG] Adding this Raw JSON OneDrive Item to jsonItemsToProcess array for further processing
[DEBUG] ------------------------------------------------------------------

but also none of the files are ultimately copied because of

[DEBUG] Number of JSON items to process is: 2839
Processing 2839 applicable changes and items received from Microsoft OneDrive 
Processing OneDrive JSON item batch [1/6] to ensure consistent local state
[DEBUG] ------------------------------------------------------------------
[DEBUG] Processing OneDrive JSON item 1 of 500 as part of JSON Item Batch 1 of 6
[DEBUG] Raw JSON OneDrive Item: {"@odata.type":"#microsoft.graph.driveItem","cTag":"adDpGMzAyOEJENzU4NTUyRkFGITQ2OTAuNjM4Mzg3MDc5MTIyNzMwMDAw","createdBy":{"application":{"id":"481710a4"},"user":{"displayName":"Robert Schulze","id":"9ba6f07668ba1ff7"}},"createdDateTime":"2023-11-06T20:37:13.387Z","eTag":"aRjMwMjhCRDc1ODU1MkZBRiE0NjkwLjIz","fileSystemInfo":{"createdDateTime":"2023-11-06T20:37:13.386Z","lastModifiedDateTime":"2023-12-20T22:25:12.273Z"},"folder":{"childCount":0,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"F3028BD758552FAF!4690","lastModifiedBy":{"application":{"id":"481710a4"},"user":{"displayName":"Constanze Schulze","id":"f3028bd758552faf"}},"lastModifiedDateTime":"2023-12-20T22:25:12.273Z","name":"OneDrive-Constanze","parentReference":{"driveId":"f3028bd758552faf","driveType":"personal","id":"F3028BD758552FAF!313","name":"Dokumente","path":"\/drives\/f3028bd758552faf\/items\/F3028BD758552FAF!313:"},"shared":{"owner":{"user":{"displayName":"Constanze Schulze","id":"f3028bd758552faf"}},"scope":"users","sharedDateTime":"2023-11-06T20:38:29.8602153Z"},"size":6224611033,"webUrl":"https:\/\/1drv.ms\/f\/s!AK8vVVjXiwLzpFI"}
[DEBUG] Parent ID is not in DB .. 
[DEBUG] Potential Shared Object Item: {"@odata.type":"#microsoft.graph.driveItem","cTag":"adDpGMzAyOEJENzU4NTUyRkFGITQ2OTAuNjM4Mzg3MDc5MTIyNzMwMDAw","createdBy":{"application":{"id":"481710a4"},"user":{"displayName":"Robert Schulze","id":"9ba6f07668ba1ff7"}},"createdDateTime":"2023-11-06T20:37:13.387Z","eTag":"aRjMwMjhCRDc1ODU1MkZBRiE0NjkwLjIz","fileSystemInfo":{"createdDateTime":"2023-11-06T20:37:13.386Z","lastModifiedDateTime":"2023-12-20T22:25:12.273Z"},"folder":{"childCount":0,"view":{"sortBy":"name","sortOrder":"ascending","viewType":"thumbnails"}},"id":"F3028BD758552FAF!4690","lastModifiedBy":{"application":{"id":"481710a4"},"user":{"displayName":"Constanze Schulze","id":"f3028bd758552faf"}},"lastModifiedDateTime":"2023-12-20T22:25:12.273Z","name":"OneDrive-Constanze","parentReference":{"driveId":"f3028bd758552faf","driveType":"personal","id":"F3028BD758552FAF!313","name":"Dokumente","path":"\/drives\/f3028bd758552faf\/items\/F3028BD758552FAF!313:"},"shared":{"owner":{"user":{"displayName":"Constanze Schulze","id":"f3028bd758552faf"}},"scope":"users","sharedDateTime":"2023-11-06T20:38:29.8602153Z"},"size":6224611033,"webUrl":"https:\/\/1drv.ms\/f\/s!AK8vVVjXiwLzpFI"}
[DEBUG] Flagging object as a directory
[DEBUG] The reported parentId is not in the database. This potentially is a shared folder as 'remoteItem.driveId' != 'appConfig.defaultDriveId'. Relevant Details: remoteItem.driveId (f3028bd758552faf), remoteItem.parentId (F3028BD758552FAF!313)
[DEBUG] Setting remoteItem.parentId to be null
[DEBUG] Update/Insert local database with remoteItem details with remoteItem.parentId as null: Item("f3028bd758552faf", "F3028BD758552FAF!4690", "OneDrive-Constanze", "", dir, "aRjMwMjhCRDc1ODU1MkZBRiE0NjkwLjIz", "adDpGMzAyOEJENzU4NTUyRkFGITQ2OTAuNjM4Mzg3MDc5MTIyNzMwMDAw", 2023-Dec-20 22:25:12.273Z, "", "", "", "", "", "Y", "")
[DEBUG] The item we are syncing is a folder
[DEBUG] Attempting to calculate local filesystem path for f3028bd758552faf and F3028BD758552FAF!313
[DEBUG] The following generated a broken tree query:
[DEBUG] Drive ID: f3028bd758552faf
[DEBUG] Item ID: F3028BD758552FAF!313
ERROR: A database consistency issue has been caught. A --resync is needed to rebuild the database.

I have sent the full log via mail.
Thanks for bearing with me on this strange issue,
Robert

@abraunegg
Copy link
Owner

abraunegg commented Dec 21, 2023

@robertschulze

Thanks for the updated log file, will look at this later.

For one, I still see %20 in the file paths below the main dir (now with the new WARNING you implemented):

Correct - because either the file online or this is a OneDrive API bug - you are getting %20 being sent by the OneDrive API ... your only option here is to rename offending files online.

ERROR: A database consistency issue has been caught. A --resync is needed to rebuild the database.

Unsure if you ran a --resync when using the updated code (the log file when I review will tell me) - if you didnt do this, this the reason is that it is failing is that the database tie record for the shared folder is missing, which is why the application is advising to do a --resync

If you did do a --resync then the application did not create the DB Tie record - which would be odd, as all of the other validation tests that I do on the code for Shared Folders would fail ...

I will know more once I review the debug log that you have sent.

--resync was used, however no DB Tie is being created .. which again is odd, given that it is being created for others. Will dig deeper, but also potentially switch you to looking at 'alpha-4' (yet to be posted) so that I am not maintaining 2 different code trees.

@abraunegg
Copy link
Owner

@robertschulze
I have posted 'alpha-4' - however I have not yet tested a Personal Account with Shared Folder as yet.

Please hold off on updating / doing any more testing until I have a chance to run some testing here.

@abraunegg abraunegg added the Bug Something isn't working label Dec 21, 2023
@abraunegg
Copy link
Owner

@robertschulze
I have tested and validated 'alpha-4' against a number of Personal Shared Folder scenarios:

  • Shared Folder is in the root of the other account
  • Shared Folder is several folders deep inside other folders

With the first scenario, I was able to replicate the ERROR: A database consistency issue has been caught. A --resync is needed to rebuild the database issue, thus this meant that the DB Tie record was not being created. This issue has been fixed.

Please can you rebuild your client, using 'alpha-4' code - refer to https://github.com/abraunegg/onedrive/discussions/2565 for details.

When you are running 'alpha-4' - your client version should be v2.5.0-alpha-4 GitHub version: v2.4.25-65-g6437b7e or greater.

@abraunegg abraunegg linked a pull request Dec 21, 2023 that will close this issue
@robertschulze
Copy link
Author

@abraunegg: Thanks for the new version: with original folder name "OneDrive-Constanze" (in source OneDrive) and link (in actual OneDrive that is to be synched) "OneDrive - Constanze" sync is now working, BUT: into subfolder "OneDrive-Constanze" instead of "OneDrive - Constanze"

DEBUG: ------------------------------------------------------------------
DEBUG: Processing OneDrive Item 10 of 201 from API Response Bundle 1
DEBUG: Raw JSON OneDrive Item: {"@microsoft.graph.downloadUrl":"https:\/\/public.dm.files.1drv.com\/y4mXKrR90rw28SbeAWKDDjPYn6UgMmThGjm96DsyXsPK9piYLfhkPJS7VFiecOzJM30IN5ZsUOswgvJm7uNX3xl6ishbrwm7aex33u-Vqo7gRZyb_6yjPZILDzUB9NHJcGnk_ocRxmOBgQvd7DWyDBN7IAs10VXI-oVVsKbi58QAx-cWBXDEcC-Sv6LghhrkQGG0GzI-2bzlLoHeBmA_dZsjbCxJ1QcL2Oo3lvGzZ5lTYcEOKLyX2B0WTxHo2ym-iIr","@odata.type":"#microsoft.graph.driveItem","cTag":"aYzpGMzAyOEJENzU4NTUyRkFGITI5NzcuMjU3","createdBy":{"application":{"id":"480728c5"},"device":{"id":"188006267c0904"},"oneDriveSync":{"@odata.type":"#microsoft.graph.identity","id":"0cf2e7c6-beb9-4dd9-aab2-ffff0a0a0000"},"user":{"displayName":"Robert Schulze","id":"9ba6f07668ba1ff7"}},"createdDateTime":"2022-11-13T23:24:20.407Z","eTag":"aRjMwMjhCRDc1ODU1MkZBRiEyOTc3LjU","file":{"hashes":{"quickXorHash":"Sbrp52ctlmFWVWsj\/IILVq5kCJ8=","sha1Hash":"522A5A36A2EBBB16DF4F6E2D8DB929C242C6E90B","sha256Hash":"ED6EF0B83D54273DC142FF3FD92BAA9B8E515C0128DD478B5D53B334C56AAEDD"},"mimeType":"application\/pdf"},"fileSystemInfo":{"createdDateTime":"2022-11-13T23:24:20.406Z","lastModifiedDateTime":"2014-07-14T18:41:13Z"},"id":"F3028BD758552FAF!2977","lastModifiedBy":{"application":{"id":"4035b959"},"user":{"displayName":"Robert Schulze","id":"9ba6f07668ba1ff7"}},"lastModifiedDateTime":"2023-09-13T00:38:22.373Z","name":"20130312 - Ausbildungsvertrag + Anschreiben.pdf","parentReference":{"driveId":"f3028bd758552faf","driveType":"personal","id":"F3028BD758552FAF!2948","name":"Logo","path":"\/drives\/f3028bd758552faf\/items\/F3028BD758552FAF!313:\/OneDrive-Constanze\/Logo"},"reactions":{"commentCount":0},"size":627299,"webUrl":"https:\/\/1drv.ms\/b\/s!AK8vVVjXiwLzlyE"}
DEBUG: This item is not a OneDrive deletion change
DEBUG: defaultRootId                                        = 9BA6F07668BA1FF7!103
DEBUG: 'search id'                                          = F3028BD758552FAF!2977
DEBUG: id == defaultRootId                                  = false
DEBUG: isItemRoot(onedriveJSONItem)                         = false
DEBUG: onedriveJSONItem['name'].str == 'root'               = false
DEBUG: itemHasParentReferenceId                             = true
DEBUG: skip_file item to check (file name only - parent path not in database): /20130312 - Ausbildungsvertrag + Anschreiben.pdf
DEBUG: skip_file evaluation for: /20130312 - Ausbildungsvertrag + Anschreiben.pdf
DEBUG: skip_file evaluation for: /20130312 - Ausbildungsvertrag + Anschreiben.pdf
DEBUG: Result: false
DEBUG: sync_list item to check: OneDrive-Constanze/Logo/20130312 - Ausbildungsvertrag + Anschreiben.pdf
DEBUG: Evaluation against 'sync_list' for this path: OneDrive-Constanze/Logo/20130312 - Ausbildungsvertrag + Anschreiben.pdf
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: OneDrive-Constanze
DEBUG: Evaluation against 'sync_list' result: parental path match
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: included for sync
DEBUG: Adding this Raw JSON OneDrive Item to jsonItemsToProcess array for further processing
DEBUG: ------------------------------------------------------------------
DEBUG: ------------------------------------------------------------------
DEBUG: Processing OneDrive JSON item 10 of 500 as part of JSON Item Batch 1 of 6
DEBUG: Raw JSON OneDrive Item: {"@microsoft.graph.downloadUrl":"https:\/\/public.dm.files.1drv.com\/y4mXKrR90rw28SbeAWKDDjPYn6UgMmThGjm96DsyXsPK9piYLfhkPJS7VFiecOzJM30IN5ZsUOswgvJm7uNX3xl6ishbrwm7aex33u-Vqo7gRZyb_6yjPZILDzUB9NHJcGnk_ocRxmOBgQvd7DWyDBN7IAs10VXI-oVVsKbi58QAx-cWBXDEcC-Sv6LghhrkQGG0GzI-2bzlLoHeBmA_dZsjbCxJ1QcL2Oo3lvGzZ5lTYcEOKLyX2B0WTxHo2ym-iIr","@odata.type":"#microsoft.graph.driveItem","cTag":"aYzpGMzAyOEJENzU4NTUyRkFGITI5NzcuMjU3","createdBy":{"application":{"id":"480728c5"},"device":{"id":"188006267c0904"},"oneDriveSync":{"@odata.type":"#microsoft.graph.identity","id":"0cf2e7c6-beb9-4dd9-aab2-ffff0a0a0000"},"user":{"displayName":"Robert Schulze","id":"9ba6f07668ba1ff7"}},"createdDateTime":"2022-11-13T23:24:20.407Z","eTag":"aRjMwMjhCRDc1ODU1MkZBRiEyOTc3LjU","file":{"hashes":{"quickXorHash":"Sbrp52ctlmFWVWsj\/IILVq5kCJ8=","sha1Hash":"522A5A36A2EBBB16DF4F6E2D8DB929C242C6E90B","sha256Hash":"ED6EF0B83D54273DC142FF3FD92BAA9B8E515C0128DD478B5D53B334C56AAEDD"},"mimeType":"application\/pdf"},"fileSystemInfo":{"createdDateTime":"2022-11-13T23:24:20.406Z","lastModifiedDateTime":"2014-07-14T18:41:13Z"},"id":"F3028BD758552FAF!2977","lastModifiedBy":{"application":{"id":"4035b959"},"user":{"displayName":"Robert Schulze","id":"9ba6f07668ba1ff7"}},"lastModifiedDateTime":"2023-09-13T00:38:22.373Z","name":"20130312 - Ausbildungsvertrag + Anschreiben.pdf","parentReference":{"driveId":"f3028bd758552faf","driveType":"personal","id":"F3028BD758552FAF!2948","name":"Logo","path":"\/drives\/f3028bd758552faf\/items\/F3028BD758552FAF!313:\/OneDrive-Constanze\/Logo"},"reactions":{"commentCount":0},"size":627299,"webUrl":"https:\/\/1drv.ms\/b\/s!AK8vVVjXiwLzlyE"}
DEBUG: Attempting to calculate local filesystem path for f3028bd758552faf and F3028BD758552FAF!2948
DEBUG: New Item calculated full path is: OneDrive-Constanze/Logo/20130312 - Ausbildungsvertrag + Anschreiben.pdf
DEBUG: The item we are syncing is a file
DEBUG: skip_file item to check: /OneDrive-Constanze/Logo/20130312 - Ausbildungsvertrag + Anschreiben.pdf
DEBUG: skip_file evaluation for: /OneDrive-Constanze/Logo/20130312 - Ausbildungsvertrag + Anschreiben.pdf
DEBUG: Result: false
DEBUG: sync_list item to check: OneDrive-Constanze/Logo/20130312 - Ausbildungsvertrag + Anschreiben.pdf
DEBUG: Evaluation against 'sync_list' for this path: OneDrive-Constanze/Logo/20130312 - Ausbildungsvertrag + Anschreiben.pdf
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: OneDrive-Constanze
DEBUG: Evaluation against 'sync_list' result: parental path match
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: included for sync
DEBUG: Flagging object as a file
DEBUG: OneDrive change is potentially a new local item
DEBUG: Local item time discrepancy detected: OneDrive-Constanze/Logo/20130312 - Ausbildungsvertrag + Anschreiben.pdf
DEBUG: This local item has a different modified time 2023-Dec-21 23:50:51Z when compared to remote modified time 2014-Jul-14 18:41:13Z
DEBUG: Local item has the same hash value as the item online - correcting timestamp
DEBUG: The source of the incorrect timestamp was OneDrive online - correcting timestamp online
DEBUG: Read token from appConfig
DEBUG: Authorised State: true
DEBUG: Existing Microsoft OneDrive Access Token Expires: 2023-Dec-22 01:49:58.4286236
DEBUG: No valid Microsoft OneDrive webhook subscription to delete
DEBUG: saveItem - creating DB item from this JSON: {"@microsoft.graph.downloadUrl":"https:\/\/cdo26g.dm.files.1drv.com\/y4pyzKT-9a6L0TnFQz_CiG6FM0ak4u8F_9cmEKxzHp7-ocSCcimMd3OSs4uSZRF14zNqDfz1cF8ww-JeTCeDPIuL9D85KF5W_iuDi5u_izjVLRAPKmRcH0LBt-IrNqZpKDATpv2LOSBmvJS6cE9NX-gS3dBm5ZcJs73Qx3eYYJ9x2tCmmwJC8JQXoLuTh-oLue7SgUi5o-InBKALhpxnOIPEXNbZB3f4JM6CJ74KvOMpxSmQ0Oh15VWP9IQfcX_xize","@odata.context":"https:\/\/graph.microsoft.com\/v1.0\/$metadata#drives('f3028bd758552faf')\/items\/$entity","cTag":"aYzpGMzAyOEJENzU4NTUyRkFGITI5NzcuMjU3","createdBy":{"application":{"displayName":"MSOffice15","id":"480728c5"},"device":{"id":"188006267c0904"},"oneDriveSync":{"@odata.type":"#microsoft.graph.identity","id":"0cf2e7c6-beb9-4dd9-aab2-ffff0a0a0000"},"user":{"displayName":"Robert Schulze","id":"9ba6f07668ba1ff7"}},"createdDateTime":"2022-11-13T23:24:20.407Z","eTag":"aRjMwMjhCRDc1ODU1MkZBRiEyOTc3LjI2","file":{"hashes":{"quickXorHash":"Sbrp52ctlmFWVWsj\/IILVq5kCJ8=","sha1Hash":"522A5A36A2EBBB16DF4F6E2D8DB929C242C6E90B","sha256Hash":"ED6EF0B83D54273DC142FF3FD92BAA9B8E515C0128DD478B5D53B334C56AAEDD"},"mimeType":"application\/pdf"},"fileSystemInfo":{"createdDateTime":"2022-11-13T23:24:20.406Z","lastModifiedDateTime":"2023-12-21T23:50:51Z"},"id":"F3028BD758552FAF!2977","lastModifiedBy":{"application":{"displayName":"OneDrive Client for Linux","id":"4035b959"},"user":{"displayName":"Robert Schulze","id":"9ba6f07668ba1ff7"}},"lastModifiedDateTime":"2023-12-21T23:50:52.2Z","name":"20130312 - Ausbildungsvertrag + Anschreiben.pdf","parentReference":{"driveId":"f3028bd758552faf","driveType":"personal","id":"F3028BD758552FAF!2948","name":"Logo","path":"\/drives\/f3028bd758552faf\/items\/F3028BD758552FAF!313:\/OneDrive-Constanze\/Logo"},"reactions":{"commentCount":0},"shared":{"owner":{"user":{"displayName":"Constanze Schulze","id":"f3028bd758552faf"}},"scope":"users"},"size":627299,"webUrl":"https:\/\/1drv.ms\/b\/s!AK8vVVjXiwLzlyE"}
DEBUG: Flagging object as a file
DEBUG: Adding to database: Item("f3028bd758552faf", "F3028BD758552FAF!2977", "20130312 - Ausbildungsvertrag + Anschreiben.pdf", "", file, "aRjMwMjhCRDc1ODU1MkZBRiEyOTc3LjI2", "aYzpGMzAyOEJENzU4NTUyRkFGITI5NzcuMjU3", 2023-Dec-21 23:50:51Z, "F3028BD758552FAF!2948", "Sbrp52ctlmFWVWsj/IILVq5kCJ8=", "", "", "", "Y", "627299")
DEBUG: The item to sync is already present on the local filesystem and is in-sync with what is reported online
DEBUG: Update/Insert local database with item details: Item("f3028bd758552faf", "F3028BD758552FAF!2977", "20130312 - Ausbildungsvertrag + Anschreiben.pdf", "", file, "aRjMwMjhCRDc1ODU1MkZBRiEyOTc3LjU", "aYzpGMzAyOEJENzU4NTUyRkFGITI5NzcuMjU3", 2014-Jul-14 18:41:13Z, "F3028BD758552FAF!2948", "Sbrp52ctlmFWVWsj/IILVq5kCJ8=", "", "", "", "Y", "627299")
DEBUG: ------------------------------------------------------------------
DEBUG: Attempting to calculate local filesystem path for f3028bd758552faf and F3028BD758552FAF!2977
DEBUG: Processing OneDrive-Constanze/Logo/20130312 - Ausbildungsvertrag + Anschreiben.pdf
DEBUG: The file has not changed

@abraunegg
Copy link
Owner

@robertschulze

with original folder name "OneDrive-Constanze" (in source OneDrive) and link (in actual OneDrive that is to be synched) "OneDrive - Constanze" sync is now working, BUT: into subfolder "OneDrive-Constanze" instead of "OneDrive - Constanze"

You will need to 'clean' up your local and remote OneDrive.

Suggested path of resolution:

  • Delete online your 'Add to onedrive' Shared Folder Links
  • Delete locally those same shared folder links
  • Delete locally any erroneous extra folders

Please dont use 'alpha-4' - I found some issues especially with Shared Folders that was not there with 'alpha-3' or prior.

I have now resolved this issue, re-validated and tested with OneDrive Personal & OneDrive Business Shared Folders.

Please can you rebuild your client so that you get: v2.5.0-alpha-4 GitHub version: v2.4.25-67-g2d9bdd1

Example output:
05Semester is a Shared Folder, that is several layers deep in the account that is sharing the data with the test account.

Processing 05Semester/Bilder/trunk/doc/beginners_guide/ecryptfs_beginners_guide.tex
The file has not changed
Scanning the local file system '~/OneDrive' for new data to upload ...
Skipping item - excluded by sync_list config: ./random_25k_files
OneDrive Client requested to create this directory online: ./05Semester/Bilder/444
The requested directory to create was not found on OneDrive - creating remote directory: ./05Semester/Bilder/444
Successfully created the remote directory ./05Semester/Bilder/444 on Microsoft OneDrive
New items to upload to OneDrive: 1
Total New Data to Upload:        26 Bytes
WARNING: Shared Folder OneDrive quota information is being restricted or providing a zero value. Space available online cannot be guaranteed.
Uploading new file ./05Semester/Bilder/444/asdfasdf.txt ... done.
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching /delta response from the OneDrive API for Drive ID: 66d53be8a5056eca
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 0
Finished processing /delta JSON response from the OneDrive API
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Syncing this OneDrive Personal Shared Folder: DESIGN _ BRASIL
Fetching /delta response from the OneDrive API for Drive ID: bc7d88ec1f539dcf
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 0
Finished processing /delta JSON response from the OneDrive API
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Syncing this OneDrive Personal Shared Folder: 05Semester
Fetching /delta response from the OneDrive API for Drive ID: bc7d88ec1f539dcf
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 4
Finished processing /delta JSON response from the OneDrive API
Processing 3 applicable changes and items received from Microsoft OneDrive 
Processing OneDrive JSON item batch [1/1] to ensure consistent local state

Sync with Microsoft OneDrive is complete

'DESIGN _ BRASIL' is a Shared Folder from the root of the account that is sharing the data with the test account.

Processing 05Semester/Bilder/444
The directory has not changed
Processing 05Semester/Bilder/444/asdfasdf.txt
The file has not changed
Scanning the local file system '~/OneDrive' for new data to upload ...
Skipping item - excluded by sync_list config: ./random_25k_files
OneDrive Client requested to create this directory online: ./05Semester/Bilder/trunk/doc/beginners_guide/Rufus
The requested directory to create was not found on OneDrive - creating remote directory: ./05Semester/Bilder/trunk/doc/beginners_guide/Rufus
Successfully created the remote directory ./05Semester/Bilder/trunk/doc/beginners_guide/Rufus on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./05Semester/Logging Update
The requested directory to create was not found on OneDrive - creating remote directory: ./05Semester/Logging Update
Successfully created the remote directory ./05Semester/Logging Update on Microsoft OneDrive
OneDrive Client requested to create this directory online: ./DESIGN _ BRASIL/Logging
The requested directory to create was not found on OneDrive - creating remote directory: ./DESIGN _ BRASIL/Logging
Successfully created the remote directory ./DESIGN _ BRASIL/Logging on Microsoft OneDrive
New items to upload to OneDrive: 8
Total New Data to Upload:        1 MB
WARNING: Shared Folder OneDrive quota information is being restricted or providing a zero value. Space available online cannot be guaranteed.
WARNING: Shared Folder OneDrive quota information is being restricted or providing a zero value. Space available online cannot be guaranteed.
WARNING: Shared Folder OneDrive quota information is being restricted or providing a zero value. Space available online cannot be guaranteed.
WARNING: Shared Folder OneDrive quota information is being restricted or providing a zero value. Space available online cannot be guaranteed.
WARNING: Shared Folder OneDrive quota information is being restricted or providing a zero value. Space available online cannot be guaranteed.
WARNING: Shared Folder OneDrive quota information is being restricted or providing a zero value. Space available online cannot be guaranteed.
WARNING: Shared Folder OneDrive quota information is being restricted or providing a zero value. Space available online cannot be guaranteed.
WARNING: Shared Folder OneDrive quota information is being restricted or providing a zero value. Space available online cannot be guaranteed.
Uploading new file ./DESIGN _ BRASIL/Logging/multidownload.d ... done.
Uploading new file ./05Semester/Logging Update/README.md ... done.
Uploading new file ./DESIGN _ BRASIL/Logging/progressBar.d ... done.
Uploading new file ./05Semester/Logging Update/notify.d ... done.
Uploading new file ./05Semester/Logging Update/dnotify.d ... done.
Uploading new file ./05Semester/Logging Update/log.d ... done.
Uploading new file ./DESIGN _ BRASIL/Logging/sync.d ... done.
Uploading new file ./05Semester/Bilder/trunk/doc/beginners_guide/Rufus/rufus-3.20.exe ... done.
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching /delta response from the OneDrive API for Drive ID: 66d53be8a5056eca
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 0
Finished processing /delta JSON response from the OneDrive API
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Syncing this OneDrive Personal Shared Folder: DESIGN _ BRASIL
Fetching /delta response from the OneDrive API for Drive ID: bc7d88ec1f539dcf
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 5
Finished processing /delta JSON response from the OneDrive API
Processing 5 applicable changes and items received from Microsoft OneDrive 
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Syncing this OneDrive Personal Shared Folder: 05Semester
Fetching /delta response from the OneDrive API for Drive ID: bc7d88ec1f539dcf
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 12
Finished processing /delta JSON response from the OneDrive API
Processing 11 applicable changes and items received from Microsoft OneDrive 
Processing OneDrive JSON item batch [1/1] to ensure consistent local state

Sync with Microsoft OneDrive is complete

@robertschulze
Copy link
Author

@abraunegg Hi, I tried with the new version but the result is still the same, the files are synced into "OneDrive-Constanze" instead of "OneDrive - Constanze". Should I send you the log file again?

@abraunegg
Copy link
Owner

@abraunegg Hi, I tried with the new version but the result is still the same, the files are synced into "OneDrive-Constanze" instead of "OneDrive - Constanze". Should I send you the log file again?

Please send the entire debug log when possible. The issue is probably to do with the difference between the actual shared folder name (remote name) and the link under your account.

I am also away from tools until the new year.

@abraunegg abraunegg linked a pull request Jan 2, 2024 that will close this issue
@abraunegg
Copy link
Owner

@robertschulze

@abraunegg Hi, I tried with the new version but the result is still the same, the files are synced into "OneDrive-Constanze" instead of "OneDrive - Constanze". Should I send you the log file again?

I have responded via email - if you could review and get back to me that would be greatly appreciated.

@abraunegg
Copy link
Owner

@robertschulze
Please could you respond as to your latest testing?

@robertschulze
Copy link
Author

@abraunegg
Thanks for the thorough explanation via mail and sorry, had a test run already the previous week but forgot to upload. I now sent the new log file via mail.
Unfortunately, it appears that the problem still persists.

@abraunegg
Copy link
Owner

@robertschulze

Unfortunately, it appears that the problem still persists.

Looking at your logfile - your remote folder name is 'OneDrive-Constanze' which is why the data is being synced to this folder:

image

If you search your logfile for: Processing OneDrive JSON item 1 of 98 as part of JSON Item Batch 1 of 1 you will find the JSON that this comes from.

This is why your data is syncing to 'OneDrive-Constanze' and not 'OneDrive - Constanze'

Please look at how your folder are all named - as you will need to do a cleanup of everything first (online and your local filesystem), then re-create how you want it to be online (using 'OneDrive - Constanze') because of the bug|issue identified where your original rename of 'OneDrive - Constanze' was done because it contained HTML Characters.

Additionally, it would be wise to also uplift your client version to 'alpha-5' - please read #2585 for details.

@robertschulze
Copy link
Author

robertschulze commented Jan 14, 2024

I have to admin I am a bit confused. Let me recap the status:

  • Account 1: the subfolder to be shared is Dokumente/OneDrive-Constanze (if I understood correctly, the name w/o spaces is necessary to avoid a bug on MS side)
    • folder Dokumente (and thus also the subfolders therein) is shared to Account 2
  • Account 2 (strangely, both, Dokumente as well as the subfolders (OneDrive-Constanze and a second one) show up separately in Shared with you view of Account 2)
    • the shared folder OneDrive-Constanze is then added to My Documents (three dots, add to My Documents) and the linked directory shows up there
    • the linked directory is then renamed to OneDrive - Constanze (with spaces)
    • the change of the link name is reflected in the OneDrive desktop client for Account 2

What I understood is that the item to be synced is still named without spaces (as it needs be due to the bug) but since the link has been renamed and the change in name is reflected both in the Web GUI as well as the Desktop client, shouldn't also the Linux client use the name of the link instead of the original folder name?

@abraunegg
Copy link
Owner

@robertschulze
You essentially need to cleanup the 'shared with you' items. Unshare, remove links and then rename the folders on the account that does the sharing to what you expect it to be.

Then, share that folder with the account you are syncing the data with.

Renaming the folder to remove the HTML Ascii cide issue should have been just a one time item - if i rename things online using a 'space' - then it uses a space ... so no idea how or why you ended up with %20 to begin with.

Please clean up everything, remove the shared with me items, rename things as per how you are expecting it, reshare and then retest.

If you look at the non debug log (just one -v) you should see that the shared folder is pulled in and used, and any local data in that local folder will go to the correct online shared folder.

@robertschulze
Copy link
Author

@abraunegg Thanks, indeed, now with the name containing (correct) spaces in Account 1 (and thus appears with correct name in Account 2, no rename necessary) the sync is working fine. Thanks!

@abraunegg
Copy link
Owner

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

Repository owner locked as resolved and limited conversation to collaborators Jan 22, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Bug Something isn't working Fixed OneDrive API Bug OneDrive API Bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants