onedrive repeat Processing same number of onedrive items and eventually fails sync

[xmoiduts]

Note: Before submitting a bug report, please ensure you are running the latest 'onedrive' client as built from 'master' and by using the latest available DMD compiler. Refer to the readme on building the client for your system.
[Yes, I rebuilt and used the latest version]

Bug Report Details

Describe the bug
A clear and concise description of what the bug is.

Onedrive constantly (repeat / stuck in) reporting "Processing the same number of Onedrive items to ensure ... " and will eventually fail syncing (typically after 1-5 hours, resulting in a failed $? .);
Using --resync will succeed a single standalone sync and generate duplicated hostname-attached filenames , but the syncing issue still exists and will happen next time when not using --resync. (I don't think resyncing every time is a good option because it corrupts filenames of modified one (hostname attached) and my full resync takes several hours where it is supposed to be <30 minutes.)
More details in 'Context' part.

Application and Operating System Details:

• Provide your OS & version (CentOS 6.x, Ubuntu 18.x etc) and the output of: uname -a
Linux onedriver24 5.4.0-51-generic #56-Ubuntu SMP Mon Oct 5 14:28:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
  (An DigitalOcean VPS)
• Are you using a headless system (no gui) or with a gui installed?
  No GUI
• OneDrive Account Type
  Personal
• Did you build from source or install from a package?
  Build from source
  Build once, save in block storage and re-install in every new server instance which are of the save Operating System within the same VPS provider.
• If you installed from source, what is your DMD or LDC compiler version: dmd --version or ldmd2 --version
DMD64 D Compiler v2.095.0
• OneDrive Application Version: Output of onedrive --version
onedrive v2.4.9-6-g1c6dd42 (but 2.4.7.x also have this issue)
• OneDrive Application Configuration: Output of onedrive --display-config

Configuration file successfully loaded                                    
onedrive version                       = v2.4.9-6-g1c6dd42                
Config path                            = /root/.config/onedrive           
Config file found in config path       = true                             
Config option 'check_nosync'           = false                            
Config option 'sync_dir'               = /mnt/volume_****_02/OneDriveMnt/ 
Config option 'skip_dir'               =                                  
Config option 'skip_file'              = ~*|.~*|*.tmp                     
Config option 'skip_dotfiles'          = false                            
Config option 'skip_symlinks'          = false                            
Config option 'monitor_interval'       = 300                              
Config option 'min_notify_changes'     = 5                                
Config option 'log_dir'                = /var/log/onedrive/               
Config option 'classify_as_big_delete' = 1000                             
Config option 'upload_only'            = false                            
Config option 'no_remote_delete'       = false                            
Config option 'remove_source_files'    = false                            
Config option 'sync_root_files'        = false                            
Selective sync 'sync_list' configured  = true                             
sync_list contents:                                                       
Codes/k********f-proj/K********f/                                         
!Codes/k********f-proj/K********f-Dev/                                    
!Codes/k********f-proj/K********f/.git                                    
!Codes/k********f-proj/K********f/.vscode                                 
!Codes/k********f-proj/K********f/__pycache__                             
Codes/k********f-proj/data-production/                                    
Business Shared Folders configured     = false                            

• Provide the version of curl you are using: Output of curl --version

curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08

• Is your configured 'sync_dir' a local directory or a network mount point?
  Local directory (mounted block storage within the VPS provider)
• If not local, provide all the mountpoints in your system: Output of: mount
• What partition format type does your configured 'sync_dir' reside on? Output of: lsblk -f

loop0   squashfs                                                            0   100% /snap/snapd/9607
loop1   squashfs                                                            0   100% /snap/core18/1885
loop2   squashfs                                                            0   100% /snap/lxd/16922
loop3   squashfs                                                            0   100% /snap/snapd/10707
loop4   squashfs                                                            0   100% /snap/core18/1944
loop5   squashfs                                                            0   100% /snap/lxd/18150
sda     ext4                     55e109f3-a5e8-48f1-bf84-a311c4d5f56c    1.6G    84% /mnt/volume_****_02
vda
├─vda1  ext4     cloudimg-rootfs db153994-91fe-426f-bd88-f751e07f97f2   17.8G    26% /
├─vda14
└─vda15 vfat     UEFI            58A1-A55A                             100.5M     4% /boot/efi

sda is the attached block storage (DigitalOcean Volumes)

• Explain your entire configuration setup - is the OneDrive folder shared with any other system, shared with any other platform at the same time, is the OneDrive account you use shared across multiple systems / platforms / Operating Systems and in use at the same time

The OneDrive folder is shared with another windows 10 machine (my personal computer) while the block storage sda is only accessible by the server. I avoid to do any operations on the sync_path during the Linux onedrive sync. When not syncing (Linux onedrive completely not running), I modify some files in the sync_path.

Note: Please generate a full debug log whilst reproducing the issue as per https://github.com/abraunegg/onedrive/wiki/Generate-debug-log-for-support and email to support@mynas.com.au
The log is too big so I didn't generate it full, but may be enough to address the issue.

To Reproduce
Steps to reproduce the behavior if not causing an application crash:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
   (I can't reproduce it from scratch, it happened after running normally for months)

If issue is replicated by a specific 'file' or 'path' please archive the file and path tree & email to support@mynas.com.au
"I also think there would be a file that triggers the issue, but in --verbose mode it didn't give me any clue, see screenshot".

Complete Verbose Log Output
A clear and full log of the problem when running the application in the following manner (ie, not in monitor mode):

onedrive --synchronize --verbose <any of your other needed options>

Run the application in a separate terminal window or SSH session and provide the entire application output including the error & crash. When posing the logs, Please format log output to make it easier to read. See https://guides.github.com/features/mastering-markdown/ for more details.

Application Log Output:
(See screenshots)

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
I have a spider script that collects information on a website. In order to code locally and run the script remotely, I chose to use OneDrive to share data between my computer and cloud server(s).

Everyday a new server is launched, attaching block storage within the same data-canter, installing the self-pre-compiled DMD/onedrive and run my script that downloads/modifies some files. Then an standalone onedrive sync is performed to send those data to MS/OneDrive service, the server is afterwards destroyed.

This approach worked seamlessly for several months, about one week ago it started failing to sync and causing chaos(duplicated files with hostnames attached to filenames, which I know is a feature). It was then onedrive version 2.4.7.19. but upgrading to 2.4.9 won't help.

I tried to --resync; deleting all sqlite indices then --resync; deleting all onedrive files on the server then perform fresh (re)sync. Everytime resync works, but when not adding --resync(ing), the verbose message tells me it stuck somewhere. A normal sync often give flunctuating Processed onedrive items numbers like [207,213,199,204,...], while now it gives [200,200,200,200,...] .

Bug Report Checklist

• Detailed description
• Application and Operating System Details provided in full
• [] Reproduction steps (if applicable)
• Verbose Log Output from your error
• Debug Log generated and submitted

[abraunegg]

@xmoiduts
Without seeing a full debug log, it is hard to say what is going on.

A normal sync often give flunctuating Processed onedrive items numbers like [207,213,199,204,...], while now it gives [200,200,200,200,...] .

This is 100% normal - the OneDrive API provides these details.

This approach worked seamlessly for several months, about one week ago it started failing to sync and causing chaos(duplicated files with hostnames attached to filenames, which I know is a feature). It was then onedrive version 2.4.7.19. but upgrading to 2.4.9 won't help.

As this is a 'personal' account, and this behaviour sounds very similar to the OneDrive API deltaLink bug, I am marking this as an API issue. This was apparently solved yesterday, so please can you recheck your process.

References:

• nextLink Bug in Delta API: Sync Changes OneDrive/onedrive-api-docs#1424
• ODP Regression: OneDrive Graph API /delta deltaLink response is malformed OneDrive/onedrive-api-docs#1408

[abraunegg]

@xmoiduts

Everyday a new server is launched, attaching block storage within the same data-canter

Given that the API issue was resolved >24hrs ago, has the sync process returned to normal behavior for you? Please can you advise.

[Konstantin-Margorin]

Our Acronis Microsoft 365 backup solution https://www.acronis.com/en-us/solutions/backup/office-365/ is still affected by the issue.

[abraunegg]

@Konstantin-Margorin

Our Acronis Microsoft 365 backup solution https://www.acronis.com/en-us/solutions/backup/office-365/ is still affected by the issue.

Suggest then that you contact Acronis for support. I cannot help you.

If you want OneDrive API support on this issue, you need to comment on these API threads:

• nextLink Bug in Delta API: Sync Changes OneDrive/onedrive-api-docs#1424
• ODP Regression: OneDrive Graph API /delta deltaLink response is malformed OneDrive/onedrive-api-docs#1408

[Konstantin-Margorin]

Suggest then that you contact Acronis for support. I cannot help you.

I'm from Acronis.
You asked "Given that the API issue was resolved >24hrs ago, has the sync process returned to normal behavior for you? Please can you advise."
And I'm reporting, that issue still here.
Should I post my comment to suggested threads?

[abraunegg]

@Konstantin-Margorin
No problem, however nothing indicated you were from Acronis either :) - this being said however that comment was directed towards the user who was building a new virtual machine each day, to sync their data. Given that Microsoft API team have advised that the change that introduced the issue has been rolled back, I was asking the user to re-test their daily VM build process.

That Acronis still presents an issue, shows that the issue might still persist, however, what 'end user accounts' are experiencing the issue - Personal or Business (Office 365 etc)? I can confirm that the API, for 'personal' account types, the deltaLink response is now different (as per before the broken change) - and agin just tested, still presents the correct data.

For Business accounts - right now I am getting an API 504 error, so I cannot reliably test.

Should I post my comment to suggested threads?

That would be my advice - specifically comment in OneDrive/onedrive-api-docs#1424. 1408 was when the response was totally garbled and generated a 400 error, which is not the case this time around.

[Konstantin-Margorin]

That Acronis still presents an issue, shows that the issue might still persist, however, what 'end user accounts' are experiencing the issue - Personal or Business (Office 365 etc)?

We still see this for business accounts last 24 hours till now:

That means, that we could not receive new changes in 30 pages in row. Response is not garbled, simply no new changes. We fail backup in this case.

[abraunegg]

@Konstantin-Margorin
I certainly would report this via OneDrive/onedrive-api-docs#1424 & escalate to your Microsoft support rep / TAM as well.

[xmoiduts]

@xmoiduts

Everyday a new server is launched, attaching block storage within the same data-canter

Given that the API issue was resolved >24hrs ago, has the sync process returned to normal behavior for you? Please can you advise.

I tested it today and everything runs smoothly.

When analyzing my logs as well as the --verbose --verbose log, the issue started happening on Jan. 5, and the delta link was malformed, meaning that your mentioned deltalink issue (below) is correct. Thank you for your effort in helping me solve the issue.

OneDrive/onedrive-api-docs#1424
OneDrive/onedrive-api-docs#1408

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.