Is it possible to prevent onedrive from executing full scan every time the system is rebooted?

[fO-000]

I use systemd to let onedrive start automatically:

$ systemctl --user cat onedrive.service       
# /usr/lib/systemd/user/onedrive.service
[Unit]
Description=OneDrive Free Client
Documentation=https://github.com/abraunegg/onedrive
After=network-online.target
Wants=network-online.target

[Service]
Environment="HTTP_PROXY=http://127.0.0.1:7890"
Environment="HTTPS_PROXY=http://127.0.0.1:7890"
ExecStart=/usr/bin/onedrive --monitor -v
Restart=on-failure
RestartSec=3
RestartPreventExitStatus=3

[Install]
WantedBy=default.target

~/.config/onedrive/config as below (no sync_list now, but used before):

skip_dir           = "/Music|/Pictures"
skip_dir           = "Documents/OneNote Notebooks"

skip_file          = "/Personal Vault|*.pyc"

skip_dotfiles      = "false"
skip_symlinks      = "false"

monitor_interval   = "300"
min_notify_changes = "1"
check_nosync       = "false"

Every time my system reboots (Kali in virtual machine), onedrive will execute a full scan:

Jan 26 20:13:09 x onedrive[1007]: Processing 206 OneDrive items to ensure consistent local state due to a full scan being requested
... ...
Jan 26 20:17:43 x onedrive[1007]: Processing 1 OneDrive items to ensure consistent local state due to a full scan being requested

This full scan can sometimes be very time-consuming, maybe because I have a lot of files, about 36,000. And during the full scan, onedrive cannot monitor and upload the files I modified.

• So why does the onedrive --monitor execute full scan every time the system is rebooted?
• Is it possible to avoid this behavior while ensuring the integrity of the data? Just like achieving the effect that onedrive can immediately monitor and upload changes after the system reboots.

[abraunegg]

@fO-000

This full scan can sometimes be very time-consuming, maybe because I have a lot of files, about 36,000. And during the full scan, onedrive cannot monitor and upload the files I modified.
So why does the onedrive --monitor execute full scan every time the system is rebooted?

This is because there is the potential for an inconsistent state. Without doing this task on first start up, there have been many instances where new files / folders have been missed - either locally or remote, for whatever reason - thus these files are either removed or at other times deleted. This is why, on application startup, a full scan is done to ensure a consistent state. Most notably, this is seen when a user has 1 OneDrive account shared with many systems at the same time. The OneDrive API only sends certain events once, thus, if that event is processed by your other platform, when you fire up Linux and this client - that event has been processed, thus gone - thus, either leads to data loss or some other undesired consequence.

To process 36,000 files either locally or remote, should not be taking an inordinate amount of time either. Additionally, since 5th January, the OneDrive API has had a major bug / flaw in how changes were being tracked. Refer to:

• ODP Regression: OneDrive Graph API /delta deltaLink response is malformed OneDrive/onedrive-api-docs#1408
• nextLink Bug in Delta API: Sync Changes OneDrive/onedrive-api-docs#1424

If the scan is very time consuming for you, and you are noticing this issue only since the 5th Jan, then you were hit by the above API issue.

Is it possible to avoid this behavior while ensuring the integrity of the data? Just like achieving the effect that onedrive can immediately monitor and upload changes after the system reboots.

No, not without application development to remove or change how this works on startup which I am highly reluctant to do because of the above issue that it solves. Having the application perform a full integrity check at startup is a small price to pay to ensure a consistent state.

Any local changes that are made during the integrity check, are also queued and processed at the end of the integrity check.

The other item here is what application version are you using on Kali Linux? You do not indicate what application version you are using. This message to ensure consistent local state due to a full scan being requested can only be generated on the next sync pass after creating a new local directory after receiving such instruction from OneDrive. I strongly suspect you are running an old version of code as well. irrelevant to question

Please confirm your application version.

[fO-000]

@abraunegg

Thanks for the quick and informative response.

Please confirm your application version.

Below is the version of my system and the onedrive:

$ onedrive --version
onedrive v2.4.9-1

$ uname -a
Linux x 5.9.0-kali5-amd64 #1 SMP Debian 5.9.15-1kali1 (2020-12-18) x86_64 GNU/Linux

and I installed onedrive using sudo apt install onedrive.

If the scan is very time consuming for you, and you are noticing this issue only since the 5th Jan, then you were hit by the above API issue.

I just started using your onedrive client a few days ago, so I can't compare. 😅 And I do appreciate your maintenance of this program. Great job 👍

Having the application perform a full integrity check at startup is a small price to pay to ensure a consistent state.

I agree with this.

The OneDrive API only sends certain events once, thus, it that event is processed by your other platform, when you fire up Linux and this client - that event has been processed, thus gone - thus, either leads to data loss or some other undesired consequence.

I am not familiar with the OneDrive API, but just for some further discussion. I think OneDrive should have a list of linked devices and maintain an event queue for each of them. When each device connects to OneDrive, it first requests the unique event queue associated with itself. In this way, there is no issue of events being processed by a linked device A (other platforms) and causing B (my Linux) not receive the event.

But according to what you mentioned, "The OneDrive API only sends certain events once". An event queue for each linked device doesn't seem to be the case.

Will official OneDrive on Windows 10 also perform a full scan after every system reboot, because there is no event queue for each linked device? Or does this limitation only affect third-party OneDrive clients, like this project? That is, OneDrive maintains a unique event queue for each official Windows 10 client (using OneDrive internal API?), while third-party (using opened OneDrive API) ones do not.

[abraunegg]

@fO-000

I think OneDrive should have a list of linked devices and maintain an event queue for each of them. When each device connects to OneDrive, it first requests the unique event queue associated with itself. In this way, there is no issue of events being processed by a linked device A (other platforms) and causing B (my Linux) not receive the event.

That would be ideal, but OneDrive & the API lacks this ability.

Will official OneDrive on Windows 10 also perform a full scan after every system reboot, because there is no event queue for each linked device?

Windows 'official' clients also do not use the API (or they use an internal only API that is not exposed publicly). This is how those clients are able to use other aspects such as differential sync or syncing SharePoint shortcuts which the public API lacks. They also do not do a full scan as, from their viewpoint - only official clients are using / accessing the data and they are not concerned with data integrity so much.

[fO-000]

@abraunegg

I see.

Thank you for taking the time to respond to these questions. And I'll close this issue. 🙏

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.