Bug: skip_file in config does not override defaults

[alantgpl]

Describe the bug

Noticed this change after upgrading onedrive from 2.4.23 to 2.5.2
Setting skip_file in config appends to defaults, it does not override the defaults.

Config file:
skip_file = ".~*|*.tmp"

--display-config:
Config option 'skip_file' = ~*|.~*|*.tmp|*.swp|*.partial|.~*|*.tmp

We specifically need to sync files starting with (~) tilde character, eg. ~testfile.txt
These files are now getting skipped on sync

Operating System Details

Ubuntu 22.04.5 LTS

Client Installation Method

From Distribution Package

OneDrive Account Type

Personal

What is your OneDrive Application Version

onedrive v2.5.2-1+np1+1.1

What is your OneDrive Application Configuration

Application version                          = onedrive v2.5.2-1+np1+1.1
Compiled with                                = LDC 2098
User Application Config path                 = <omitted>
System Application Config path               = /etc/onedrive
Applicable Application 'config' location     = <omitted>
Configuration file found in config location  = true - using 'config' file values to override application defaults
Applicable 'sync_list' location              = <omitted>
Applicable 'items.sqlite3' location          = <omitted>
Config option 'drive_id'                     = <omitted>
Config option 'sync_dir'                     = <omitted>
Config option 'enable_logging'               = true
Config option 'log_dir'                      = /var/log/onedrive/
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     =
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial|.~*|*.tmp
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 12
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = true
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = true
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = <omitted>
Config option 'azure_ad_endpoint'            =
Config option 'azure_tenant_id'              =
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.2-1+np1+1.1
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 0
Config option 'threads'                      = 8
Environment var 'XDG_RUNTIME_DIR'            = true
Environment var 'DBUS_SESSION_BUS_ADDRESS'   = true
Config option 'notify_file_actions'          = false

Selective sync 'sync_list' configured        = false

Config option 'sync_business_shared_items'   = false

Config option 'webhook_enabled'              = false

What is your 'curl' version

curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.18
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

<omitted>

What are all your local file system partition types

<omitted>

How do you use 'onedrive'

Local folder, syncing up to a Sharepoint folder.

Steps to reproduce the behaviour

Set a value in config file for skip_file

Complete Verbose Log Output

Skipping file - excluded by skip_dir config: ./<omitted>/~abc.zip
Skipping file - excluded by skip_dir config: ./<omitted>/~def.csv

Screenshots

No response

Other Log Information or Details

No response

Additional context

No response

[abraunegg]

@alantgpl
I had noticed this before, but not focused specifically on this - so thanks for raising it.

One other note:

.......
Config option 'force_http_11'                = false
Config option 'ip_protocol_version'          = 0

and

curl 7.81.0

Please read https://github.com/abraunegg/onedrive/blob/master/docs/usage.md#compatibility-with-curl very carefully.

Your choice of distribution has given you a broken curl version that contains issues (bugs) when performing HTTP/2 operations.

If you build from master there are workarounds in place, and when a PR is developed for this, the PR will be based on master so you will get those work arounds until v2.5.3 is released - so in the meantime if you encounter any other issues - please ensure you read that URL very carefully.

[alantgpl]

Your choice of distribution has given you a broken curl version that contains issues (bugs) when performing HTTP/2 operations.

Thanks for this abraunegg.

As far as I know, Ubuntu backports security patches to cURL:
https://launchpad.net/ubuntu/jammy/+source/curl/+changelog

At least onedrive has been working fine for over a year with current config:

Config option 'force_http_11'                = false
Config option 'ip_protocol_version'          = 0

[abraunegg]

This has nothing to do with security - HTTP/2 bugs are not security issues that Ubuntu cares about.

The way this application uses curl changed in v2.5.x amd exposes these flaws not fixed by Ubuntu.

Choices are clear;

• apply work arounds
• upgrade curl
• dont use Ubuntu

[XenonSup]

Just wanted to add that this is also the case on Fedora Linux 40 (Workstation Edition) with onedrive-2.5.2-1.fc40.src.rpm
Let me know if there's any config test we could run

[abraunegg]

@alantgpl , @XenonSup
Please can you test PR #2917 to resolve your issue.

First install all the require platform dependencies to build the client on your respective platforms. Please read https://github.com/abraunegg/onedrive/blob/master/docs/install.md#building-from-source---high-level-requirements and then follow correctly for your platform.

Once this is done, to clone the PR to resolve your issue, you can use a script like the following:

#!/bin/bash

PR=2917

rm -rf ./onedrive-pr${PR}
git clone https://github.com/abraunegg/onedrive.git onedrive-pr${PR}
cd onedrive-pr${PR}
git fetch origin pull/${PR}/head:pr${PR}
git checkout pr${PR}

# Configure and Build
./configure --enable-debug --enable-notifications; make clean; make;
./onedrive --version

This script will create a local folder called onedrive-pr2917 with the PR version.

To run the PR, you need to run the client from the PR build directory:

./onedrive <any other options needed>

To install the PR, you will need to perform sudo make install to install the PR version to your system.

When running the PR, your version should be: onedrive v2.5.2-16-g5116a4b or greater.

[abraunegg]

@alantgpl , @XenonSup
Any update in testing the provided PR for resolution of this issue?

[XenonSup]

Thank you for the quick fix!
This does seem to resolve the configuration issue for me (see difference in reported config below) but I don't have the bandwidth to test it more thoroughly for overall functionality right now.

config file:

cat  ~/.config/onedrive/accounts/account1/config
sync_dir = "~/OneDrive"
log_dir = "/home/user/.var/log/onedrive_gui"
enable_logging = "true"
monitor_log_frequency = "5"
skip_file = "~*|.~*|*.tmp"
skip_dir = "/OnlineOnly|/Uploads|/Downloads|/Remote|/Local"

BEFORE

$ onedrive --confdir ~/.config/onedrive/accounts/account1/ --display-config
Reading configuration file: /home/user/.config/onedrive/accounts/account1/config
Configuration file successfully loaded
D-Bus message bus daemon is available; GUI notifications are now enabled
Application version                          = onedrive v2.5.2
Compiled with                                = LDC 2105
User Application Config path                 = /home/user/.config/onedrive/accounts/account1/
System Application Config path               = /etc/onedrive
Applicable Application 'config' location     = /home/user/.config/onedrive/accounts/account1/config
Configuration file found in config location  = true - using 'config' file values to override application defaults
Applicable 'sync_list' location              = /home/user/.config/onedrive/accounts/account1/sync_list
Applicable 'items.sqlite3' location          = /home/user/.config/onedrive/accounts/account1/items.sqlite3
Config option 'drive_id'                     = 
Config option 'sync_dir'                     = ~/OneDrive
Config option 'enable_logging'               = true
Config option 'log_dir'                      = /home/user/.var/log/onedrive_gui
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     = /OnlineOnly|/Uploads|/Downloads|/Remote|/Local
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial|~*|.~*|*.tmp
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 5
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = 
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.2
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 0
Config option 'threads'                      = 8
Environment var 'XDG_RUNTIME_DIR'            = true
Environment var 'DBUS_SESSION_BUS_ADDRESS'   = true
Config option 'notify_file_actions'          = false

Selective sync 'sync_list' configured        = true
sync_list config option 'sync_root_files'    = false
sync_list contents:

Config option 'sync_business_shared_items'   = false

Config option 'webhook_enabled'              = false

AFTER

$ ./onedrive --confdir ~/.config/onedrive/accounts/account1/ --display-config
Reading configuration file: /home/user/.config/onedrive/accounts/account1/config
Configuration file successfully loaded
Application version                          = onedrive v2.5.2-18-g7e0f78e
Compiled with                                = DMD 2109
Curl version                                 = libcurl/8.6.0 OpenSSL/3.2.2 zlib/1.3.1 brotli/1.1.0 libidn2/2.3.7 libpsl/0.21.5 libssh/0.10.6/openssl/zlib nghttp2/1.59.0 OpenLDAP/2.6.7
User Application Config path                 = /home/user/.config/onedrive/accounts/account1/
System Application Config path               = /etc/onedrive
Applicable Application 'config' location     = /home/user/.config/onedrive/accounts/account1/config
Configuration file found in config location  = true - using 'config' file values to override application defaults
Applicable 'sync_list' location              = /home/user/.config/onedrive/accounts/account1/sync_list
Applicable 'items.sqlite3' location          = /home/user/.config/onedrive/accounts/account1/items.sqlite3
Config option 'drive_id'                     = 
Config option 'sync_dir'                     = ~/OneDrive
Config option 'enable_logging'               = true
Config option 'log_dir'                      = /home/user/.var/log/onedrive_gui
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     = /OnlineOnly|/Uploads|/Downloads|/Remote|/Local
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 5
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = 
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.2-18-g7e0f78e
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 0
Config option 'threads'                      = 8
Config option 'max_curl_idle'                = 120
Environment var 'XDG_RUNTIME_DIR'            = true
Environment var 'DBUS_SESSION_BUS_ADDRESS'   = true
Config option 'notify_file_actions'          = false

Selective sync 'sync_list' configured        = true
sync_list config option 'sync_root_files'    = false
sync_list contents:

Config option 'sync_business_shared_items'   = false

Config option 'webhook_enabled'              = false

[abraunegg]

@alantgpl
Please can you respond. This issue is holding up prepping a v2.5.3 release.

[alantgpl]

Sorry @abraunegg I've had no time to test this. I just reverted to previous onedrive version for now. Appreciate your and XenonSup's work on fixing and testing.

[abraunegg]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.