Multiple copies of files (binary files) are saved

[soheilred]

Application and Operating System Details:

• OS: 5.5.8-1-MANJARO
• GUI
• OneDrive Account Type: Non-business
• DMD or LDC compiler version: DMD64 D Compiler v2.090.1
• Application configuration: Output of onedrive --display-config
• Curl Version: curl 7.69.0
• Config:

onedrive version                    = v2.3.13
Config path                         = /home/soheil/.config/onedrive
Config file found in config path    = false
Config option 'check_nosync'        = false
Config option 'sync_dir'            = /home/soheil/OneDrive
Config option 'skip_dir'            = 
Config option 'skip_file'           = ~*|.~*|*.tmp
Config option 'skip_dotfiles'       = false
Config option 'skip_symlinks'       = false
Config option 'monitor_interval'    = 45
Config option 'min_notify_changes'  = 5
Config option 'log_dir'             = /var/log/onedrive/
Config option 'sync_root_files'     = false
Selective sync configured           = false

• Debug log: emailed to support@mynas.com.au

I have three computers (2 running Manjaro Cinnamon, and one running Kubuntu 19.04), and I use onedrive to sync my files. Whenever I change a binary file (edit a pdf, or compile a code which generates a bunch of executable file), onedrive starts to save a multiple copies of the changed files, which is very annoying. Sometimes it just doesn't stop, and populates tens of copies. Is there any way around this problem?

[abraunegg]

@soheilred
Please update your post with the required information as you have not followed these directions: https://github.com/abraunegg/onedrive#reporting-issues

Critically missing is what version you are running, what configuration you are using and a debug log.

[abraunegg]

@soheilred
Thanks for the debug log however I am not seeing anything in this log that is representative of the issue you are describing.

All this log is showing is the following:

• Validating data
• Checking local data

Please can you provide a debug log that clearly details and shows the issue you are facing. You 'might' need to manually run the application in monitor mode (--monitor) and perform your replication steps.

Please also do not attempt to sanitise the debug log file as in doing so you are destroying any correlative information that could be walked through to understand what is going on.

Additionally, what applications are you using to:

• Create the files
• Build the binaries

This will assist with re-creation steps if required.

[soheilred]

@soheilred
Thanks for the debug log however I am not seeing anything in this log that is representative of the issue you are describing.

All this log is showing is the following:

* Validating data

* Checking local data

Please can you provide a debug log that clearly details and shows the issue you are facing. You 'might' need to manually run the application in monitor mode (--monitor) and perform your replication steps.

Please also do not attempt to sanitise the debug log file as in doing so you are destroying any correlative information that could be walked through to understand what is going on.

Additionally, what applications are you using to:

* Create the files

* Build the binaries

This will assist with re-creation steps if required.

Thanks for your response. I will try to replicate the problem and have it in the debug file. I didn't want to share the information about the files I have in my onedrive. That's why I removed the file names in the debug file.
The problem is mostly produced when I am editing a pdf, for instance, and save it using one computer, when the same pdf is open in another computer. In this situation the problem happens for sure. However, this is not the only time that the issue occurs. For instance when I am compiling my c codes, with each make command that I issue, multiple copies of the bin file (executable file) is generated by onedrive client. In this situation, the problem is different than the previous one, as no other computer has the same file open. The same thing happens to synctex.gz files in a latex document directory. With every compilation of the tex file, there is a chance that the file duplicates.

[abraunegg]

@soheilred
Please can you provide the debug log ASAP & details of what application you are using to create / edit / save the PDF.

[soheilred]

@soheilred
Please can you provide the debug log ASAP & details of what application you are using to create / edit / save the PDF.

Debug file is sent to you.
I use okular to edit my pdfs. For compiling my tex files, I use pdflatex. And, to compile my c codes, I use gcc.

[abraunegg]

@soheilred
Thanks for the debug log, however this does not show or indicate any issue or problem.

Please can you:

1. On all your systems, ensure you are running the 'v2.3.13' or latest 'master' on ALL your systems
2. Provide a DEBUG log that captures the events occurring that multiple copies of files are being created

[soheilred]

@soheilred
Thanks for the debug log, however this does not show or indicate any issue or problem.

Please can you:

1. On all your systems, ensure you are running the 'v2.3.13' or latest 'master' on ALL your systems

2. Provide a DEBUG log that captures the events occurring that multiple copies of files are being created

I can confirm that I am running the latest version on all of my systems. In the debug file that I have sent, it shows ChengRo13.pdf is created three times when I edited it. Now I have the following files in my directory:

• ChengRo13.pdf
• ChengRo13-soheil-lab.pdf
• ChengRo13-soheil-x1.pdf
• ChengRo13-soheil-x1-2.pdf

Can you see all of these in the debug file?

[abraunegg]

@soheilred
You need to capture the DEBUG Log on the system that is creating those DUPLICATE files.

The log file that has been provided thus far shows NO edit / creation / rename activities - thus does NOT show why this is occurring. You probably need to capture the log file on 3 systems:

• soheil-lab
• soheil-x1
• soheil-x2

My current thoughts here is - this 'feels' like one of the original 'skilion' bugs where the 'hostname' of the linux system is added to the file name when there is a conflict. This I KNOW was fixed.

Please check ALL your systems - you may even need to uninstall any 'package', do a 'search on your system' to determine if there is any 'onedrive' binary still present, and if so remove it. Then compile and install from source to ensure you have the RIGHT version installed on all your systems.

[abraunegg]

@soheilred
Example:

On the system, where the 'duplication' is happening, you will see this sort of entry in your debug log file:

Remote item modified time is newer based on UTC time conversion
The local item is out-of-sync with OneDrive, renaming to preserve existing file: %oldname% -> %newname

The 'rename' ONLY occurs if:

• The file exists on the local disk
• The file does NOT exist in the local cache database of what has been synced

If the local file is newer, & in the database, you will get this sort of message and no rename:

Local item modified time is newer based on UTC time conversion - keeping local item as this exists in the local database

or

Local item modified time is newer based on UTC time conversion - keeping local item

So - please can you:

• Ensure that the system clock on all your systems is correct, configured to use NTP, has the correct date & time
• Verify what version you are running on all your systems - installing via package is not the right approach to do this.
• Perform the debug logging on ALL your systems at the same time, and critically watching for the above log entries

[soheilred]

@soheilred
Example:

On the system, where the 'duplication' is happening, you will see this sort of entry in your debug log file:

Remote item modified time is newer based on UTC time conversion
The local item is out-of-sync with OneDrive, renaming to preserve existing file: %oldname% -> %newname

The 'rename' ONLY occurs if:

* The file exists on the local disk

* The file does NOT exist in the local cache database of what has been synced

If the local file is newer, & in the database, you will get this sort of message and no rename:

Local item modified time is newer based on UTC time conversion - keeping local item as this exists in the local database

or

Local item modified time is newer based on UTC time conversion - keeping local item

So - please can you:

* Ensure that the system clock on all your systems is correct, configured to use NTP, has the correct date & time

* Verify what version you are running on all your systems - installing via package is not the right approach to do this.

* Perform the debug logging on ALL your systems at the same time, and critically watching for the above log entries

Alright! this is what I have done so far:

1. I made sure that the system clock is correct in all of my systems, and in fact they are all NTP synchronized now. Here is what I get issuing timedatectl

               Local time: Mon 2020-03-16 23:25:28 EDT
           Universal time: Tue 2020-03-17 03:25:28 UTC
                 RTC time: Tue 2020-03-17 03:25:28
                Time zone: America/New_York (EDT, -0400)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

2. I removed any old packages using the installation instructions on the github, downloaded the latest version of the app, and installed it using the installation instructions. Now, I have onedrive v2.3.13 on my Manjaros, and onedrive v2.3.13-24-g4cb0ebb on my Kubuntu.

The problem has not gone away yet.
One of the issues I am facing with creating the debug logs is that the command terminates after a few seconds on my Manjaros, so I might not be able to capture the issue in the debug files. That's, maybe, why you can only see the files created after the fact, and not at the middle of the process of making them. But, anyways, I will send what I have from all of my systems.

[abraunegg]

@soheilred

Now, I have onedrive v2.3.13 on my Manjaros

If you followed the instructions, you should have v2.3.13-24-g4cb0ebb on all your systems

One of the issues I am facing with creating the debug log is the command terminates after a few seconds on my Manjaros

This is not correct & should not happen - are you starting the application manually and not relying on any systemd / init.d script ?

[soheilred]

@soheilred

Now, I have onedrive v2.3.13 on my Manjaros

If you followed the instructions, you should have v2.3.13-24-g4cb0ebb on all your systems

One of the issues I am facing with creating the debug log is the command terminates after a few seconds on my Manjaros

This is not correct & should not happen - are you starting the application manually and not relying on any systemd / init.d script ?

I use systemd to enable and start the application.

[abraunegg]

@soheilred
As per the debug directions, you need to run the application manually - not via systemd / init.d to capture the logs

[soheilred]

@soheilred
As per the debug directions, you need to run the application manually - not via systemd / init.d to capture the logs

Oh, I am sorry about that! I wasn't aware of it. The debug directions Generate debug log for support didn't specify that though.
I have, now, stopped onedrive systemd services in all of the systems. And, also, I have installed onedrive v2.3.13-24-g4cb0ebb on my Manjaros too (previously, I was using the aur package that was suggested in the git repo to install them, and now, I am purely using the git package).
But, still
onedrive --synchronize --verbose --verbose > debug_output.log 2>&1
gets terminated every few seconds and I am not able to run continuously so that I can reproduce the problem.

[abraunegg]

@soheilred
When running this command:

onedrive --synchronize --verbose --verbose > debug_output.log 2>&1

This will sync only once, then finish.

If you want a continuous sync, replicating what the systemd service does, use:

onedrive --monitor --verbose --verbose > debug_output.log 2>&1

[abraunegg]

Analysis of log files performed. There is some sort of 'process' modifying the files locally. This needs to investigated and fixed first before further analysis can be done.

[abraunegg]

Update:
Based on all the application & file tracing, the following was occurring on the users systems:

• Dolphin File Indexer constantly accessing files and updating the modified timestamp. The process at fault was baloo_file_extractor. Every time this process ran, this created the issue.
• x_thumbnailer also appeared to touch and access every file
• kdeinit5 which is part of Okular appears to also touch and modify every file

As such, this issue is 100% not an issue with this application as, if a file is modified locally - either by process, or manually, the local file will be out-of-sync with OneDrive, and will be renamed to preserve existing file to prevent data loss IF there is also a modified & newer file listed as a change on OneDrive to download.

PR #831 has been written to provide a 'work around' for when these applications modify files locally - but really, none of them should be and I consider that they are actually misbehaving as they should never be editing / updating the modified timestamp of a file.

Closing this issue as not a bug. OP should open issue tickets with offending applications and complain to them and ask why is their application is modifying the timestamps of files in the way that it does.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.