Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash during sync due to CLDAPI.dll error #3349

Closed
burakkaraceylan opened this issue May 22, 2021 · 163 comments · Fixed by #3461
Closed

Crash during sync due to CLDAPI.dll error #3349

burakkaraceylan opened this issue May 22, 2021 · 163 comments · Fixed by #3461

Comments

@burakkaraceylan
Copy link

burakkaraceylan commented May 22, 2021

Expected behaviour

Should sync the files.

Actual behaviour

Crashes.

Steps to reproduce

  1. Start nextcloud
  2. Press resume sync
  3. Crash

Client configuration

Client version: 3.2.1

Operating system: Windows 10 Build 19043

OS language: Turkish

Installation path of client: C:\Program Files\Nextcloud\nextcloud.exe

Logs

In the debugger the error comes down to:

cldapi.dll;IssueHsmControl()

"Unhandled exception at 0x00007FF8BFD24B2A (cldapi.dll) in nextcloud.exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFE."

@j4ym0
Copy link

j4ym0 commented Jun 7, 2021

Same issue here.

OS : Windows 10 (21390)
Lang : English
Nextcloud : Client 3.2.2
Server : Apache/docker 21.0.2.1

Windows Event vewer log

Faulting application name: nextcloud.exe, version: 3.2.2.25439, time stamp: 0x60af933e
Faulting module name: CLDAPI.dll, version: 10.0.21390.1, time stamp: 0x7bcd084b
Exception code: 0xc0000005
Fault offset: 0x00000000000157b7
Faulting process ID: 0x8924
Faulting application start time: 0x01d75b9493390e7d
Faulting application path: C:\Program Files\Nextcloud\nextcloud.exe
Faulting module path: C:\WINDOWS\SYSTEM32\CLDAPI.dll
Report ID: 093b4e15-dbdf-4d2f-91a0-c2ed233b2379
Faulting package full name:
Faulting package-relative application ID:

Nextcloud app log

2021-06-07 14:13:09:469 [ debug nextcloud.sync.database ] [ OCC::SyncJournalDb::commitInternal ]: Transaction commit "download file start2" and starting new transaction
2021-06-07 14:13:09:469 [ info nextcloud.sync.propagator ]: Completed propagation of "Elton John/One Night Only- The Greatest Hits [Live]/AlbumArt_{B5020207-474E-4720-9913-9C2E56BE0B00}Large.jpg" by OCC::PropagateDownloadFile(0x1cc7929fa00) with status OCC::SyncFileItem::Success
2021-06-07 14:13:09:469 [ debug nextcloud.sync.statustracker ] [ OCC::SyncFileStatusTracker::slotItemCompleted ]: Item completed "Elton John/One Night Only- The Greatest Hits [Live]/AlbumArt
{B5020207-474E-4720-9913-9C2E56BE0B00}Large.jpg" OCC::SyncFileItem::Success CSyncEnums::CSYNC_INSTRUCTION_NEW
2021-06-07 14:13:09:469 [ debug nextcloud.sync.database.sql ] [ OCC::SqlQuery::bindValue ]: SQL bind 1 5918834552439107580
2021-06-07 14:13:09:470 [ debug nextcloud.sync.database.sql ] [ OCC::SqlQuery::exec ]: SQL exec "SELECT path, inode, modtime, type, md5, fileid, remotePerm, filesize, ignoredChildrenRemote, contentchecksumtype.name || ':' || contentChecksum, e2eMangledName, isE2eEncrypted FROM metadata LEFT JOIN checksumtype as contentchecksumtype ON metadata.contentChecksumTypeId == contentchecksumtype.id WHERE phash=?1"
2021-06-07 14:13:09:470 [ warning nextcloud.gui.activity ]: Item "Elton John/One Night Only- The Greatest Hits [Live]/AlbumArt
{B5020207-474E-4720-9913-9C2E56BE0B00}Large.jpg" retrieved resulted in ""
2021-06-07 14:13:09:470 [ warning nextcloud.gui.activity ]: Item "Elton John/One Night Only- The Greatest Hits [Live]/AlbumArt
{B5020207-474E-4720-9913-9C2E56BE0B00}_Large.jpg" retrieved successfully.
2021-06-07 14:13:09:470 [ info nextcloud.gui.activity ]: Successfully added to the activity list: ""
2021-06-07 14:13:09:471 [ debug nextcloud.gui.folderwatcher ] [ OCC::WatcherThread::watchChanges ]: Skipping syncing of "C:/Users/jay_m/Music/Elton John/One Night Only- The Greatest Hits [Live]"

Nextcloud folder log

#=#=#=# Syncrun started 2021-06-07T13:12:59Z
#=#=#=#=# Propagation starts 2021-06-07T13:13:05Z (last step: 6060 msec, total: 6060 msec)
13:13:05||Elton John/Greatest Hits 1970-2003 [Disc 2]/Folder.jpg|8|2|1622677925|906d73873b1d61672df4a215cb4c2b43|72762|00006438ocur7m1ooggd|4||200|0|0|1f56d3d2-3e20-4039-93d2-5cf06435ebd7|
13:13:06||Elton John/Greatest Hits 1970-2003 [Disc 2]/02 Levon.mp3|8|2|1622677884|484d9cabf09d4e0b8727a8de23ea706b|7777469|00976651ocur7m1ooggd|4||206|0|0|76bb7d97-81c9-43f1-a0c0-169288545cd2|
13:13:06||Elton John/One Night Only- The Greatest Hits [Live]/AlbumArtSmall.jpg|8|2|1622677965|ee84be59f32a8fdfb6356c83f3625f64|162014|00006422ocur7m1ooggd|4||200|0|0|9e61840a-0d56-4225-b246-409098f6881d|
13:13:06||Elton John/One Night Only- The Greatest Hits [Live]/17 I Guess That's Why They Call It The Blues.mp3|8|2|1622677954|93a1d64aca82a8d86748ce1456c39b5c|12388964|00006429ocur7m1ooggd|4||200|0|0|d5f110a6-f3d3-4aab-bff1-2dd3435d84a6|
13:13:06||Elton John/One Night Only- The Greatest Hits [Live]/14 Saturday Night's Alright (For Fighting).mp3|8|2|1622677936|61287127860985ee0ffae2e960ad3b4c|11162135|00976652ocur7m1ooggd|4||200|0|0|d7958547-859e-4d50-a69a-b2dd4d08a0d5|
13:13:09||Elton John/One Night Only- The Greatest Hits [Live]/AlbumArt_{B5020207-474E-4720-9913-9C2E56BE0B00}Small.jpg|8|2|1622677983|37868dcd101f012d8026389f4cb4a622|162014|00006426ocur7m1ooggd|4||200|0|0|3216f9ea-7b47-4297-9b81-3259e93c1f1f|
13:13:09||Elton John/One Night Only- The Greatest Hits [Live]/Folder.jpg|8|2|1622677993|e0220bda8b104ce915705867e98806b9|162014|00006427ocur7m1ooggd|4||200|0|0|38c14221-82f5-41fe-8dbd-2be9bc64b06b|
13:13:08||Elton John/One Night Only- The Greatest Hits [Live]/AlbumArt
{B5020207-474E-4720-9913-9C2E56BE0B00}_Large.jpg|8|2|1622677974|0d824be0aaa36537417cd0fc4214d0ef|162014|00006423ocur7m1ooggd|4||200|0|0|5443b146-3713-4a75-9bfd-f7a983b8cd75|

My initial thought had been if the file had been i use or corrupt could have caused the issue. but ruled out corruption very quickly with another instance of the server and multiple folders and sizes of folders and files. i had noticed "folderwatcher" so had disabled AV but issue is still persistent.
Even cleared the folder re setup the sync and added the files in one by one both from server and locally but issue was still persistent.

I have noticed that the DB can fail to write quite a lot although the app can crash before a DB write error occurs. @burakkaraceylan Do you get the "error writing metadata to database" error in the settings panel?

2021-06-07 14:25:44:619 [ debug nextcloud.sync.networkjob ] [ OCC::AbstractNetworkJob::slotFinished ]: Network job OCC::LsColJob finished for "/Music/Paramore/Brand New Eyes"
2021-06-07 14:25:44:619 [ debug nextcloud.sync.database.sql ] [ OCC::SqlQuery::bindValue ]: SQL bind 1 "Paramore/Brand New Eyes"
2021-06-07 14:25:44:619 [ debug nextcloud.sync.database.sql ] [ OCC::SqlQuery::exec ]: SQL exec "SELECT lastTryEtag, lastTryModtime, retrycount, errorstring, lastTryTime, ignoreDuration, renameTarget, errorCategory, requestId FROM blacklist WHERE path=?1 COLLATE NOCASE"
2021-06-07 14:25:44:620 [ debug nextcloud.sync.database.sql ] [ OCC::SqlQuery::bindValue ]: SQL bind 1 "Paramore"
2021-06-07 14:25:44:620 [ debug nextcloud.sync.database.sql ] [ OCC::SqlQuery::exec ]: SQL exec "SELECT lastTryEtag, lastTryModtime, retrycount, errorstring, lastTryTime, ignoreDuration, renameTarget, errorCategory, requestId FROM blacklist WHERE path=?1 COLLATE NOCASE"
2021-06-07 14:25:44:621 [ info nextcloud.sync.engine ]: #### Discovery end #################################################### 6221 ms
2021-06-07 14:25:44:621 [ info nextcloud.gui.activity ]: Activity/Notification/Error successfully dismissed: "Error writing metadata to the database"
2021-06-07 14:25:44:621 [ info nextcloud.gui.activity ]: Trying to remove Activity/Notification/Error from view...
2021-06-07 14:25:44:621 [ info nextcloud.gui.activity ]: Activity/Notification/Error successfully removed from the list.
2021-06-07 14:25:44:621 [ info nextcloud.gui.activity ]: Updating Activity/Notification/Error view.
2021-06-07 14:25:44:622 [ debug nextcloud.sync.database.sql ] [ OCC::SqlQuery::exec ]: SQL exec "SELECT fingerprint FROM datafingerprint"
2021-06-07 14:25:44:622 [ info nextcloud.sync.engine ]: #### Reconcile (aboutToPropagate) #################################################### 6222 ms

Any ideas on what to try next? happy to try and help

@mmdanggg2
Copy link
Contributor

mmdanggg2 commented Jun 17, 2021

I've run into this as well. Digging in a bit, it's crashing here:

const qint64 result = CfSetPinState(handle.get(), cfState, flags, nullptr);

Here's a bit more debug info:
image
image

I tried building nextcloudsync.dll with optimizations disabled /Od to get more info but it's not crashing and instead giving sync errors:
||file/path.xxx|8|2|1590070825|60cb51720c7c5|21504386|00034415oc2h86xyea2m|1|Error writing metadata to the database|0|0|0||

mmdanggg2 added a commit to mmdanggg2/desktop that referenced this issue Jun 18, 2021
mmdanggg2 added a commit to mmdanggg2/desktop that referenced this issue Jun 18, 2021
Fixes nextcloud#3349

Signed-off-by: James Horsley <jbhorsley@gmail.com>
@jctace
Copy link

jctace commented Jun 30, 2021

Same issue on 3.2.3 for me:

Windows Event vewer log

Faulting application name: nextcloud.exe, version: 3.2.3.25536, time stamp: 0x60d461a4
Faulting module name: CLDAPI.dll, version: 10.0.22000.1, time stamp: 0x8f746c0f
Exception code: 0xc0000005
Fault offset: 0x00000000000157b7
Faulting process id: 0x5734
Faulting application start time: 0x01d76dc019beb92a
Faulting application path: C:\Program Files\Nextcloud\nextcloud.exe
Faulting module path: C:\WINDOWS\SYSTEM32\CLDAPI.dll
Report Id: 02ec1b70-bbca-4504-8c42-8c0e23523562
Faulting package full name: 
Faulting package-relative application ID: 

We all seem to be on Insider Preview Builds seems like the app doesn't like newer versions of CLDAPI.dll

@urbenlegend
Copy link

We all seem to be on Insider Preview Builds seems like the app doesn't like newer versions of CLDAPI.dll

I was running into this issue when I was on Windows 10 as well. My account was part of the Windows Insider program, but my machine was not subscribed to any flights at the time. Problem persists into Windows 11.

@jctace
Copy link

jctace commented Jul 7, 2021

Still an issue for me with 3.2.4

mmdanggg2 added a commit to mmdanggg2/desktop that referenced this issue Jul 8, 2021
Fixes nextcloud#3349

Signed-off-by: James Horsley <jbhorsley@gmail.com>
@jbruck
Copy link

jbruck commented Jul 9, 2021

Also an issue on my Win 10

Faulting application name: nextcloud.exe, version: 3.2.3.25536, time stamp: 0x60d461a4
Faulting module name: CLDAPI.dll, version: 10.0.19041.1081, time stamp: 0xadcd83c9
Exception code: 0xc0000005
Fault offset: 0x00000000000159ba
Faulting process id: 0x250c
Faulting application start time: 0x01d7747922a035ab
Faulting application path: C:\Program Files\Nextcloud\nextcloud.exe
Faulting module path: C:\WINDOWS\SYSTEM32\CLDAPI.dll
Report Id: 5572d573-e799-47e5-b750-3e2bfccbfa57
Faulting package full name: 
Faulting package-relative application ID: 

I tried looking at the nextcloud desktop client log file, after reaching some 500MB of logs checking files upon startup, the client crashes and there is nothing in the log that looks to be helpful, it just stops.

Doesn't look like there are relevant messages on the server either.

This is a reliable crash on the most recent desktop client.

@4iter4life
Copy link

4iter4life commented Jul 25, 2021

same here:
Windows 10 21H1 (build 19043.1147)
Nexcloud desktop client version: 3.2.4-x64

Name der fehlerhaften Anwendung: nextcloud.exe, Version: 3.2.4.25618, Zeitstempel: 0x60e484a1 Name des fehlerhaften Moduls: CLDAPI.dll, Version: 10.0.19041.1081, Zeitstempel: 0xadcd83c9 Ausnahmecode: 0xc0000005 Fehleroffset: 0x00000000000159ba ID des fehlerhaften Prozesses: 0x4c60 Startzeit der fehlerhaften Anwendung: 0x01d7816b76d981be Pfad der fehlerhaften Anwendung: C:\Program Files\Nextcloud\nextcloud.exe Pfad des fehlerhaften Moduls: C:\WINDOWS\SYSTEM32\CLDAPI.dll

@pwnorbitals
Copy link

(Same)

@dawid-woitaschek
Copy link

Just installed the client on W11 and yeah, crashes for me after creating some virtual files...

@LucaOverflow
Copy link

I had the same behavior of the app on stable w10. But maybe it had another reason? Can someone still confirm this bug on a regular w10 build?
Im Running the dev w11 as well.

mmdanggg2 added a commit to mmdanggg2/desktop that referenced this issue Jul 30, 2021
Fixes nextcloud#3349

Change placeholder functions to take QStrings instead of handles

Signed-off-by: James Horsley <jbhorsley@gmail.com>
@4iter4life
Copy link

@LucaOverflow yes its on regular Windows 10 2H1 (build 19043.1147)

allexzander pushed a commit that referenced this issue Aug 3, 2021
Fixes #3349

Change placeholder functions to take QStrings instead of handles

Signed-off-by: James Horsley <jbhorsley@gmail.com>
@allexzander
Copy link
Contributor

allexzander commented Aug 3, 2021

You may want to try this build with a potential fix to CLDAPI from @mmdanggg2 https://cloud.nextcloud.com/s/9CiRxYQaFnXcwqK?

Updated Link: https://cloud.nextcloud.com/s/JGj2FXW5mTGqmPj

@LucaOverflow
Copy link

LucaOverflow commented Aug 3, 2021

The Client isn't crashing anymore. But at least for the last 30 minutes it isn't syncing as well.
It searches for changes and after that i get the following error code:
image
after a while it starts scanning for changes again.

I will let it run for now.. maybe it will sync after a while.
I tried deleting the sync.db with no luck.
Anything I can debug to help?

Edit:
I tried it on a smaller folder so it doesn't need to scan for 15 minutes all the time. Eventually all virtual files are synced and the client downloads them on demand. But in the process i get a lot of those error messages above and after every message it starts scanning again.
Right now i can just let it do it's thing in the background. I will take a deeper look into this issue when i have more time. I already found some issues related to this message.

@4iter4life
Copy link

fixed for me. Thx for sharing.
Yes its resync after start (but was expected behavior for me).

Runs for half hour stable. Can u provide a MergeRequest for this Build?

@allexzander
Copy link
Contributor

@LucaOverflow I am wondering if there is a pattern for those files. Do they have something in common? Those with Couldn't update placeholder info I will check it again tomorrow to see if I am getting the same error and if the issue is present in a normal build or is it something new with this build that's fixing a cras.

@allexzander
Copy link
Contributor

@4iter4life This is the original PR by @mmdanggg2 #3461 Currently it is under review and I hope we can merge it sooner as I can see it does fix an issue for you and for @LucaOverflow. I will just need to dig deeper into an issue that @LucaOverflow is having.

@dsahlberg-apache-org
Copy link

The issue has existed for a long time so I don't think switching back to an old CLDAPI.DLL would help. Also, trying to cherrypick/replace DLL files in C:\Windows\System32 is probably asking for trouble.

After reading through the excellent investigation by @mmdanggg2 and the win32 api documentation, I believe this behaviour in intentional by Microsoft. SOMETHING is causing the OpLock to be broken and the client is supposed to handle this, however it is not clear to me how to check for an OpLock break. Compare for example the description of Opportunistic locking in the SMB protocol (https://en.wikipedia.org/wiki/Server_Message_Block#Opportunistic_locking).

@gvansanden
Copy link

@dsahlberg-apache-org Probably so, but as it is spreading quickly I'm at a point where mass installing the client is the only option left (this crash blocks almost everything in explorer, also access to SMB shares). So before I take that drastic decision, I rather experiment on a VM which can easily be reverted.

@coldacid
Copy link

Is there any reason why the cherry-picked CLDAPI.DLL can't go into the same directory as nextcloud.exe? It's not a protected "known DLL" that Windows always force-loads from a known location, so having it in the same directory as nextcloud.exe should load that particular version of the DLL instead of the system one. It would be much safer than trying to replace the copy in System32.

@gvansanden
Copy link

@coldacid I don't think so because if I understand it correctly, it's used for the interaction between windows explorer and the Nextcloud client. So both will have to use it.

@coldacid
Copy link

With how dynamic linking works on NT-based Windows, there's no sharing loaded DLLs between different processes. This would be an issue if the client was running within the Explorer process, but running in its own process it can have its own copy of a DLL.

@gvansanden
Copy link

Sorry to ping again, but is there anything more we can test?
This problem strangely keeps on spreading to more PCs

@CoYoNq
Copy link

CoYoNq commented Aug 5, 2022

+1, please give this ticket priority. Its critical.

@coldacid
Copy link

coldacid commented Aug 8, 2022

Latest version essentially crashes on start now on my Windows 10 computer. On the other hand, it runs like a beauty on my new Windows 11 device. Unfortunately, it's the former I use the most.

@BasicUser22
Copy link

Is there a possible link with the #4226 bug ?
Because on my computer, I've got both problems

@gvansanden
Copy link

Is there a possible link with the #4226 bug ? Because on my computer, I've got both problems

Yes, I think it's caused by the same

@gvansanden
Copy link

@mgallien Any new builds we can test?
It's very reproducible on many systems now (and still spreading)

@CoYoNq
Copy link

CoYoNq commented Aug 23, 2022

I notice, during a sync that crashed, some kind of memory leak.
During a regular (and completed sync), in my actual scenario, the nc client uses to consume about ~50mb of memory in the process (and idles at ~30mb)
During a "going to crash" sync, i notice memory consumption grows up to ~150mb before crashing. In the middle of the resync, stop advancing on counting synced files, memory use of the process grows suddenly, and it ends crashing.
Hope this info helps.

@gvansanden
Copy link

I notice, during a sync that crashed, some kind of memory leak. During a regular (and completed sync), in my actual scenario, the nc client uses to consume about ~50mb of memory in the process (and idles at ~30mb) During a "going to crash" sync, i notice memory consumption grows up to ~150mb before crashing. In the middle of the resync, stop advancing on counting synced files, memory use of the process grows suddenly, and it ends crashing. Hope this info helps.

I guess the memory leak can be a consequence of the missing handle. I don't think it is the cause of the crashes

mgallien pushed a commit to mmdanggg2/desktop that referenced this issue Aug 31, 2022
Fixes nextcloud#3349

Change placeholder functions to take QStrings instead of handles

Signed-off-by: James Horsley <jbhorsley@gmail.com>
@gvansanden
Copy link

@mgallien Does that mean we have a confirmed fix!?
That would be so great

@mgallien
Copy link
Collaborator

mgallien commented Sep 1, 2022

I did a new build of a version that may work fine
please can you test this https://cloud.nextcloud.com/s/K4ynRkJJMFDw4xx
thanks to @mmdanggg2

@uberiain
Copy link

uberiain commented Sep 1, 2022

Testing on:
WIndows Server 1809 ( 17763.3287)

Clean installation.

By the moment it is going well. Maybe this is the solution.

NextCloud

@limatus
Copy link

limatus commented Sep 1, 2022

I did a new build of a version that may work fine please can you test this https://cloud.nextcloud.com/s/K4ynRkJJMFDw4xx thanks to @mmdanggg2

It’s also working for me!

@Krautmaster
Copy link

Oh wow will test it these days as I'm currently drunk at an Greek beach. Sounds perfect, thanks for the effort

@gvansanden
Copy link

I did a new build of a version that may work fine please can you test this https://cloud.nextcloud.com/s/K4ynRkJJMFDw4xx thanks to @mmdanggg2

It not only seems to work fine, it is X times faster even on a new install...

@CoYoNq
Copy link

CoYoNq commented Sep 1, 2022

Under test in production enviroment. Looks good! no crashes at the time (about 6 hours of heavy load)
Thanks!!!

@gvansanden
Copy link

It could not have been a better timing, it's rolling out to all my clients today.
I just started migrating away from the client, happy that we don't need to anymore and I hope it lands in a release soon

@bkraul
Copy link

bkraul commented Sep 2, 2022

So I am wondering, is the intention to have this merged into upstream and have it roll out with the official binaries? It is working great for me, but security-wise I would feel much better with the official binary (no offense, but you know how it goes, especially when you have secops asking about where is this binary coming from, etc).

mgallien pushed a commit to mmdanggg2/desktop that referenced this issue Sep 5, 2022
Fixes nextcloud#3349

Change placeholder functions to take QStrings instead of handles

Signed-off-by: James Horsley <jbhorsley@gmail.com>
@Ador-able
Copy link

我做了一个新版本的版本,可能会工作正常,请你能测试这个 https://cloud.nextcloud.com/s/K4ynRkJJMFDw4xx 感谢@mmdanggg2

For me, it can solve the #4410 this problem,

@TheThief
Copy link

TheThief commented Sep 7, 2022

Same here. The pin state error is gone on my PC that had it after installing that build.

tokun-mobica pushed a commit to tokun-mobica/desktop that referenced this issue Sep 11, 2022
Fixes nextcloud#3349

Change placeholder functions to take QStrings instead of handles

Signed-off-by: James Horsley <jbhorsley@gmail.com>
@ShadaxTheWizard
Copy link

I did a new build of a version that may work fine please can you test this https://cloud.nextcloud.com/s/K4ynRkJJMFDw4xx thanks to @mmdanggg2

Has this been merged into the official release? I'm still getting crashes on windows 11 and the link posted earlier is broken.

Screenshot_1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.