Server-side encryption is incompatible with using an object store as a primary data store #22077
Comments
mashedkeyboard
added
0. Needs triage
|
Similar to #11826 |
|
I noted that the Edit: Actually, no, it was a different issue. The message I get after copying the keys in manually is a 500 error with |
|
Possibly related to #22018, although @juliushaertl may be able to offer some more insight on that front. |
|
Similar issue, using:
I can upload and download encrypted files, with some issues. In particular:
On further inspection, the leading blocks of the file are decrypted and downloaded correctly, but the last block of the file (or blocks? I don't know the block size used by the cipher/Nextcloud) is instead an HTML Nextcloud error page. For example, for a JPG file of 77553 bytes, the download is 80059 bytes. The discrepancy between the two files is that their first 65536 bytes (
Logs are below: Upload request logsPreview request logsDownload request logs |
@mashedkeyboard, I believe this is because the object storage holds the keys. I see an additional 32-byte file and 512-byte file in my S3 bucket for each encrypted file, and I think these are used for decryption and integrity checking. |
|
If the object storage does actually store the keys, that seems like a wasted opportunity to improve security by storing the keys separately to the data. |
|
@mashedkeyboard, it is my understanding that different AES keys are used to encrypt each file. Each of these "file keys" is then encrypted with the same "master key / key encryption key". These encrypted file keys are stored in object storage, and the master key is stored on the Nextcloud server. |
|
That follows my understanding too, @jivanpal - I'd thought, however, that the generated master key was meant to reside in |
Haha, I'm new here, too. Just trying to figure things out from the documentation. The database table @mashedkeyboard, it seems you are right that the master key ought to be stored on the Nextcloud server in
It seems to me that
|
It turns out that the private key files are protected with the user's password (doh!), or the server secret from |
|
It would be helpful to have someone who's worked on these parts of the code confirm this, so that assuming it is the case, it can be documented somewhere (unless it is already, but if it is, I've missed it, and I assume @jivanpal has too). |
|
@mashedkeyboard It's all here, but it's not the clearest thing in the world. In particular, it mentions a "master key only" encryption scheme as well as the asymmetric crypto scheme that I've seen used, but it appears that the "master key only" scheme is not used at all. Here's the relevant section on private keys:
|
|
I believe I have the same issue described by @mashedkeyboard . I posted lots of details here a few days ago. I read the discussion above and the (apparently similar) issue #11826, but it's not clear to me that we understand yet what's going on. Or do we? To offer a detail that I haven't seen described (pardon me if I missed it): in my tests I found that files can occasionally be downloaded/previewed correctly after a couple of attempts. However, the errors I see in the log still appear, whether or not the file is downloaded correctly. |
|
@oliversturm, interesting that you can sometimes download a file, sometimes not (regardless of whether the downloaded file matches the original file). I can always download a file (albeit with the last block being incorrectly retrieved) but never preview it in the web interface, and if I understand @mashedkeyboard correctly, he can never do either of these. As such, I am not entirely sure that all our issues are related, but I am very much interested in getting this working, so I will be looking into it this weekend. I will be checking whether encryption on local storage works as it should. If it doesn't, then I expect an issue with the encryption library; but if it does, then I expect an issue with how the S3 and encryption libraries are used and/or how data is being passed around. |
|
I have not tested this yet, but I have a hunch that this could have something to do with #20033, which enabled fseek support for object storage. I'll try to get a test environment setup to test with this reverted. |
|
@jivanpal I have just observed a second (for me) behavior for the first time. Previously, I was testing mainly with PDF files, some images and videos. The test with PDF files was easy to do from the client side - the first time I clicked a file in the web frontend, the PDF viewer would come up and its progress bar move to around 70% - then nothing. Then I would navigate back and click the same file again, and it would come up correctly in the PDF viewer. This pattern was fully reproducible for me, and still is - I just tried it yesterday with a number of newly uploaded PDF files. Today I have a different problem though. I uploaded a zip file, and now I can't download it. If I use the download action in the web interface, I see a blue "internal server error" screen. If I use a public share link, the browser shows an error (something like "can't open the link") when I start the download. I have tried this numerous times, and it is impossible to download the file. One additional note: I also have a VM set up to sync a large (~22GB) local folder with my Nextcloud installation, using the Linux desktop client. All the files were initially uploaded without any trouble. I have added and changed files since, and the VM uploaded these correctly. Now I just tried to add a file through the web interface and see what the sync client does. Predictably, it shows an error: I really hope we can find out what this is all about... I was just getting started with Nextcloud and found the setup promising, but I don't know where to look for these bugs and I'm very hesitant to proceed with my migration at this point. |
|
And now it appears that things have changed again. Here's what I did:
Now I can't access a PDF file that was previously readable (at least on second attempt). The browser displays an error message if I try - this didn't happen previously. In Admin/Logging I see this: For some reason, I see a second part to the exception message in the web frontend, but this is dropped when I copy it to the clipboard. Here it is, copied manually: Finally, there are still some files I can access correctly, first time - images and PDF files. Others I can't access at all now. What to do? This setup has become completely unusable now and I'm wondering if I'm losing my data. |
|
#8349 has been mentioned above, but the poster said the issue was a different one after all. For me it is exactly the same now. The issue is from February 2018, that doesn't look promising. I also found this report: https://help.nextcloud.com/t/encryption-not-ready-multikeydecrypt-with-share-key-failed/42806 Forgive me if this is a stupid question, but I'm new to Nextcloud. Unless I'm going to read loads of source code and debug it myself, is there hope this may be fixed? I thought NC was a well-supported and production-ready solution, but this bug is a complete showstopper, at least 30 months old and it doesn't appear that anybody has been looking into it. I don't mean any offense, but I need to weigh my options. |
|
I think there are possibly several bugs at play here; one is #8349, but there seems to me to be another one as well. @juliushaertl suggested that #22018 may fix one of them, I don't know if the other one will be fixed in that PR as well. For what it's worth, @oliversturm, I'm in much the same position as you; new to Nextcloud, tried to set it up with this, saw that it didn't work and subsequently haven't set it up. Clearly, this is FOSS software, and there's no expectation on anyone individually to make this work; however, encryption definitely should not be marked as a production-ready feature if there are bugs like these still in it. I've not had the time to look deep into the code to try and root out the bug myself, and I wouldn't expect the same of anyone else volunteering on the project, but people shouldn't be misled into thinking support for encryption is further along than it actually is. No disrespect whatsoever to the people who've worked on encryption in Nextcloud - it looks like a clever system - but it can't reasonably be called complete or production ready if there's serious risk of irretrievable data loss when using object storage, at the very least without a large warning attached to it saying "do not enable this with primary object storage". |
|
So... trying to figure out what could have happened in my installation so that the behavior changed compared to my initial report. I read in some other issue that somebody's master key changed as a result of an update. I would like to double-check my key - but where is it? All the docs I can find mention locations in the filesystem underneath |
|
@oliversturm, @jivanpal mentioned earlier that the keys are stored in object storage when it's being used as primary storage, with the keys themselves encrypted with the user's password and/or server secret. I've spent this evening trying to think of ways around this problem, and ended up using goofys, a fuse filesystem for S3 written in go, to map my data directory directly to an S3 bucket. That way, Nextcloud doesn't mess up, as from its perspective, it's just looking at part of the server's disk; S3 is all handled by the FS layer. That's definitely a suboptimal solution, but it's working for now, and it does allow encryption so long as the option to encrypt the home storage is enabled. |
And that server secret is what? The
Hm... yeah. I'm afraid I'm not interested anymore - what I've seen of Nextcloud has made it very clear to me that this is not the platform I will trust with my data. |
|
@oliversturm, very odd behaviour on your end. Could be an issue with the Snap package you're using, but we should rule out upstream issues first. It's a shame to hear that this has discouraged you from using Nextcloud, but it's completely understandable. I'm in the same boat as you, new to using it, continuing to use Google Drive until I'm confident it has no issues for my use case and I have a good backup system in place. I haven't had time to explore this yet, but will be doing so today. |
Just to be clear, it was not this issue in itself. It is a bad issue with potentially serious consequences, and it clearly deserves immediate and intense attention. However, this issue is now three weeks old, and the only comments recorded here are from people with the same problem and no real understanding of NC. And then there are other issues, up to 30 months old, which seem to indicate the same problem. Yes it's FOSS and we are morally obliged not to expect others to fix our problems. But in reality, somebody once wrote the code related to this combination of S3 and server-side encryption, and obviously they didn't test it properly. In addition, they have now been ignoring reports about the deficiencies of said code for years. Perhaps somebody else should step up to fix these issues? Probably, and probably somebody will. But in the meantime I am (or was going to be) "just" a user of this tool called Nextcloud - my life is already full of projects, I can't be a developer on every single tool I use. And, to put it bluntly and colloquially, this particular tool seems to be broken in a very important and dangerous way, and nobody cares. So I can't justify using it - at least until I return with more time on my hands, ready to fix important security issues myself. |
|
The issue relates to the way in which the signature of the final block of an encrypted file's ciphertext is computed — probably an overflow issue. Not sure yet whether the issue is:
As a workaround for now, removing the signature check in I'll see if I can figure out a proper fix, hopefully I'll have time tomorrow and on Friday to work on this. |
|
Hey @jivanpal, did you make any headway on this in the end? No pressure - it's not your job to! Just wanted to check before I start throwing myself further into it to avoid duplication of effort. |
szaimen
added
1. to develop
|
is there now a known solution to this problem? Thanks |
|
I also would like to see this addresses. Just wanted to switch to NC and setup a professional installation with S3 just to read about this major pitfall. Either using S3 or SSE is something that shouldn't be a mutual exclusive decision to make. That'*s a real deal breaker for me :-( |
|
you can fix files with "bad signature" using this new command: https://docs.nextcloud.com/server/latest/admin_manual/issues/general_troubleshooting.html#problems-when-downloading-or-decrypting-files I haven't tested object store primary storage with server side encryption, but I'm surprised that it doesn't work anyone got a chance to test with NC 22.2.0 ? |
I will today and i tell you :) |
So it's much difficult that i 'm thinking .. all of my "corrupts" files are in files_versions directory and i can't scan this folders .. only files directory of users nextcloud. If someone have i idea, i'll take :) |
|
Bump............. Configured Wasabi (S3-compatible) and cannot view or download any files. Seeing bad sig in the log files. Ran the command above on the entire user account (only one user) and it see this no matter how many times I run it
Command does not fix. Decrypted all, disabling server encryption - everything is accessible, everything can be downloaded S3 + encryption is broken with NextCloud? Encryption works fine if stored locally. |
|
I've found this: #28389 |
|
Just adding my own findings to this issue, when using AWS Lightsail with Ubuntu and the latest snap install of NextCloud, I get the same issue when S3 is the primary storage (S3 provided by Wasabi) If I revert to using the server storage and S3 as external only, all work flawlessly across all devices (macOS and iOS). Therefore it seems, at least in my case, this only occurs when S3 is the primary storage object and does not occur when S3 is simply used as external storage. Personally speaking, not being able to use S3 as the primary storage is a dealbreaker and I'm somewhat unnerved that this issue is ongoing for about 2 years. |
|
Good information! Indeed, I was forced to exclude sensitive data from Nextcloud since I'm using S3 unencrypted server storage at present due to this bug. |
|
Same issue with NC 23.0.0 |
|
Any update on this? I would love to start using Nextcloud for my organization when this functionality becomes available. |
|
there's a WIP PR here that fixes size issues: #31966 |
|
I just upgraded to v24, and the issue seems to be intermittent. In v22, files would fail to load regularly. Now, errors are being thrown, but the files eventually open. Is this consistent with what everyone else is experiencing?
@PVince81 - what are the odds that this PR gets merged? I may patch my system now if it is likely to be approved. |
|
the odds get higher if you help testing |
|
Now that PR #31966 has been merged and backported to 24.0.2 (#32708), is this issue fixed? If it is expected to work, I'll take the time to give it a try and give you a feedback here. |
|
yes, this is expected to work with 24.0.2. closing |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
and others
How to use GitHub
Steps to reproduce
Expected behaviour
The file should work just like any other file, being stored in the objectstore and being encrypted at rest there, with encryption keys remaining on the Nextcloud host.
Actual behaviour
Attempting to preview or open the file results in various 503 and 500 errors, with the message
Bad Signature. The file is unrecoverable. This behaviour is consistent irrespective of whether you enable SSE before you enable S3, or the other way around.Server configuration
Operating system: Ubuntu 20.04
Web server: Nginx 1.18.0
Database: MySQL 8.0.21
PHP version: 7.4
Nextcloud version: (see Nextcloud admin page) 19.0.1
Updated from an older Nextcloud/ownCloud or fresh install: Fresh install
Where did you install Nextcloud from: Both web installer and manually unzipping
Signing status:
Signing status
List of activated apps:
App list
Nextcloud configuration:
Config report
Are you using external storage, if yes which one: S3
Are you using encryption: Yes
Are you using an external user-backend, if yes which one: No
Client configuration
Browser: Firefox 78.0.2
Operating system: Manjaro Linux
Logs
Web server error log
Web server error log
There are no recorded errors in the Nginx log.Nextcloud log (data/nextcloud.log)
Nextcloud log
Note the
OCP\\Files\\NotFoundException; this was because I initially logged in with an existing user, rather than deleting the user and re-adding a new one. Following this exception, I deleted the user, and added a new one; all log lines beyond that point are from the new user, who was created with S3 enabled.Browser log
Browser log
An example of one of the failed requests:Response:
The text was updated successfully, but these errors were encountered: