Figuring out why encryption on S3 object storage as primary storage is not working + trying to fix it

[lkreher]

Steps to reproduce

1. Install NC
2. Setup S3 Object Storage as primary storage
3. enable Encryption
4. upload files > 10kb
5. try to retrieve files

Expected behaviour

Files should be retrieved

Actual behaviour

Nextcloud throws an error and files can't be retrieved

Server configuration

Operating system: Ubuntu 16.04.3 LTS

Web server: nginx

Database: MySQL 5.7

PHP version: 7.2 FPM

Nextcloud version: 15.0.2

Updated from an older Nextcloud/ownCloud or fresh install: fresh install

Where did you install Nextcloud from: Nextcloud web installer

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - accessibility: 1.1.0
  - activity: 2.8.2
  - cloud_federation_api: 0.1.0
  - comments: 1.5.0
  - dav: 1.8.1
  - encryption: 2.3.0
  - federatedfilesharing: 1.5.0
  - federation: 1.5.0
  - files: 1.10.0
  - files_pdfviewer: 1.4.0
  - files_sharing: 1.7.0
  - files_texteditor: 2.7.0
  - files_trashbin: 1.5.0
  - files_versions: 1.8.0
  - files_videoplayer: 1.4.0
  - firstrunwizard: 2.4.0
  - gallery: 18.2.0
  - logreader: 2.0.0
  - lookup_server_connector: 1.3.0
  - nextcloud_announcements: 1.4.0
  - notifications: 2.3.0
  - oauth2: 1.3.0
  - password_policy: 1.5.0
  - provisioning_api: 1.5.0
  - serverinfo: 1.5.0
  - sharebymail: 1.5.0
  - support: 1.0.0
  - survey_client: 1.3.0
  - systemtags: 1.5.0
  - theming: 1.6.0
  - twofactor_backupcodes: 1.4.1
  - updatenotification: 1.5.0
  - workflowengine: 1.5.0
Disabled:
  - admin_audit
  - files_external
  - user_ldap

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "objectstore": {
            "class": "OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "bucket",
                "autocreate": true,
                "key": "key",
                "secret": "secret",
                "region": "eu-central-1",
                "use_ssl": true
            }
        },
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "xxx"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "check_data_directory_permissions": false,
        "dbtype": "mysql",
        "version": "15.0.2.0",
        "overwrite.cli.url": "xxx",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "maintenance": false
    }
}

Are you using external storage, if yes which one: S3 object storage

Are you using encryption: yes

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Chrome

Operating system: Windows

Logs

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"zjolUv0wdkO8YqTg4DKt","level":4,"time":"2019-02-04T11:05:56+00:00","remoteAddr":"192.168.10.1","user":"me","app":"webdav","method":"GET","url":"\/remote.php\/webdav\/Nextcloud%20Manual.pdf","message":{"Exception":"OC\\Encryption\\Exceptions\\DecryptionFailedException","Message":"Encryption library: Decryption (symmetric) of content failed: ","Code":0,"Trace":[{"file":"\/home\/vagrant\/Sites\/nextcloud\/apps\/encryption\/lib\/Crypto\/Crypt.php","line":473,"function":"decrypt","class":"OCA\\Encryption\\Crypto\\Crypt","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/home\/vagrant\/Sites\/nextcloud\/apps\/encryption\/lib\/Crypto\/Encryption.php","line":379,"function":"symmetricDecryptFileContent","class":"OCA\\Encryption\\Crypto\\Crypt","type":"->","args":["*** sensitive parameter replaced ***","*** sensitive parameter replaced ***","*** sensitive parameter replaced ***",1,"*** sensitive parameter replaced ***"]},{"file":"\/home\/vagrant\/Sites\/nextcloud\/lib\/private\/Files\/Stream\/Encryption.php","line":479,"function":"decrypt","class":"OCA\\Encryption\\Crypto\\Encryption","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/home\/vagrant\/Sites\/nextcloud\/lib\/private\/Files\/Stream\/Encryption.php","line":299,"function":"readCache","class":"OC\\Files\\Stream\\Encryption","type":"->","args":[]},{"function":"stream_read","class":"OC\\Files\\Stream\\Encryption","type":"->","args":[8192]},{"file":"\/home\/vagrant\/Sites\/nextcloud\/3rdparty\/icewind\/streams\/src\/Wrapper.php","line":91,"function":"fread","args":[null,8192]},{"file":"\/home\/vagrant\/Sites\/nextcloud\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php","line":91,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"Icewind\\Streams\\CallbackWrapper","type":"->","args":[8192]},{"file":"\/home\/vagrant\/Sites\/nextcloud\/3rdparty\/sabre\/http\/lib\/Sapi.php","line":80,"function":"stream_copy_to_stream","args":[null,null,"6276760"]},{"file":"\/home\/vagrant\/Sites\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":498,"function":"sendResponse","class":"Sabre\\HTTP\\Sapi","type":"::","args":[{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/home\/vagrant\/Sites\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"http:\/\/nextcloud.lars\/remote.php\/webdav\/Nextcloud%20Manual.pdf","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/home\/vagrant\/Sites\/nextcloud\/apps\/dav\/appinfo\/v1\/webdav.php","line":80,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"\/home\/vagrant\/Sites\/nextcloud\/remote.php","line":163,"args":["\/home\/vagrant\/Sites\/nextcloud\/apps\/dav\/appinfo\/v1\/webdav.php"],"function":"require_once"}],"File":"\/home\/vagrant\/Sites\/nextcloud\/apps\/encryption\/lib\/Crypto\/Crypt.php","Line":590,"Hint":"Encryption library: Decryption (symmetric) of content failed: ","CustomMessage":"--"},"userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/71.0.3578.98 Safari\/537.36","version":"15.0.2.0"}

Report

This was previously discussed for NC14 in #11826 and others. I've been trying to fix this issue, and noticed multiple problems:

1. The IV generated to encrypt one block uses random_bytes(16) at Crypt.php#634
   When written to file, read back and evaluated, this block gets interpreted as different character encoding, so 16 characters turn into more or less than 16 bytes.
   Changing Crypt.php#634 to return substr(base64_encode(random_bytes(16)), 0, 16); makes the IV safe from character encoding issues. This way I can successfully decrypt the first and the last block of each file, which means small files can be retrieved.

2. Blocks don't get read properly. In Encryption.php#473 less than 8192 bytes of data are returned after the first block, even when there are more than 8192 bytes remaining. Retrieving a partial block is guaranteed to result in decryption errors.
   I don't have a solution for this problem. The stream resource is being handled by a 3rdparty library which does not behave as expected.

Looking for someone with an idea how to tackle problem 2. I've been stepping through the process with xdebug for a while and can't figure out how to retrieve a full 8192 bytes at once.

[lkreher]

I managed to find a solution for problem 2:
Replacing Encryption.php#473 with

            $data = $this->data;
            while(!preg_match('/00sig00[0-9a-f]{64}xxx/', $data)) {
                $data .= parent::stream_read($this->util->getBlockSize());
            }
            preg_match('/(.*?00sig00[0-9a-f]{64}xxx)(.*)/', $data, $matches);
            $data = $matches[1];
            $this->data = $matches[2] ?? '';

ensures that always at least one full block of encrypted data is being loaded from the stream source.

In combination with the fix I posted above I can successfully store encrypted files on S3 object storage and retrieve them. Tested for files up to 32 MB so far, but I don't think there should be any issues with larger files. This will not allow for decryption of files that have been stored before the fix concerning problem 1 was applied.

[himpierre]

I would love to see a solution here too. Thanks for you effort though @lkreher

[jknockaert]

I will have a look at the suggested solution for problem 2.

[lkreher]

I will have a look at the suggested solution for problem 2.

Both of my "fixes" are supposed to be workarounds, because the issue is with a 3rdparty library that handles the stream resource.

I am however currently using Nextcloud with S3 as primary storage and encryption enabled using the two changes I mentioned above.

Concerning problem 1, I don't understand why the IV is originally written as binary data when all the encrypted data is converted to base64 to prevent character encoding issues in the first place.

[jknockaert]

I remember to have seen other issues with stream wrappers when we were developing the encryption wrapper in Encryption.php, so problem 2 doesn't really surprise me. Workarounds are fine if possible, but I first should have a look at the code. It's a while ago since I last worked on it.

As for problem 1 I have not been involved in developing that part of the code.

[craysiii]

I would just like to chime in and offer to test or help out with any sort of pull requests that relate to this issue. I am not a PHP developer but am willing to get my hands dirty to aid in getting this issue resolved.

[plumbeo]

Hi, is anyone working on this?

[kesselb]

Blocks don't get read properly. In Encryption.php#473 less than 8192 bytes of data are returned after the first block, even when there are more than 8192 bytes remaining. Retrieving a partial block is guaranteed to result in decryption errors.
I don't have a solution for this problem. The stream resource is being handled by a 3rdparty library which does not behave as expected.

Fixed by #15946

Would be nice if some of you with this issue could check with nextcloud 17 if the problem still exist.

[plumbeo]

@kesselb I tested nextcloud 17 (new installation) and I can confirm that the problems mentioned in this issue are solved. Once server-side encryption is enabled new files gets correctly encrypted and they can be previewed and downloaded (from the web interface, couldn't test the clients yet). the occ commands encrypt-all and decrypt-all work too, even when called repeatedly, and the files are always accessible.

There are a couple of remaining issues tho, and I'm not sure if they're related to the S3 object storage backend or if they're generic encryption issues:

1. after the unencrypted files already on the server are encrypted by encrypt-all their creation/modification times change to the time of encryption and the filetype isn't recognised so previews and thumbnails don't work. The same happens with decrypt-all.

2. after running encrypt-all/decrypt-all the file histories (in the activities section) get replaced by a single entry saying "Renamed by remote user."

3. the file size displayed on the web interface changes to the encrypted file size instead of the true file size. This happens both for files uploaded after encryption is enabled and for files encrypted by encrypt-all.

[goarano]

I've also tested this setup in nextcloud 17.
My experience is that while uploading files works, the first download (via the webinterface) fails, because the download manager tries to download a file with the encrypted file size (but the stream ends earlier).
After the failed download however, the webinterface displays the unencrypted file size and the download completes successfully.

[szaimen]

Let us track this in #22077