PAYARA-3324 APPSERV-16 APPSERV-23 Allow Encryption of Web Session Data and SFSBs Stored in Hazelcast #4433

Pandrex247 · 2020-01-14T17:36:06Z

Description

This is a new feature to replace the encryption that used to be provided by Shoal (GMS).

In short, this is done by generating a symmetric encryption key, propagating it to each instance in the domain (in the same manner as the keystore), and changing the Hazelcast backing store used by the web and ejb containers to use the clustered store service (rather than accessing Hazelcast directly themselves) which checks whether or not it should encrypt/decode the values it stores/retrieves respectively (keys are not encrypted).

Values are stored in a new value-holder class to prevent Hazelcast jumping in front with its own non-overridable serialiser.

This PR brings in a new Asadmin CLI command, generate-encryption-key, which requires the master password and the domain to not be running. Encryption is enabled using the set-hazelcast-configuration command with a new option --encryptdatagrid: this only applies after a restart, regardless of whether or not --dynamic was specified. This is also a domain-wide option, it is not config specific.

Testing

New tests

New test added under appserver/tests/payara-samples/samples/datagrid-encryption/sfsb-passivation.
This test has a very simple EJB that stores data, which is poked a number of times to store data. The bean is then passivated by spamming a lookup of 1200 EJBs (default cache size is 512) before requesting it again to check if the data is still there and correct.
As part of the test set-up it generates a symmetric key, enables encryption, and restarts the domain to apply the changes - these changes are removed once the test has completed.

Testing Performed

The above test, plus essentially these two blogs:
Configuring Sticky Sessions for Payara Server with Apache Web Server & Session Replication in Payara Server with Hazelcast

Datagrid encryption was enabled before adding session data to the clusterjsp application noted in the blog. One instance was then killed to force failover, before starting it back up again and killing the other to force a second failover (checking that the data was correct on each instance between failovers).

This is a bit tricky to reliably automate in a portable fashion!

I also manually tested that the changes on whether encryption should or should not occur are only applied after a restart.

Test suites executed

Java EE7 Samples (with datagrid encryption enabled and disabled)

Testing Environment

Windows 10 64-bit, Zulu JDK 8u212.

Documentation

https://github.com/payara/Payara-Server-Documentation/pull/696

Pandrex247 · 2020-01-14T17:37:28Z

Jenkins test please

Pandrex247 · 2020-01-15T10:51:15Z

Something I hadn't tested before creating this PR: what happens if you change the master password after generating the key?

Answer: it breaks the key and you get Hazelcast exceptions on startup (but doesn't shut down the server or prevent joining the data grid)

Pandrex247 · 2020-01-15T11:16:20Z

I'll add a message to the "change-master-password" command that informs the user that they'll need to generate a new key - I can't change the key on the fly.

I'll also make it so the server fails to start if it can't read the key during initialisation.

jbee · 2020-01-16T07:48:55Z

jenkins test please

jbee

As you are still working on some details I submit some comments as a round 1 of reviewing the PR.

I did run your tests and they work (on linux).
Once I got

[ERROR] fish.payara.samples.datagridencryption.sfsb.SfsbEncryptionTest  Time elapsed: 35.02 s  <<< ERROR!
javax.ws.rs.ProcessingException: Failed to write asset to output: /WEB-INF/classes/fish/payara/samples/datagridencryption/sfsb/SfsbEncryptionTest.class
Caused by: org.jboss.shrinkwrap.api.exporter.ArchiveExportException: Failed to write asset to output: /WEB-INF/classes/fish/payara/samples/datagridencryption/sfsb/SfsbEncryptionTest.class
Caused by: java.lang.IllegalArgumentException: fish/payara/samples/datagridencryption/sfsb/SfsbEncryptionTest.class not found in classloader sun.misc.Launcher$AppClassLoader@7f31245a

but I think this is an arquillian/test runner issue.

I did not test the feature manually. Will do that later or in connection with round 2 of the review.

...a/ha-hazelcast-store/src/main/java/fish/payara/ha/hazelcast/store/HazelcastBackingStore.java

...-passivation/src/main/java/fish/payara/samples/datagridencryption/sfsb/TestEjbEndpoints.java

...rver/tests/payara-samples/test-utils/src/main/java/fish/payara/samples/ServerOperations.java

...mt/src/main/java/com/sun/enterprise/admin/servermgmt/cli/ChangeMasterPasswordCommandDAS.java

.../admin/server-mgmt/src/main/java/fish/payara/admin/servermgmt/cli/GenerateEncryptionKey.java

...ast-bootstrap/src/main/java/fish/payara/nucleus/hazelcast/encryption/SymmetricEncryptor.java

.../main/java/fish/payara/nucleus/hazelcast/encryption/PayaraHazelcastEncryptedValueHolder.java

...ast-bootstrap/src/main/java/fish/payara/nucleus/hazelcast/encryption/SymmetricEncryptor.java

...yara-modules/hazelcast-bootstrap/src/main/java/fish/payara/nucleus/store/ClusteredStore.java

...a-modules/hazelcast-bootstrap/src/main/java/fish/payara/nucleus/hazelcast/HazelcastCore.java

jbee · 2020-01-17T16:19:05Z

Had a second review. Main question mark for me is the use of salt in the actual encryption. This is more a question for @rdebusscher .

I'm still not sure about multi-threading safety for fields in HZ core service but if you think this is fine I am happy with that.

Not sure if I get to running the manual testing today. Otherwise it will be on Thursday when I am back.

rdebusscher

Some small improvements suggested

rdebusscher · 2020-01-20T09:20:10Z

.../admin/server-mgmt/src/main/java/fish/payara/admin/servermgmt/cli/GenerateEncryptionKey.java

+
+    private byte[] generateAndEncryptKey(char[] masterpasswordChars) throws CommandException {
+        byte[] saltBytes = new byte[20];
+        random.nextBytes(saltBytes);


If the salt is random there is no way to redo this even if we would know the master password. With random salt I assume there is no way to verify that an encryption key belongs to a certain master password?

@jbee Correct, but we also store the salt when we need the symmetric key. This implementation is correct. Lets recap the process.

We need a Symmetric Key for the encryption with the Data Grid (call it DataGridSymKey).

This key needs to be stored on disk because it needs to be transferred to to other nodes/instances. So we need to encrypt it when stored on disk.

We need to encrypt it with something user/domain specific which is already safely stored, like the master password

Password are bad for encryption because they are mostly too short, hence the use of the Key Derivation function.

PBKDF needs a salt and password to generate an enhanced 'password' (which is again a Symmetric Key EncryptionSymKey, which can be used to encrypt the DataGridSymKey.

When decoding to retrieve the DataGridSymKey, we need the EncryptionSymKey. This can be generated again using the Master password and the salt.

That is the reason why the salt is also part of the output (see line 165 and next)

So the salt is used to generate a unique key, although 2 systems have the same master password, the key is different .

rdebusscher · 2020-01-20T10:26:29Z

...in/server-mgmt/src/main/java/com/sun/enterprise/admin/servermgmt/cli/LocalServerCommand.java

+                if (attribute == null) {
+                    return false;
+                }
+                return Boolean.valueOf(attribute.getValue());


With Boolean.parseBoolean there is no autoboxing in the code.

rdebusscher · 2020-01-20T10:27:28Z

...zelcast-store/src/main/java/fish/payara/ha/hazelcast/store/HazelcastBackingStoreFactory.java

    @Override
    public <K extends Serializable, V extends Serializable> BackingStore<K, V> createBackingStore(BackingStoreConfiguration<K, V> bsc) throws BackingStoreException {
-        return new HazelcastBackingStore<K, V>(this, bsc.getStoreName(), core);
+        return new HazelcastBackingStore<K, V>(this, bsc.getStoreName(), clusteredStore);


Explicit types not needed.

rdebusscher · 2020-01-20T10:30:18Z

...trap/src/main/java/fish/payara/nucleus/hazelcast/encryption/HazelcastSymmetricEncryptor.java

+            throw new HazelcastException("Error reading encrypted key", ioe);
+        }
+
+        if (encryptedBytes == null) {


encryptedBytes is never null as Files.readAllBytes never returns null.

Remove unnecessary autoboxing and null check, and use diamond operator

Pandrex247 · 2020-01-21T12:22:03Z

Jenkins test please

jbee

Tested successful manually as well as running provided tests locally.

Only point I already mentioned in chat is that user feedback on active encryption in form of a log entry at startup and maybe something in the admin-console would be nice.

Pandrex247 · 2020-01-23T08:40:52Z

Jenkins test please

Pandrex247 added 7 commits January 14, 2020 16:50

PAYARA-3324 Generate encryption key and initial symmetric encryptor impl

2ddb2b4

PAYARA-3324 sync datagrid key, and throw unprocessed event

0a427d9

PAYARA-3324 Don't apply changes until restart and correct logic

9a3f942

PAYARA-3324 APPSERV-23 Initial impl of session data encryption

363119b

PAYARA-3324 Add SFSB Test

cf36e0d

PAYARA-3324 Copyright and formatting

62bcc30

PAYARA-3324 Moar copyright

36c779e

Pandrex247 changed the title ~~PAYARA-3324 Allow Encryption of Websession and SFSBs using Hazelcast~~ PAYARA-3324 Allow Encryption of Websession and SFSBs Stored in Hazelcast Jan 14, 2020

Pandrex247 changed the title ~~PAYARA-3324 Allow Encryption of Websession and SFSBs Stored in Hazelcast~~ PAYARA-3324 Allow Encryption of Websession and SFSBs stored in Hazelcast Jan 14, 2020

Pandrex247 changed the title ~~PAYARA-3324 Allow Encryption of Websession and SFSBs stored in Hazelcast~~ PAYARA-3324 APPSERV-16 APPSERV-23 Allow Encryption of Websession and SFSBs stored in Hazelcast Jan 14, 2020

Pandrex247 changed the title ~~PAYARA-3324 APPSERV-16 APPSERV-23 Allow Encryption of Websession and SFSBs stored in Hazelcast~~ PAYARA-3324 APPSERV-16 APPSERV-23 Allow Encryption of Web Session Data and SFSBs Stored in Hazelcast Jan 14, 2020

Pandrex247 requested review from jbee and rdebusscher January 15, 2020 10:46

PAYARA-3324 Prompt for key regeneration upon changing master password

9302434

jbee reviewed Jan 16, 2020

View reviewed changes

Pandrex247 added 3 commits January 16, 2020 11:58

PAYARA-3324 Kill Hazelcast if error reading encryption key

4d545b6

PAYARA-3324 Better null handling when checking if encryption is enabled

7d449bf

PAYARA-3324 Address most review comments

34b0369

jbee reviewed Jan 17, 2020

View reviewed changes

...a-modules/hazelcast-bootstrap/src/main/java/fish/payara/nucleus/hazelcast/HazelcastCore.java Outdated Show resolved Hide resolved

D'oh!

7a84767

rdebusscher approved these changes Jan 20, 2020

View reviewed changes

Pandrex247 added this to the 5.201 milestone Jan 21, 2020

Rdebusscher Review Comments

969f3cf

Remove unnecessary autoboxing and null check, and use diamond operator

jbee approved these changes Jan 22, 2020

View reviewed changes

APPSERV-41 Add encryption enabled log messages

bab2e6b

Pandrex247 merged commit 2348519 into payara:master Jan 23, 2020

Pandrex247 deleted the PAYARA-3324-Datagrid-Encryption branch March 29, 2022 14:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PAYARA-3324 APPSERV-16 APPSERV-23 Allow Encryption of Web Session Data and SFSBs Stored in Hazelcast #4433

PAYARA-3324 APPSERV-16 APPSERV-23 Allow Encryption of Web Session Data and SFSBs Stored in Hazelcast #4433

Pandrex247 commented Jan 14, 2020 •

edited

Loading

Pandrex247 commented Jan 14, 2020

Pandrex247 commented Jan 15, 2020

Pandrex247 commented Jan 15, 2020

jbee commented Jan 16, 2020

jbee left a comment

jbee commented Jan 17, 2020

rdebusscher left a comment

rdebusscher Jan 20, 2020

rdebusscher Jan 20, 2020

rdebusscher Jan 20, 2020

rdebusscher Jan 20, 2020

Pandrex247 commented Jan 21, 2020

jbee left a comment

Pandrex247 commented Jan 23, 2020

PAYARA-3324 APPSERV-16 APPSERV-23 Allow Encryption of Web Session Data and SFSBs Stored in Hazelcast #4433

PAYARA-3324 APPSERV-16 APPSERV-23 Allow Encryption of Web Session Data and SFSBs Stored in Hazelcast #4433

Conversation

Pandrex247 commented Jan 14, 2020 • edited Loading

Description

Testing

New tests

Testing Performed

Test suites executed

Testing Environment

Documentation

Pandrex247 commented Jan 14, 2020

Pandrex247 commented Jan 15, 2020

Pandrex247 commented Jan 15, 2020

jbee commented Jan 16, 2020

jbee left a comment

Choose a reason for hiding this comment

jbee commented Jan 17, 2020

rdebusscher left a comment

Choose a reason for hiding this comment

rdebusscher Jan 20, 2020

Choose a reason for hiding this comment

rdebusscher Jan 20, 2020

Choose a reason for hiding this comment

rdebusscher Jan 20, 2020

Choose a reason for hiding this comment

rdebusscher Jan 20, 2020

Choose a reason for hiding this comment

Pandrex247 commented Jan 21, 2020

jbee left a comment

Choose a reason for hiding this comment

Pandrex247 commented Jan 23, 2020

Pandrex247 commented Jan 14, 2020 •

edited

Loading