Support for FIPS compliance mode

[beanuwave]

Description

This PR makes FIPS mode available through the OPENSEARCH_CRYPTO_STANDARD=FIPS-140-3 environmental parameter instead of the tests.fips.enabled setting. It provides FIPS 140-3 support by replacing all BC dependencies with BCFIPS dependencies and making FIPS approved-only mode configurable at launch. Running this mode restricts the BCFIPS provider to rely solely on FIPS-certified ciphers.

• The fips.gradle build script is removed in order to support a single-build solution.
• All BC dependencies are replaced by BCFIPS.
• Fixed creation of password protected internal keystore for valuable settings at node start. The add-string command was made with additional --stdin option, which interferes with password input.
• The Password Matcher inside Identity-Shiro that relies on BC to check if hashed passwords match with OpenBSDBCrypt is replaced by the password4j implementation.
• Adds full support for the BCFKS format (*.bcfks) for key and trust stores, also making it the default.
• Google's truststore is converted to the BCFKS format.
• Makes the best guess of which store type is provided based on the filename extension.
• Store types are strictly limited to JKS, PKCS12, PKCS11, and BCFKS.
• Refactors PemUtils to parse private keys in formats EC, PKCS8, PKCS1, and DSA, with or without encryption, and with or without parameters.
• dependency ':libs:opensearch-common' was added to rest-client build, to support strict keystore types. It's also the reason for JVM bump JAVA8 to JAVA11.
• allow ingest-attachment plugin to run in FIPS mode, since BC dependencies find no use.
• The java.security file is added to the build to distinguish between FIPS and non-FIPS environments.
• The fips_java.security file is altered due to evolving security standards.
• The security.policy file is altered to grant necessary security permissions.

Runtime limitations (known so far) that come with enabling FIPS mode:

Adminis can continue to manage their systems without being impacted by this change. However, for those keen on FIPS compliance, the most common obstacle will likely be the requirement to set a stronger password for the internal keystore and also convert key and truststores to *.bcfks format.

• Does not allow empty passwords for keystores or private keys (they need to be at least 112 bits in strength).
• The ssl.verification_mode=NONE setting is not permitted.
• JKS and PKCS12 key and trust store types are not supported at all.
• The internal keystore cannot be auto-migrated from versions 1 or 2.
• Azure Classic Discovery plugin -> deprecated.
• SQL-CLI client.

Reasons for refactoring PemUtils, which is used by the Reindex API in cases of migrating data from a remote cluster that is TLS protected:

• Lack of support for evolving standards like PKCS#8.
• Password-Based Key Derivation Functions such as PBKDF-OPENSSL are not supported in FIPS mode in favor of the PBKDF2 standard.
• Java type safety.
• It is generally a good idea to let ASN1 annotation parsing be done by external security libraries.

Related Issues

opensearch-project/security#3420

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 6016d5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8e8ed47: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6016d5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[dblock]

Could use some help maybe from @cwperks or @peternied reviewing this, please.

[terryquigleysas]

@beanuwave @peternied
I agree this is all moving in the right direction and getting closer to FIPS support but I think there is much more verification required before it can be merged. Getting the tests working is a major step but I'd be wary of other gaps and side-effects.

Have you been able to perform any of the following testing?

• Tested a full build on a standalone cluster?
• Tested with other plugins, in particular the Security plugin?
• Tested on a FIPS-enabled environment, eg RHEL in FIPS mode?

As this stands (correct me if I'm mistaken), at the very least, OpenSAML would be broken and there may be multiple versions of the Bouncy Castle libraries running in the JVM resulting in jar hell?

@dancristiancecoi has raised opensearch-project/security#4915 to discuss the OpenSAML options.

[terryquigleysas]

There is also a potential issue on setting this password. On a FIPS-enabled environment and org.bouncycastle.fips.approved_only=true this appears to throw the following exception. Have you seen this?

Exception in thread "main" org.bouncycastle.crypto.fips.FipsUnapprovedOperationError: password must be at least 112 bits
        at org.bouncycastle.crypto.fips.FipsPBKD$Parameters.<init>(Unknown Source)
        at org.bouncycastle.crypto.fips.FipsPBKD$Parameters.<init>(Unknown Source)
        at org.bouncycastle.crypto.fips.FipsPBKD$ParametersBuilder.using(Unknown Source)
        at org.bouncycastle.jcajce.provider.ProvPBEPBKDF2$BasePBKDF2.engineGenerateSecret(Unknown Source)
        at java.base/javax.crypto.SecretKeyFactory.generateSecret(SecretKeyFactory.java:340)
        at org.opensearch.common.settings.KeyStoreWrapper.createCipher(KeyStoreWrapper.java:349)
        at org.opensearch.common.settings.KeyStoreWrapper.encrypt(KeyStoreWrapper.java:434)
        at org.opensearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:545)
        at org.opensearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java

[beanuwave]

Yes, it's also the reason why all ITs do have keystore password set when executed in FIPS mode. I've also listed this requirement under 'Runtime limitations' in the opening description.

[terryquigleysas]

Yes, it's also the reason why all ITs do have keystore password set when executed in FIPS mode. I've also listed this requirement under 'Runtime limitations' in the opening description.

Cool, that makes sense.

I think there are ways that this code can be made more forgiving for FIPS mode - one possibility could be to apply the KEYSTORE_PASSWORD if set in the environment? It may need more investigation to confirm.

[beanuwave]

@beanuwave @peternied I agree this is all moving in the right direction and getting closer to FIPS support but I think there is much more verification required before it can be merged. Getting the tests working is a major step but I'd be wary of other gaps and side-effects.

Have you been able to perform any of the following testing?

• Tested a full build on a standalone cluster?
• Tested with other plugins, in particular the Security plugin?
• Tested on a FIPS-enabled environment, eg RHEL in FIPS mode?

As this stands (correct me if I'm mistaken), at the very least, OpenSAML would be broken and there may be multiple versions of the Bouncy Castle libraries running in the JVM resulting in jar hell?

@dancristiancecoi has raised opensearch-project/security#4915 to discuss the OpenSAML options.

You are correct, the security plugin needs many tweaks before it can run with FIPS libs only. I've done no further digging since we've been discussing the OpenSAML issue. Regarding a full build on a standalone cluster, most scenarios are well tested through ITs - do you see the need for more test scenarios?

I'm also curious about the outcome when the test suite is executed against FIPS-enabled CentOS9. In some cases, such as RC4, the implementation is done by SunJCE and can be instantiated by any third-party framework, regardless of any restrictions by the OS. I am currently working on a solution to this issue.

[terryquigleysas]

@beanuwave @peternied I agree this is all moving in the right direction and getting closer to FIPS support but I think there is much more verification required before it can be merged. Getting the tests working is a major step but I'd be wary of other gaps and side-effects.
Have you been able to perform any of the following testing?

• Tested a full build on a standalone cluster?
• Tested with other plugins, in particular the Security plugin?
• Tested on a FIPS-enabled environment, eg RHEL in FIPS mode?

As this stands (correct me if I'm mistaken), at the very least, OpenSAML would be broken and there may be multiple versions of the Bouncy Castle libraries running in the JVM resulting in jar hell?
@dancristiancecoi has raised opensearch-project/security#4915 to discuss the OpenSAML options.

You are correct, the security plugin needs many tweaks before it can run with FIPS libs only. I've done no further digging since we've been discussing the OpenSAML issue. Regarding a full build on a standalone cluster, most scenarios are well tested through ITs - do you see the need for more test scenarios?

I'm also curious about the outcome when the test suite is executed against FIPS-enabled CentOS9. In some cases, such as RC4, the implementation is done by SunJCE and can be instantiated by any third-party framework, regardless of any restrictions by the OS. I am currently working on a solution to this issue.

Yes I appreciate that your tests have added much more coverage. I'm just wary of gaps, especially in other plugins, where testing in this area understandably doesn't exist yet. I find that running a standalone deployment will flush out more issues and was curious to know the testing approaches you had taken so far.

[github-actions]

❌ Gradle check result for af1f3ee: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for b7f6360: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for fdce790: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for b612556: SUCCESS

[github-actions]

❌ Gradle check result for 7a93e82: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 21dd2b2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 2fdb220: SUCCESS