[BUG] opensearch attempts to create a directory in path.home (and fails)

[MagicalTux]

Describe the bug
OpenSearch fails to run with the following exception:

java.lang.IllegalStateException: Unable to access 'path.home' (/pkg/main/app-misc.opensearch.core.2.8.0.linux.amd64/extensions)
Likely root cause: java.nio.file.FileSystemException: /pkg/main/app-misc.opensearch.core.2.8.0.linux.amd64/extensions: Read-only file system

Should be noted that /pkg/main is a read only filesystem (for security) used for programs/etc and should never be written to. path.data and path.logs are configured to point to the relevant directories.

After some investigation it looks like the root cause can be found in #6561 (which turned out to be more invasive than described)

To Reproduce
Steps to reproduce the behavior:

1. run opensearch from a read only filesystem with configuration to write outside of said filesystem
2. opensearch still attempts to write into readonly filesystem
3. See error

Expected behavior
Add a opensearch.yml configuration for extensions path, maybe path.extesions ?

Plugins
N/A

Screenshots
N/A

Host/Environment (please complete the following information):

• OS: Linux Azusa
• Version OpenSearch 2.8.0 (tar)

Additional context

WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/pkg/main/app-misc.opensearch.core.2.8.0.linux.amd64/lib/opensearch-2.8.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
[2023-07-10T13:50:46,213][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [subaru] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.home' (/pkg/main/app-misc.opensearch.core.2.8.0.linux.amd64/extensions)
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.8.0.jar:2.8.0]
	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.8.0.jar:2.8.0]
Caused by: java.lang.IllegalStateException: Unable to access 'path.home' (/pkg/main/app-misc.opensearch.core.2.8.0.linux.amd64/extensions)
	at org.opensearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:95) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Security.addFilePermissions(Security.java:319) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Security.createPermissions(Security.java:276) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Security.configure(Security.java:139) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:237) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.8.0.jar:2.8.0]
	... 6 more
Caused by: java.nio.file.FileSystemException: /pkg/main/app-misc.opensearch.core.2.8.0.linux.amd64/extensions: Read-only file system
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397) ~[?:?]
	at java.nio.file.Files.createDirectory(Files.java:700) ~[?:?]
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) ~[?:?]
	at java.nio.file.Files.createDirectories(Files.java:793) ~[?:?]
	at org.opensearch.bootstrap.Security.ensureDirectoryExists(Security.java:427) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:93) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Security.addFilePermissions(Security.java:319) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Security.createPermissions(Security.java:276) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Security.configure(Security.java:139) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:237) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.8.0.jar:2.8.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.8.0.jar:2.8.0]
	... 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: Unable to access 'path.home' (/pkg/main/app-misc.opensearch.core.2.8.0.linux.amd64/extensions)
Likely root cause: java.nio.file.FileSystemException: /pkg/main/app-misc.opensearch.core.2.8.0.linux.amd64/extensions: Read-only file system
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
	at java.base/java.nio.file.Files.createDirectory(Files.java:700)
	at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
	at java.base/java.nio.file.Files.createDirectories(Files.java:793)
	at org.opensearch.bootstrap.Security.ensureDirectoryExists(Security.java:427)
	at org.opensearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:93)
	at org.opensearch.bootstrap.Security.addFilePermissions(Security.java:319)
	at org.opensearch.bootstrap.Security.createPermissions(Security.java:276)
	at org.opensearch.bootstrap.Security.configure(Security.java:139)
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:237)
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
For complete error details, refer to the log at /home/magicaltux/.cache/opensearch/logs/dev.log

[dblock]

@saratvemulapalli take a look?

[dbwiddis]

@dblock I don't think @saratvemulapalli will be back before 2.9 release. I suspect we can quickly revert #6561 but it may be that we don't even need that directory anymore after we changed our initialization sequence. Investigating.

[dbwiddis]

We no longer use that directory anywhere. @owaiskazi19 will handle removing that offending line of code (and an unneeded test elsewhere).

[owaiskazi19]

@MagicalTux we have removed the extensionsdir as a part of this PR which is merged to 2.9 release branch. There was a leftover in the test for which I have raised another PR.