8370344: Arbitrary Java frames on stack during scoped access

[JornVernee]

See the JBS issue for a problem description.

This patch changes to the shared scope closure handshake to be able to handle 'arbitrary' Java frames on the stack during a scoped memory access.

For the purposes of this change, we assume that 'arbitrary' is limited to the following:

1. Frames added by calling the constructor of InternalError as a result of a faulting access to a truncated memory-mapped file (see HandshakeState::handle_unsafe_access_error). This is the only handshake operation (i.e. may be triggered during a scoped access) that calls back into Java.
2. Frames added by a JVMTI agent that calls back into Java code while handling a JVMTI event that happens during a scoped access.
3. Any other Java code that runs as part of the linking process.

For (1), we can assume that only code in java.base will be executed. So, if we see a frame outside of java.base, we can stop walking the stack. For (2), all bets are off. So, if any agents are loaded during the stack walk, we have to walk the entire stack. For (3), all code is also assumed to be in java.base.

The test TestSharedCloseJFR is added for scenario (1), and the test TestSharedCloseJvmti is added for scenario (2).

Testing: tier 1-3.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8370344: Arbitrary Java frames on stack during scoped access (Bug - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/27919/head:pull/27919
$ git checkout pull/27919

Update a local copy of the PR:
$ git checkout pull/27919
$ git pull https://git.openjdk.org/jdk.git pull/27919/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 27919

View PR using the GUI difftool:
$ git pr show -t 27919

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/27919.diff

[bridgekeeper]

👋 Welcome back jvernee! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@JornVernee The following labels will be automatically applied to this pull request:

• core-libs
• hotspot
• serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[JornVernee]

/label remove hotspot
/label remove serviceability

[openjdk]

@JornVernee
The hotspot label was successfully removed.

[openjdk]

@JornVernee
The serviceability label was successfully removed.

[openjdk]

@JornVernee hotspot has been added to this pull request based on files touched in new commit(s).