Running plugin with "gradle run --debug-jvm" seems to be broken after version 7.4.x

[alexklibisz]

Elasticsearch version: Pertains to 7.4.x and up

Plugins installed: []

JVM version (java -version): Pertains to 12 and up

OS version (uname -a if on a Unix-like system): Ubuntu 19.04, 20.04

Description of the problem including expected versus actual behavior:

I'm developing a plugin that started with ES version 7.4.x and I want to update it to use the latest 7.7.x. I found that after bumping from 7.4.2 to 7.5.0 I'm no longer able to debug the plugin using ./gradlew run --debug-jvm. The regular ./gradlew run still works like normal.

Steps to reproduce:

I was able to replicate this with an open source plugin maintained by an elastic employee: https://github.com/spinscale/elasticsearch-ingest-langdetect, which I'll use for my example below.

If you checkout the commit that updates to 7.4.2, you'll find you can run ./gradlew run --debug-jvm without issue.

$ git checkout 3b533ee8bd50c72b242629e2606f73edeb1ddbf6
$ ./gradlew run --debug-jvm

This eventually prints "Running elasticsearch in debug mode, suspending until connected on port 8000" and you can connect from an IDE. So it works.

Then if you checkout the next commit that updates to 7.5.0, and run with --stacktrace, you get an error.

$ git checkout 28c4c8851cb0d00945286f8c02327ebd00995b09
$ ./gradlew --stacktrace run --debug-jvm

This eventually prints a large stacktrace, summarized below:

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':run'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.accept(ExecuteActionsTaskExecuter.java:166)
        ...
        ...
Caused by: org.gradle.process.internal.ExecException: Process 'command './bin/elasticsearch-plugin'' finished with non-zero exit value 2
        at org.gradle.process.internal.DefaultExecHandle$ExecResultImpl.assertNormalExitValue(DefaultExecHandle.java:409)
        ...

I also tried this with the subsequent 7.6.x and 7.7.x updates and it produces the same errors.
Note that both 7.4.2 and 7.5.0 are running on JDK 12. So it doesn't seem to be a JDK bug.
I'm using OpenJDK 12.0.2 in the above example.

Provide logs (if relevant):

Here is the full log: https://pastebin.com/6RctUHpL

[alexklibisz]

I suspect this part of the code might be relevant:

elasticsearch/buildSrc/src/main/java/org/elasticsearch/gradle/testclusters/ElasticsearchNode.java

Line 461 in bd64da0

plugins.forEach(plugin -> runElasticsearchBinScript("elasticsearch-plugin", "install", "--batch", plugin.toString()));

Is there any way to turn on more verbose logging to show the exact command being invoked?

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Build)

[breskeby]

Thank you @alexklibisz for reporting. I'll look into this and come back to you in the next couple of days.

[alexklibisz]

Thank you @alexklibisz for reporting. I'll look into this and come back to you in the next couple of days.

Thanks! 😄

If you have any pointers about where to look, I'm happy to do so as well. I'm not really familiar with the buildSrc part of the repo but could probably find my way around given a general direction.

[alexklibisz]

Note there's an easier repro. In the elasticsearch repo, running

./gradlew --stacktrace :example-plugins:rescore:run --debug-jvm

fails with the same error.

[alexklibisz]

Good news:

The issue seems to be fixed by reverting the changes in this PR: #48188

I'm not sure what that intention of that PR was, but if you revert its changes, the gradle task arrives at the log line Running elasticsearch in debug mode, node{:example-plugins:rescore:integTest-0} suspending until connected on debugPort 8000.

Also if you set the debug port to 5000 or 5006 it works.. I guess there's something else running on 5005?

I have no clue why it says that it's failing on the ./bin/elasticsearch-plugin script. It seems like this is the last place it gets before exceptions start:

elasticsearch/buildSrc/src/minimumRuntime/java/org/elasticsearch/gradle/LoggedExec.java

Line 123 in 007ec1f

try {

If I set a break point there, I can manually run the ./bin/elasticsearch-plugin script and it exits with 0.

Hopefully that helps a bit.

[breskeby]

@alexklibisz The motivation behind that change is described here: #48188 (comment) So it fails to run as it expects the debug server to be already active. We probably can add a warning when this --debug-jvm flag is used that a debugging server process is expected the app can connect too. WDTY? //cc @alpar-t has

[alexklibisz]

@breskeby Ahh. I see. Yes re-starting the debug server several times used to be annoying.

To make sure I understand, the correct debugging workflow is:

1. Run the Debug Elasticsearch remote configuration in IntelliJ (or start some other debugger).
2. Run the gradle command with --debug-jvm

I tried that with the rescore plugin and it starts up without issues.

And yes it would be very helpful if you could add a warning, especially since the problem seems to manifest via the ./bin/elasticsearch-plugin script.