feat(discoveryplugin): look up plugin callback credentials in database #1377
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test image available: |
andrewazores
force-pushed
the
agent-connection
branch
from
8175dc3 to
fe6c4c6
Compare
|
Test image available: |
maxcao13
reviewed
Feb 23, 2023
There was a problem hiding this comment.
Just a question:
Feb 23, 2023 7:18:19 PM org.slf4j.impl.JDK14LoggerAdapter fillCallerData
INFO: 127.0.0.1 - - [Thu, 23 Feb 2023 19:18:19 GMT] 182ms "POST /api/v2.2/discovery HTTP/1.1" 201 813 bytes "-" "Apache-HttpClient/4.5.13 (Java/17.0.5)"
Feb 23, 2023 7:18:19 PM io.cryostat.core.log.Logger info
INFO: GET http://agent:i16pw781fehyuify@localhost:9988/ status 401: Unauthorized
The quarkus-agent-plugin-2 sends a post request to the discovery endpoint here and is met with a 201 response along with the userinfo within the uri. Then Cryostat responds with a GET on the root endpoint of the agent but is met with a 401 status.
Later the plugin tries to register again:
Feb 23, 2023 7:18:19 PM org.slf4j.impl.JDK14LoggerAdapter fillCallerData
INFO: 127.0.0.1 - - [Thu, 23 Feb 2023 19:18:19 GMT] 100ms "POST /api/v2.2/discovery HTTP/1.1" 201 813 bytes "-" "Apache-HttpClient/4.5.13 (Java/17.0.5)"
Feb 23, 2023 7:18:19 PM io.cryostat.core.log.Logger info
INFO: Observing new target: io.cryostat.platform.ServiceRef@3490e520[alias=quarkus-test-agent,annotations=io.cryostat.platform.ServiceRef$Annotations@21711384[cryostat={HOST=cryostat, PORT=9988, JAVA_MAIN=/deployments/quarkus-run.jar, PID=1, START_TIME=1677179882, REALM=quarkus-test-agent},platform={}],jvmId=sAdr8LLOIRdXa8Wp6pQjrZSVLC46oG-x4D4mYlVpJWY=,labels={},serviceUri=http://agent:i16pw781fehyuify@localhost:9988/]
This time, cryostat seems to successfully register the plugin.
Then sometimes we are met with another 401 unauthorized callback somewhere later even though we have registered successfully before
Feb 23, 2023 7:23:16 PM io.cryostat.core.log.Logger info
INFO: POST http://agent:i16pw781fehyuify@localhost:9988/ status 401: Unauthorized
Hibernate:
select
plugininfo0_.id as id1_0_0_,
plugininfo0_.callback as callback2_0_0_,
plugininfo0_.realm as realm3_0_0_,
plugininfo0_.subtree as subtree4_0_0_
from
PluginInfo plugininfo0_
where
plugininfo0_.id=?
Hibernate:
/* delete io.cryostat.discovery.PluginInfo */ delete
from
PluginInfo
where
id=?
Feb 23, 2023 7:23:16 PM io.cryostat.core.log.Logger info
INFO: Stale discovery service http://agent:i16pw781fehyuify@localhost:9988/ removed
For future reference, what is happening here?
|
Weird. I'll circle back around to these PRs and try to reproduce that. |
|
Oh, I wonder if this is something going on with the callback POST vs GET. The callback ping on registration is a GET and on heartbeat check it's a POST I think, or maybe the other way around. Perhaps the plugin is only responding successfully to one or the other. |
|
Hmm... I don't think I'm seeing that behaviour. I diff --git a/smoketest.sh b/smoketest.sh
index 081f2a70..47343d8a 100755
--- a/smoketest.sh
+++ b/smoketest.sh
@@ -89,29 +89,29 @@ runPostgres() {
}
runDemoApps() {
- podman run \
- --name vertx-fib-demo-1 \
- --env HTTP_PORT=8081 \
- --env JMX_PORT=9093 \
- --pod cryostat-pod \
- --rm -d quay.io/andrewazores/vertx-fib-demo:0.9.1
+ # podman run \
+ # --name vertx-fib-demo-1 \
+ # --env HTTP_PORT=8081 \
+ # --env JMX_PORT=9093 \
+ # --pod cryostat-pod \
+ # --rm -d quay.io/andrewazores/vertx-fib-demo:0.9.1
- podman run \
- --name vertx-fib-demo-2 \
- --env HTTP_PORT=8082 \
- --env JMX_PORT=9094 \
- --env USE_AUTH=true \
- --pod cryostat-pod \
- --rm -d quay.io/andrewazores/vertx-fib-demo:0.9.1
+ # podman run \
+ # --name vertx-fib-demo-2 \
+ # --env HTTP_PORT=8082 \
+ # --env JMX_PORT=9094 \
+ # --env USE_AUTH=true \
+ # --pod cryostat-pod \
+ # --rm -d quay.io/andrewazores/vertx-fib-demo:0.9.1
- podman run \
- --name vertx-fib-demo-3 \
- --env HTTP_PORT=8083 \
- --env JMX_PORT=9095 \
- --env USE_SSL=true \
- --env USE_AUTH=true \
- --pod cryostat-pod \
- --rm -d quay.io/andrewazores/vertx-fib-demo:0.9.1
+ # podman run \
+ # --name vertx-fib-demo-3 \
+ # --env HTTP_PORT=8083 \
+ # --env JMX_PORT=9095 \
+ # --env USE_SSL=true \
+ # --env USE_AUTH=true \
+ # --pod cryostat-pod \
+ # --rm -d quay.io/andrewazores/vertx-fib-demo:0.9.1
local webPort;
if [ -z "$CRYOSTAT_WEB_PORT" ]; then
@@ -127,15 +127,15 @@ runDemoApps() {
# this config is broken on purpose (missing required env vars) to test the agent's behaviour
# when not properly set up
- podman run \
- --name quarkus-test-agent-0 \
- --pod cryostat-pod \
- --env JAVA_OPTS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -javaagent:/deployments/app/cryostat-agent.jar" \
- --env QUARKUS_HTTP_PORT=10009 \
- --env ORG_ACME_CRYOSTATSERVICE_ENABLED="false" \
- --env CRYOSTAT_AGENT_WEBCLIENT_SSL_TRUST_ALL="true" \
- --env CRYOSTAT_AGENT_WEBCLIENT_SSL_VERIFY_HOSTNAME="false" \
- --rm -d quay.io/andrewazores/quarkus-test:latest
+ # podman run \
+ # --name quarkus-test-agent-0 \
+ # --pod cryostat-pod \
+ # --env JAVA_OPTS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -javaagent:/deployments/app/cryostat-agent.jar" \
+ # --env QUARKUS_HTTP_PORT=10009 \
+ # --env ORG_ACME_CRYOSTATSERVICE_ENABLED="false" \
+ # --env CRYOSTAT_AGENT_WEBCLIENT_SSL_TRUST_ALL="true" \
+ # --env CRYOSTAT_AGENT_WEBCLIENT_SSL_VERIFY_HOSTNAME="false" \
+ # --rm -d quay.io/andrewazores/quarkus-test:latest
podman run \
--name quarkus-test-agent-1 \
@@ -154,24 +154,25 @@ runDemoApps() {
--env CRYOSTAT_AGENT_AUTHORIZATION="Basic $(echo user:pass | base64)" \
--env CRYOSTAT_AGENT_HARVESTER_PERIOD_MS=60000 \
--env CRYOSTAT_AGENT_HARVESTER_MAX_FILES=10 \
+ --env CRYOSTAT_AGENT_REGISTRATION_RETRY_MS=30000 \
--rm -d quay.io/andrewazores/quarkus-test:latest
- podman run \
- --name quarkus-test-agent-2 \
- --pod cryostat-pod \
- --env JAVA_OPTS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -javaagent:/deployments/app/cryostat-agent.jar" \
- --env QUARKUS_HTTP_PORT=10011 \
- --env ORG_ACME_CRYOSTATSERVICE_ENABLED="false" \
- --env CRYOSTAT_AGENT_APP_NAME="quarkus-test-agent" \
- --env CRYOSTAT_AGENT_WEBCLIENT_SSL_TRUST_ALL="true" \
- --env CRYOSTAT_AGENT_WEBCLIENT_SSL_VERIFY_HOSTNAME="false" \
- --env CRYOSTAT_AGENT_WEBSERVER_HOST="localhost" \
- --env CRYOSTAT_AGENT_WEBSERVER_PORT="9988" \
- --env CRYOSTAT_AGENT_CALLBACK="http://localhost:9988/" \
- --env CRYOSTAT_AGENT_BASEURI="${protocol}://localhost:${webPort}/" \
- --env CRYOSTAT_AGENT_TRUST_ALL="true" \
- --env CRYOSTAT_AGENT_AUTHORIZATION="Basic $(echo user:pass | base64)" \
- --rm -d quay.io/andrewazores/quarkus-test:latest
+ # podman run \
+ # --name quarkus-test-agent-2 \
+ # --pod cryostat-pod \
+ # --env JAVA_OPTS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -javaagent:/deployments/app/cryostat-agent.jar" \
+ # --env QUARKUS_HTTP_PORT=10011 \
+ # --env ORG_ACME_CRYOSTATSERVICE_ENABLED="false" \
+ # --env CRYOSTAT_AGENT_APP_NAME="quarkus-test-agent" \
+ # --env CRYOSTAT_AGENT_WEBCLIENT_SSL_TRUST_ALL="true" \
+ # --env CRYOSTAT_AGENT_WEBCLIENT_SSL_VERIFY_HOSTNAME="false" \
+ # --env CRYOSTAT_AGENT_WEBSERVER_HOST="localhost" \
+ # --env CRYOSTAT_AGENT_WEBSERVER_PORT="9988" \
+ # --env CRYOSTAT_AGENT_CALLBACK="http://localhost:9988/" \
+ # --env CRYOSTAT_AGENT_BASEURI="${protocol}://localhost:${webPort}/" \
+ # --env CRYOSTAT_AGENT_TRUST_ALL="true" \
+ # --env CRYOSTAT_AGENT_AUTHORIZATION="Basic $(echo user:pass | base64)" \
+ # --rm -d quay.io/andrewazores/quarkus-test:latest
# copy a jboss-client.jar into /clientlib first
# manual entry URL: service:jmx:remote+http://localhost:9990so that there is only the one sample app instance (to keep the logs as clean as possible), and to delay the registration timing so that all of Cryostat's startup stuff is completed including discovering itself over JDP etc. I don't think any of that should affect how the plugin registration/publication happens. I'll keep tinkering with it but at the moment I'm not sure what could be causing the behaviour you observed. |
|
Here are some server-side logs: and the agent-side logs from |
|
Hm... let me try again. |
andrewazores
force-pushed
the
agent-connection
branch
from
fe6c4c6 to
05597db
Compare
|
Test image available: |
|
Oh, I think I was using the wrong version of cryostat and checked out the pr without packing, but it's interesting how the plugin is registered for a bit without giving correct credentials, before deregistering. When I used the correct version, it worked as expected. Good work, sorry about that! |
maxcao13
previously approved these changes
Feb 23, 2023
Yea hold on, this doesn't sound good. Could you retrace your steps and figure out a reproducer? I need to be sure that this isn't an attack vector getting around the authentication mechanism. |
I think you can checkout current |
|
I run with and I get these logs: Feb 23, 2023 11:35:23 PM io.cryostat.core.log.Logger info
INFO: GET http://agent:3lr7orzimq1hmt5m@localhost:9977/ status 401: Unauthorizedlater... Feb 23, 2023 11:35:23 PM org.slf4j.impl.JDK14LoggerAdapter fillCallerData
INFO: 127.0.0.1 - - [Thu, 23 Feb 2023 23:35:23 GMT] 180ms "POST /api/v2.2/discovery HTTP/1.1" 201 807 bytes "-" "Apache-HttpClient/4.5.13 (Java/17.0.5)"
Feb 23, 2023 11:35:23 PM io.cryostat.core.log.Logger info
INFO: Observing new target: io.cryostat.platform.ServiceRef@610563ab[alias=quarkus-test-agent,annotations=io.cryostat.platform.ServiceRef$Annotations@12dbf1e9[cryostat={HOST=cryostat, PORT=9977, JAVA_MAIN=/deployments/quarkus-run.jar, PID=1, START_TIME=1677195291, REALM=quarkus-test-agent},platform={}],jvmId=0z5t3_RDHt06hrHreke2qwKoFeOLsAMS1e3NZe24Roo=,labels={},serviceUri=http://agent:3lr7orzimq1hmt5m@localhost:9977/]and then even later... INFO: 127.0.0.1 - - [Thu, 23 Feb 2023 23:35:23 GMT] 72ms "POST /api/v2.2/discovery/ffba92da-ff02-4009-900b-7e946f91d872?token=eyJjdHkiOiJKV1QiLCJlbmMiOiJBMjU2R0NNIiwiYWxnIjoiZGlyIn0..Jxy7Jwq12G0z5HGA.AtQy-y6ky6nPAKeyz_wOPtxiF0O6oWus3HCptVlRDhDd6dOLSD0FNO_vft3Ty2lP4qfBigJmGP5XOE3xxo8dFYHgl61H1AS4s2VPfF66dWTaqZzl_ffR6jPsJr_yS9Cf57phBlhiL0hms9KMSiGWjAcaty8ZCKuf3W5SsGxbG14Y7DKq8fyhLk1HDX9as9eY3XMSbAxlGmXfU_aNQ3mbs4OwBfw8-kOM1SVKNSMPhdqqig9csCpXvHv84HPErlTYUtYPC5swhH2taXAtVr7QhOI2LAVyPyHSo5_FpncxpVgq3Jvc3B0192pPssScP4sZRu6rUKZ9ixnNrpYPqobxkOB7xPlJKfntm_m2V8FojC4HR8ujqoIPbLxWPlLdNl_ckO7AFNTco7ZVAdJYuSY_WkcfVTwKP9PI65R29MU3z8c5j9N_lOTV5N4DAqRJ6l34AvXKaJchfE34vOxZQBAJlGfg5o3vqahiF_Mf-5TH5n-CdBXyTM_5pznbb_fifAYOBUX7N0Pg_d-DcyY6JnUIAxCfHWueeEYBKXq8KZdOtyXhq4hP9oVenZ18xSWzUW1dPqyS.9ThG7gMtF1Y-rMim0jhzLw HTTP/1.1" 200 65 bytes "-" "Apache-HttpClient/4.5.13 (Java/17.0.5)"
Feb 23, 2023 11:35:23 PM io.cryostat.core.log.Logger info
INFO: Outgoing WS message: {"meta":{"category":"TargetJvmDiscovery","type":{"type":"application","subType":"json"},"serverTime":1677195323},"message":{"event":{"serviceRef":{"jvmId":"0z5t3_RDHt06hrHreke2qwKoFeOLsAMS1e3NZe24Roo=","connectUrl":"http://agent:3lr7orzimq1hmt5m@localhost:9977/","alias":"quarkus-test-agent","labels":{},"annotations":{"platform":{},"cryostat":{"HOST":"cryostat","PORT":"9977","JAVA_MAIN":"/deployments/quarkus-run.jar","PID":"1","START_TIME":"1677195291","REALM":"quarkus-test-agent"}}},"kind":"FOUND"}}} |
|
podman logs: Starting the Java application using /opt/jboss/container/java/run/run-java.sh ...
INFO exec java -Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dcom.sun.management.jmxremote.port=9097 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -javaagent:/deployments/app/cryostat-agent.jar -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -cp "." -jar /deployments/quarkus-run.jar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/deployments/app/cryostat-agent.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/deployments/lib/main/org.jboss.slf4j.slf4j-jboss-logmanager-1.1.0.Final.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Slf4jLoggerFactory]
__ ____ __ _____ ___ __ ____ ______
--/ __ \/ / / / _ | / _ \/ //_/ / / / __/
-/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
2023-02-23 23:34:52,011 INFO [io.cry.age.Agent] (cryostat-agent-main) Cryostat Agent starting...
2023-02-23 23:34:52,722 INFO [io.cry.cor.net.JFRConnectionToolkit] (cryostat-agent-main) Computed self JVM ID: 0z5t3_RDHt06hrHreke2qwKoFeOLsAMS1e3NZe24Roo=
2023-02-23 23:34:52,727 INFO [io.cry.age.CryostatClient] (cryostat-agent-main) Using Cryostat baseuri http://localhost:8181/
2023-02-23 23:34:52,753 INFO [io.cry.age.Registration] (cryostat-agent-main) io.cryostat.agent.Registration started
2023-02-23 23:34:52,753 INFO [io.cry.age.Agent] (cryostat-agent-main) Startup complete
2023-02-23 23:34:52,829 INFO [io.cry.age.CryostatClient] (cryostat-agent-worker-0) POST http://localhost:8181/api/v2.2/discovery HTTP/1.1
2023-02-23 23:34:52,904 SEVERE [io.cry.age.Registration] (cryostat-agent-worker-0) Registration failure: java.util.concurrent.ExecutionException: io.cryostat.agent.RegistrationException: java.util.concurrent.CompletionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8181 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at io.cryostat.agent.Registration.tryRegister(Registration.java:137)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.cryostat.agent.RegistrationException: java.util.concurrent.CompletionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8181 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused
at io.cryostat.agent.Registration.lambda$1(Registration.java:130)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
... 6 more
Caused by: java.util.concurrent.CompletionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8181 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused
at io.cryostat.agent.CryostatClient.executeQuiet(CryostatClient.java:273)
at io.cryostat.agent.CryostatClient.lambda$11(CryostatClient.java:265)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
... 6 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8181 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at io.cryostat.agent.CryostatClient.executeQuiet(CryostatClient.java:271)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
... 19 more
2023-02-23 23:34:52,905 INFO [io.cry.age.Registration] (cryostat-agent-worker-0) Registration retry period: PT30S
2023-02-23 23:34:53,480 INFO [io.quarkus] (main) quarkus-test 1.0.0-SNAPSHOT on JVM (powered by Quarkus 2.7.2.Final) started in 1.182s. Listening on: http://0.0.0.0:10010
2023-02-23 23:34:53,480 INFO [io.quarkus] (main) Profile prod activated.
2023-02-23 23:34:53,480 INFO [io.quarkus] (main) Installed features: [cdi, rest-client, rest-client-jackson, resteasy, smallrye-context-propagation, vertx]
2023-02-23 23:35:22,906 INFO [io.cry.age.CryostatClient] (cryostat-agent-worker-0) POST http://localhost:8181/api/v2.2/discovery HTTP/1.1
2023-02-23 23:35:23,059 INFO [io.cry.age.WebServer] (cryostat-agent-worker-2) GET /
2023-02-23 23:35:23,075 INFO [io.cry.age.WebServer] (cryostat-agent-worker-2) GET / : 401
2023-02-23 23:35:23,177 INFO [io.cry.age.CryostatClient] (cryostat-agent-worker-1) POST http://localhost:8181/api/v2.2/discovery : 201
2023-02-23 23:35:23,205 INFO [io.cry.age.Registration] (cryostat-agent-worker-1) Registered as ffba92da-ff02-4009-900b-7e946f91d872
2023-02-23 23:35:23,207 INFO [io.cry.age.Harvester] (cryostat-agent-worker-1) JFR Harvester starting
2023-02-23 23:35:23,415 INFO [io.cry.age.Harvester] (cryostat-agent-worker-1) JFR Harvester started using template "default" with period PT16H40M
2023-02-23 23:35:23,425 INFO [io.cry.age.Registration] (cryostat-agent-worker-1) publishing self as http://agent:3lr7orzimq1hmt5m@localhost:9977/
2023-02-23 23:35:23,459 INFO [io.cry.age.CryostatClient] (cryostat-agent-worker-1) POST http://localhost:8181/api/v2.2/discovery/ffba92da-ff02-4009-900b-7e946f91d872?token=eyJjdHkiOiJKV1QiLCJlbmMiOiJBMjU2R0NNIiwiYWxnIjoiZGlyIn0..Jxy7Jwq12G0z5HGA.AtQy-y6ky6nPAKeyz_wOPtxiF0O6oWus3HCptVlRDhDd6dOLSD0FNO_vft3Ty2lP4qfBigJmGP5XOE3xxo8dFYHgl61H1AS4s2VPfF66dWTaqZzl_ffR6jPsJr_yS9Cf57phBlhiL0hms9KMSiGWjAcaty8ZCKuf3W5SsGxbG14Y7DKq8fyhLk1HDX9as9eY3XMSbAxlGmXfU_aNQ3mbs4OwBfw8-kOM1SVKNSMPhdqqig9csCpXvHv84HPErlTYUtYPC5swhH2taXAtVr7QhOI2LAVyPyHSo5_FpncxpVgq3Jvc3B0192pPssScP4sZRu6rUKZ9ixnNrpYPqobxkOB7xPlJKfntm_m2V8FojC4HR8ujqoIPbLxWPlLdNl_ckO7AFNTco7ZVAdJYuSY_WkcfVTwKP9PI65R29MU3z8c5j9N_lOTV5N4DAqRJ6l34AvXKaJchfE34vOxZQBAJlGfg5o3vqahiF_Mf-5TH5n-CdBXyTM_5pznbb_fifAYOBUX7N0Pg_d-DcyY6JnUIAxCfHWueeEYBKXq8KZdOtyXhq4hP9oVenZ18xSWzUW1dPqyS.9ThG7gMtF1Y-rMim0jhzLw HTTP/1.1
2023-02-23 23:35:23,602 INFO [io.cry.age.Harvester] (cryostat-agent-worker-2) cryostat-agent(1) RUNNING
2023-02-23 23:35:23,684 INFO [io.cry.age.CryostatClient] (cryostat-agent-worker-2) POST http://localhost:8181/api/v2.2/discovery/ffba92da-ff02-4009-900b-7e946f91d872?token=eyJjdHkiOiJKV1QiLCJlbmMiOiJBMjU2R0NNIiwiYWxnIjoiZGlyIn0..Jxy7Jwq12G0z5HGA.AtQy-y6ky6nPAKeyz_wOPtxiF0O6oWus3HCptVlRDhDd6dOLSD0FNO_vft3Ty2lP4qfBigJmGP5XOE3xxo8dFYHgl61H1AS4s2VPfF66dWTaqZzl_ffR6jPsJr_yS9Cf57phBlhiL0hms9KMSiGWjAcaty8ZCKuf3W5SsGxbG14Y7DKq8fyhLk1HDX9as9eY3XMSbAxlGmXfU_aNQ3mbs4OwBfw8-kOM1SVKNSMPhdqqig9csCpXvHv84HPErlTYUtYPC5swhH2taXAtVr7QhOI2LAVyPyHSo5_FpncxpVgq3Jvc3B0192pPssScP4sZRu6rUKZ9ixnNrpYPqobxkOB7xPlJKfntm_m2V8FojC4HR8ujqoIPbLxWPlLdNl_ckO7AFNTco7ZVAdJYuSY_WkcfVTwKP9PI65R29MU3z8c5j9N_lOTV5N4DAqRJ6l34AvXKaJchfE34vOxZQBAJlGfg5o3vqahiF_Mf-5TH5n-CdBXyTM_5pznbb_fifAYOBUX7N0Pg_d-DcyY6JnUIAxCfHWueeEYBKXq8KZdOtyXhq4hP9oVenZ18xSWzUW1dPqyS.9ThG7gMtF1Y-rMim0jhzLw : 200
2023-02-23 23:35:23,685 INFO [io.cry.age.Registration] (cryostat-agent-worker-2) Publish success |
|
Oh and now I get this: Feb 23, 2023 11:40:06 PM io.cryostat.core.log.Logger info
INFO: POST http://agent:3lr7orzimq1hmt5m@localhost:9977/ status 401: Unauthorized
Hibernate:
select
plugininfo0_.id as id1_0_0_,
plugininfo0_.callback as callback2_0_0_,
plugininfo0_.realm as realm3_0_0_,
plugininfo0_.subtree as subtree4_0_0_
from
PluginInfo plugininfo0_
where
plugininfo0_.id=?
Hibernate:
/* delete io.cryostat.discovery.PluginInfo */ delete
from
PluginInfo
where
id=?
Feb 23, 2023 11:40:06 PM io.cryostat.core.log.Logger info
INFO: Stale discovery service http://agent:3lr7orzimq1hmt5m@localhost:9977/ removed
Feb 23, 2023 11:40:06 PM io.cryostat.core.log.Logger info
INFO: Outgoing WS message: {"meta":{"category":"TargetJvmDiscovery","type":{"type":"application","subType":"json"},"serverTime":1677195606},"message":{"event":{"serviceRef":{"jvmId":"0z5t3_RDHt06hrHreke2qwKoFeOLsAMS1e3NZe24Roo=","connectUrl":"http://agent:3lr7orzimq1hmt5m@localhost:9977/","alias":"quarkus-test-agent","labels":{},"annotations":{"platform":{},"cryostat":{"HOST":"cryostat","PORT":"9977","JAVA_MAIN":"/deployments/quarkus-run.jar","PID":"1","START_TIME":"1677195291","REALM":"quarkus-test-agent"}}},"kind":"LOST"}}} |
|
Got it. The backend has a bug causing it to consider any response from the plugin as a successful callback ping, not only |
|
The bug should be fixed now. The agent generates Basic credentials at startup and clears the password from memory after it has finished registering and publishing itself. It then regenerates fresh credentials whenever it is pinged and prompted to refresh its registration, and again clears that password from memory once the registration refresh is completed. This does mean unfortunately that the HTTP connection URL that it publishes for itself is now unstable, since the URL includes the Basic credentials, so each time Cryostat prompts Agents to refresh their registration they end up being picked up as lost and re-found targets. I need to think some more about how to resolve this. Maybe the correct approach is actually that the agent should be talking to Cryostat and registering an actual Credential resource for itself and updating that over time, rather than embedding the credentials in the connection URL. |
|
Test image available: |
|
New approach. The plugin generates its random credentials in-memory, then As a nice side effect, this authenticated two-way handshake protocol also means that the old timing issue where stale plugin definitions could conflict with new plugin instances when both are torn down and recreated in close timing to each other, should no longer happen. If Cryostat has a stale definition for a plugin then when it attempts to do a startup ping to verify which plugins are still present, the ping should fail because the definition will point to stored credentials that are no longer valid for the new plugin instance, so Cryostat will prune the stale plugin definition. The plugin should then eventually try to re-register and succeed now that there is no stale duplicate definition in the way. |
|
Test image available: |
andrewazores
changed the title
|
Test image available: |
|
Seems like the frontend doesnt recognize the credential to match any target for some reason... maybe just a visual bug. |
|
I noticed that too. I don't know if it's a visual bug on the frontend or actually a bug in the backend's evaluation of the expression against the list of targets. I suspect it's on the backend. I'll take a further look into it - it should obviously have exactly one match (or zero, if the agent has disappeared without properly deregistering itself and cleaning up its credential) for each such credential defined in this way. |
|
On that note though, the frontend needs to be updated to remove specific references to JMX, since these stored credentials will also be used for HTTP-based Agent connections. |
maxcao13
previously approved these changes
Feb 28, 2023
andrewazores
force-pushed
the
agent-connection
branch
from
9206eec to
eb76e56
Compare
|
Test image available: |
andrewazores
force-pushed
the
agent-connection
branch
from
eb76e56 to
9697a2a
Compare
|
@maxcao13 very very minor change to |
|
Test image available: |
andrewazores
force-pushed
the
agent-connection
branch
from
9697a2a to
9459a3b
Compare
|
Test image available: |
andrewazores
force-pushed
the
agent-connection
branch
from
9459a3b to
ff660b5
Compare
|
Test image available: |
andrewazores
force-pushed
the
agent-connection
branch
from
ff660b5 to
af605be
Compare
|
Test image available: |
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Welcome to Cryostat! 👋
Before contributing, make sure you have:
mainbranch[chore, ci, docs, feat, fix, test]git commit --amend --signoffFixes: #1376
Description of the change:
If a discovery plugin (ex. the Agent) has registered using a callback URL that includes userinfo (ex.http://user:pass@example.com),this same userinfo should be included by Cryostat both in the request URL as well as copied to
Authorizationheader when Cryostat communicates with that plugin, for example when pinging the plugin to ensure it is still live and reachable.When a discovery plugin wants to register with Cryostat, it will first define a stored credential for itself. Then it will register itself using a special value for the userinfo of the plugin callback URI which references that stored credential. Cryostat looks up that credential and includes it for authentication on the plugin callback requests it makes in the future, both to validate the plugin during registration time as well as periodically later for heartbeat checks.
Motivation for the change:
Plugins should require an authentication challenge on requests since these requests can trigger the plugin to refresh registration with the Cryostat server. Without an authentication challenge, any client may ping the plugin instance and cause it to resend registration to Cryostat. This causes undue increased load on the Cryostat server for no productive purpose.
How to manually test:
quarkus-testcontainer imagesmoketest.shusing this PR image and the updatedquarkus-testwith updated-agent