Application keeps on connecting to Key vault server in a loop if the app has Actuator in OpenJ9 JDK11

[VikramNisarga]

Environment

• Spring boot starter:

  • key vault spring boot starter

• OS Type: Linux

  • 11

Summary

When spring boot actuator is added to the app, the app keeps on creating new connections to keyvault server in a loop even after the app is started.

Reproduce steps

configure keyvault in application.yml and add actuator to pom.xml

I've created a sample app to reproduce the bug

Expected Results

It should connect once at the start of the app

Actual Results

app log is full of c.m.aad.adal4j.AuthenticationAuthority : [Correlation ID: 57ca7499-728d-4827-9099-febdbf7b77fa] Instance discovery was successful it never ends.

if i remove actuator from pom.xml the I don't get those logs. It'll work as expected.

2019-03-21 13:17:15.743  INFO 20452 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2019-03-21 13:17:15.778  INFO 20452 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2019-03-21 13:17:15.778  INFO 20452 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.16]
2019-03-21 13:17:15.793  INFO 20452 --- [           main] o.a.catalina.core.AprLifecycleListener   : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/lib/jvm/jdk-11.0.2+9_openj9-0.12.1/lib/compressedrefs:/usr/lib/jvm/jdk-11.0.2+9_openj9-0.12.1/lib:/usr/lib64:/usr/lib]
2019-03-21 13:17:15.890  INFO 20452 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2019-03-21 13:17:15.890  INFO 20452 --- [           main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 2056 ms
2019-03-21 13:17:16.829  INFO 20452 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2019-03-21 13:17:17.528  INFO 20452 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 2 endpoint(s) beneath base path '/actuator'
2019-03-21 13:17:17.664  INFO 20452 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2019-03-21 13:17:17.671  INFO 20452 --- [           main] com.example.demo.DemoApplication         : Started DemoApplication in 13.214 seconds (JVM running for 14.68)
2019-03-21 13:17:21.468  INFO 20452 --- [pool-3-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 57ca7499-728d-4827-9099-febdbf7b77fa] Instance discovery was successful
2019-03-21 13:17:22.190  INFO 20452 --- [pool-4-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 7db3a2f7-aa72-4769-bf78-79851d73118d] Instance discovery was successful
2019-03-21 13:17:22.509  INFO 20452 --- [pool-5-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 882edf73-4075-4f3b-9b3a-4a25681974ce] Instance discovery was successful
2019-03-21 13:17:22.931  INFO 20452 --- [pool-6-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: c79c0147-9eaf-4c22-bf52-38b88ec77d5f] Instance discovery was successful
2019-03-21 13:17:23.467  INFO 20452 --- [pool-7-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 856bba40-ba0a-4aaf-a3bb-dc1ea5a329d9] Instance discovery was successful
2019-03-21 13:17:23.856  INFO 20452 --- [pool-8-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 9ece982a-647a-48ed-a548-f1c562bc3421] Instance discovery was successful
2019-03-21 13:17:24.395  INFO 20452 --- [on(2)-127.0.0.1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2019-03-21 13:17:24.396  INFO 20452 --- [on(2)-127.0.0.1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2019-03-21 13:17:24.409  INFO 20452 --- [on(2)-127.0.0.1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 13 ms
2019-03-21 13:17:29.673  INFO 20452 --- [pool-9-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: aeea333e-ab58-466f-be59-58f156442c7a] Instance discovery was successful
2019-03-21 13:17:30.943  INFO 20452 --- [ool-10-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: d13d88f4-507c-4261-9466-303172c7e0c9] Instance discovery was successful
2019-03-21 13:17:31.663  INFO 20452 --- [ool-11-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 8083bca6-799b-4641-b243-da1593459e57] Instance discovery was successful
2019-03-21 13:17:32.162  INFO 20452 --- [ool-12-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 3c2f9c30-b4f6-4fea-b89b-c67e24328618] Instance discovery was successful
2019-03-21 13:17:32.676  INFO 20452 --- [ool-13-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 58f6221b-614b-43d1-8d21-054ee5f980c0] Instance discovery was successful
2019-03-21 13:17:33.075  INFO 20452 --- [ool-14-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 6e7fc7a5-5777-477c-a394-383f685a0630] Instance discovery was successful
2019-03-21 13:17:38.692  INFO 20452 --- [ool-15-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 4bf7269a-40d2-473b-af7e-f37b97981ef5] Instance discovery was successful
2019-03-21 13:17:39.956  INFO 20452 --- [ool-16-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 06170441-13b1-44da-8d9d-b92cd37aa118] Instance discovery was successful
2019-03-21 13:17:40.395  INFO 20452 --- [ool-17-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: d84f1e55-6a66-44a2-b008-2dbf18c61248] Instance discovery was successful
2019-03-21 13:17:40.764  INFO 20452 --- [ool-18-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: c4577ad5-4217-46b3-b21d-48ae6eebec97] Instance discovery was successful
2019-03-21 13:17:41.277  INFO 20452 --- [ool-19-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: a4bbe466-24aa-4cc9-af21-9451d18b4a6f] Instance discovery was successful
2019-03-21 13:17:41.843  INFO 20452 --- [ool-20-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 6a502b17-eaa6-4b14-9f20-0b4151629322] Instance discovery was successful
2019-03-21 13:17:47.464  INFO 20452 --- [ool-21-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: cc3c4687-90ee-4bb9-b3ad-189a18dc8504] Instance discovery was successful
2019-03-21 13:17:48.132  INFO 20452 --- [ool-22-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: d43e70d4-5a1b-4f85-85d8-d30b0255e3ea] Instance discovery was successful
2019-03-21 13:17:48.763  INFO 20452 --- [ool-23-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 91d5a365-d2d9-406e-a560-271f76d052e6] Instance discovery was successful
2019-03-21 13:17:49.338  INFO 20452 --- [ool-24-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 7b1f5ab2-4d96-46a9-8b49-edd540a06e25] Instance discovery was successful
2019-03-21 13:17:49.851  INFO 20452 --- [ool-25-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: dd9b7264-acca-4402-8722-41df5037276e] Instance discovery was successful
2019-03-21 13:17:50.569  INFO 20452 --- [ool-26-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 51d906b7-ac4a-40c3-a0bc-de5ab0517bee] Instance discovery was successful
2019-03-21 13:17:56.101  INFO 20452 --- [ool-27-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 4b1be544-8492-493d-b6f0-980587b2dd08] Instance discovery was successful
2019-03-21 13:17:57.104  INFO 20452 --- [ool-28-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 0516b9b2-161a-49e7-8e9f-f4b91cf2a76a] Instance discovery was successful
2019-03-21 13:17:57.669  INFO 20452 --- [ool-29-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 96907638-63c3-4e1c-a33d-b9b1e35b68f5] Instance discovery was successful
2019-03-21 13:17:59.203  INFO 20452 --- [ool-30-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 858f262f-c139-4571-8679-c4a5024370d3] Instance discovery was successful
2019-03-21 13:17:59.821  INFO 20452 --- [ool-31-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: b356caf8-6c7a-40fa-a7c0-e450600e94a6] Instance discovery was successful
2019-03-21 13:18:00.536  INFO 20452 --- [ool-32-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 7bbfc0c5-e034-49d6-9f5d-93e06e531659] Instance discovery was successful
2019-03-21 13:18:06.021  INFO 20452 --- [ool-33-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 9daf5f85-8e4b-45e2-b859-7c7548e3c180] Instance discovery was successful
2019-03-21 13:18:06.760  INFO 20452 --- [ool-34-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 41ef2724-2e0a-47d5-98a6-26fabe0ea63b] Instance discovery was successful
2019-03-21 13:18:07.257  INFO 20452 --- [ool-35-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: ea4872d7-ab9d-42ea-a7bb-ecbbc0a6f936] Instance discovery was successful
2019-03-21 13:18:07.582  INFO 20452 --- [ool-36-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 20fd43e0-9ea1-4121-8fb0-2f7559995003] Instance discovery was successful
2019-03-21 13:18:07.983  INFO 20452 --- [ool-37-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 8f93157e-ede9-47e0-ae15-9133ed8bd524] Instance discovery was successful
2019-03-21 13:18:08.383  INFO 20452 --- [ool-38-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 7db4f104-8227-400a-a425-45cf6682a17e] Instance discovery was successful
2019-03-21 13:18:13.936  INFO 20452 --- [ool-39-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 7f95479c-4f55-49a2-893a-c5b6fb9a2280] Instance discovery was successful
2019-03-21 13:18:14.593  INFO 20452 --- [ool-40-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 740e1f09-a642-4e9c-8830-984586db2e35] Instance discovery was successful
2019-03-21 13:18:14.929  INFO 20452 --- [ool-41-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 0c72ca97-ee07-4b30-a5eb-949d415b2e06] Instance discovery was successful
2019-03-21 13:18:15.213  INFO 20452 --- [ool-42-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: d496e461-e551-4064-9337-9a9f93e7ef9f] Instance discovery was successful
2019-03-21 13:18:15.555  INFO 20452 --- [ool-43-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 5455dbad-6fee-40b9-9dbc-a288eedfe292] Instance discovery was successful
2019-03-21 13:18:15.905  INFO 20452 --- [ool-44-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: e903581f-6624-4489-b189-7e9d7a605864] Instance discovery was successful

[jpnaidu07]

I have Spring cloud dependencies for Spring cloud config server, getting the same issue after removing the actuator. Please help.

[saragluna]

@jpnaidu07 Will look into this. Could you please more info to help reproduce?

[jpnaidu07]

@saragluna thanks for quick response

I have the dependency spring cloud config in my project

org.springframework.cloud
spring-cloud-starter-config

along with actuator

org.springframework.boot
spring-boot-starter-actuator

then im using

com.microsoft.azure
azure-keyvault-secrets-spring-boot-starter
2.1.7
to load the secrets

When i dont have spring cloud config and actuator it is working fine.
Along with those dependencies it seems it is explicitly fetching one by one secret(showing [Correlation ID: e903581f-6624-4489-b189-7e9d7a605864] Instance discovery was successful) taking more than 15 mins in my case and depends on how many secrets are there in keyvault.

[jpnaidu07]

@saragluna any update?

[saragluna]

@jpnaidu07 Sorry for the late response.
I've run the demo project and got the same logs as you encountered. Still digging into it to find the root cause.

[chenrujun]

Hi, @VikramNisarga, @jpnaidu07 , I can not reproduce your problem.
How many keys do you have?

When I try 11 keys, it works well within 4 seconds.
Could you please try my demo ?

Here is the log in my computer:

"C:\Program Files\Java\jdk-11.0.7\bin\java.exe" "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2020.1.1\lib\idea_rt.jar=53041:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2020.1.1\bin" -Dfile.encoding=UTF-8 -classpath C:\Users\rujche\Work\key-vault-demo\demo\target\classes;C:\Users\rujche\.m2\repository\org\springframework\boot\spring-boot-starter-actuator\2.1.3.RELEASE\spring-boot-starter-actuator-2.1.3.RELEASE.jar;C:\Users\rujche\.m2\repository\org\springframework\boot\spring-boot-starter\2.1.3.RELEASE\spring-boot-starter-2.1.3.RELEASE.jar;C:\Users\rujche\.m2\repository\org\springframework\boot\spring-boot-starter-logging\2.1.3.RELEASE\spring-boot-starter-logging-2.1.3.RELEASE.jar;C:\Users\rujche\.m2\repository\ch\qos\logback\logback-classic\1.2.3\logback-classic-1.2.3.jar;C:\Users\rujche\.m2\repository\ch\qos\logback\logback-core\1.2.3\logback-core-1.2.3.jar;C:\Users\rujche\.m2\repository\org\apache\logging\log4j\log4j-to-slf4j\2.11.2\log4j-to-slf4j-2.11.2.jar;C:\Users\rujche\.m2\repository\org\apache\logging\log4j\log4j-api\2.11.2\log4j-api-2.11.2.jar;C:\Users\rujche\.m2\repository\org\slf4j\jul-to-slf4j\1.7.25\jul-to-slf4j-1.7.25.jar;C:\Users\rujche\.m2\repository\javax\annotation\javax.annotation-api\1.3.2\javax.annotation-api-1.3.2.jar;C:\Users\rujche\.m2\repository\org\yaml\snakeyaml\1.23\snakeyaml-1.23.jar;C:\Users\rujche\.m2\repository\org\springframework\boot\spring-boot-actuator-autoconfigure\2.1.3.RELEASE\spring-boot-actuator-autoconfigure-2.1.3.RELEASE.jar;C:\Users\rujche\.m2\repository\org\springframework\boot\spring-boot-actuator\2.1.3.RELEASE\spring-boot-actuator-2.1.3.RELEASE.jar;C:\Users\rujche\.m2\repository\com\fasterxml\jackson\core\jackson-databind\2.9.8\jackson-databind-2.9.8.jar;C:\Users\rujche\.m2\repository\com\fasterxml\jackson\core\jackson-annotations\2.9.0\jackson-annotations-2.9.0.jar;C:\Users\rujche\.m2\repository\com\fasterxml\jackson\core\jackson-core\2.9.8\jackson-core-2.9.8.jar;C:\Users\rujche\.m2\repository\org\springframework\spring-context\5.1.5.RELEASE\spring-context-5.1.5.RELEASE.jar;C:\Users\rujche\.m2\repository\com\fasterxml\jackson\datatype\jackson-datatype-jsr310\2.9.8\jackson-datatype-jsr310-2.9.8.jar;C:\Users\rujche\.m2\repository\io\micrometer\micrometer-core\1.1.3\micrometer-core-1.1.3.jar;C:\Users\rujche\.m2\repository\org\hdrhistogram\HdrHistogram\2.1.9\HdrHistogram-2.1.9.jar;C:\Users\rujche\.m2\repository\org\latencyutils\LatencyUtils\2.0.3\LatencyUtils-2.0.3.jar;C:\Users\rujche\.m2\repository\org\springframework\boot\spring-boot-starter-web\2.1.3.RELEASE\spring-boot-starter-web-2.1.3.RELEASE.jar;C:\Users\rujche\.m2\repository\org\springframework\boot\spring-boot-starter-json\2.1.3.RELEASE\spring-boot-starter-json-2.1.3.RELEASE.jar;C:\Users\rujche\.m2\repository\com\fasterxml\jackson\datatype\jackson-datatype-jdk8\2.9.8\jackson-datatype-jdk8-2.9.8.jar;C:\Users\rujche\.m2\repository\com\fasterxml\jackson\module\jackson-module-parameter-names\2.9.8\jackson-module-parameter-names-2.9.8.jar;C:\Users\rujche\.m2\repository\org\springframework\boot\spring-boot-starter-tomcat\2.1.3.RELEASE\spring-boot-starter-tomcat-2.1.3.RELEASE.jar;C:\Users\rujche\.m2\repository\org\apache\tomcat\embed\tomcat-embed-core\9.0.16\tomcat-embed-core-9.0.16.jar;C:\Users\rujche\.m2\repository\org\apache\tomcat\embed\tomcat-embed-el\9.0.16\tomcat-embed-el-9.0.16.jar;C:\Users\rujche\.m2\repository\org\apache\tomcat\embed\tomcat-embed-websocket\9.0.16\tomcat-embed-websocket-9.0.16.jar;C:\Users\rujche\.m2\repository\org\hibernate\validator\hibernate-validator\6.0.14.Final\hibernate-validator-6.0.14.Final.jar;C:\Users\rujche\.m2\repository\javax\validation\validation-api\2.0.1.Final\validation-api-2.0.1.Final.jar;C:\Users\rujche\.m2\repository\org\jboss\logging\jboss-logging\3.3.2.Final\jboss-logging-3.3.2.Final.jar;C:\Users\rujche\.m2\repository\com\fasterxml\classmate\1.4.0\classmate-1.4.0.jar;C:\Users\rujche\.m2\repository\org\springframework\spring-web\5.1.5.RELEASE\spring-web-5.1.5.RELEASE.jar;C:\Users\rujche\.m2\repository\org\springframework\spring-beans\5.1.5.RELEASE\spring-beans-5.1.5.RELEASE.jar;C:\Users\rujche\.m2\repository\org\springframework\spring-webmvc\5.1.5.RELEASE\spring-webmvc-5.1.5.RELEASE.jar;C:\Users\rujche\.m2\repository\org\springframework\spring-aop\5.1.5.RELEASE\spring-aop-5.1.5.RELEASE.jar;C:\Users\rujche\.m2\repository\org\springframework\spring-expression\5.1.5.RELEASE\spring-expression-5.1.5.RELEASE.jar;C:\Users\rujche\.m2\repository\com\microsoft\azure\azure-keyvault-secrets-spring-boot-starter\2.1.6\azure-keyvault-secrets-spring-boot-starter-2.1.6.jar;C:\Users\rujche\.m2\repository\com\microsoft\azure\azure-spring-boot-starter\2.1.6\azure-spring-boot-starter-2.1.6.jar;C:\Users\rujche\.m2\repository\org\springframework\boot\spring-boot-starter-validation\2.1.3.RELEASE\spring-boot-starter-validation-2.1.3.RELEASE.jar;C:\Users\rujche\.m2\repository\com\microsoft\azure\azure-spring-boot\2.1.6\azure-spring-boot-2.1.6.jar;C:\Users\rujche\.m2\repository\com\microsoft\azure\azure-keyvault\1.0.0\azure-keyvault-1.0.0.jar;C:\Users\rujche\.m2\repository\com\microsoft\azure\azure-client-runtime\1.0.0\azure-client-runtime-1.0.0.jar;C:\Users\rujche\.m2\repository\com\microsoft\rest\client-runtime\1.6.0\client-runtime-1.6.0.jar;C:\Users\rujche\.m2\repository\com\squareup\retrofit2\retrofit\2.4.0\retrofit-2.4.0.jar;C:\Users\rujche\.m2\repository\com\squareup\okhttp3\okhttp\3.11.0\okhttp-3.11.0.jar;C:\Users\rujche\.m2\repository\com\squareup\okio\okio\1.14.0\okio-1.14.0.jar;C:\Users\rujche\.m2\repository\com\squareup\okhttp3\logging-interceptor\3.11.0\logging-interceptor-3.11.0.jar;C:\Users\rujche\.m2\repository\com\squareup\okhttp3\okhttp-urlconnection\3.11.0\okhttp-urlconnection-3.11.0.jar;C:\Users\rujche\.m2\repository\com\squareup\retrofit2\converter-jackson\2.4.0\converter-jackson-2.4.0.jar;C:\Users\rujche\.m2\repository\com\fasterxml\jackson\datatype\jackson-datatype-joda\2.9.8\jackson-datatype-joda-2.9.8.jar;C:\Users\rujche\.m2\repository\joda-time\joda-time\2.10.1\joda-time-2.10.1.jar;C:\Users\rujche\.m2\repository\io\reactivex\rxjava\1.3.8\rxjava-1.3.8.jar;C:\Users\rujche\.m2\repository\com\squareup\retrofit2\adapter-rxjava\2.4.0\adapter-rxjava-2.4.0.jar;C:\Users\rujche\.m2\repository\com\microsoft\azure\azure-keyvault-webkey\1.0.0\azure-keyvault-webkey-1.0.0.jar;C:\Users\rujche\.m2\repository\com\google\guava\guava\20.0\guava-20.0.jar;C:\Users\rujche\.m2\repository\com\microsoft\azure\azure-client-authentication\1.6.1\azure-client-authentication-1.6.1.jar;C:\Users\rujche\.m2\repository\commons-codec\commons-codec\1.11\commons-codec-1.11.jar;C:\Users\rujche\.m2\repository\com\microsoft\azure\azure-annotations\1.2.0\azure-annotations-1.2.0.jar;C:\Users\rujche\.m2\repository\com\microsoft\azure\adal4j\1.6.0\adal4j-1.6.0.jar;C:\Users\rujche\.m2\repository\com\nimbusds\oauth2-oidc-sdk\5.24.1\oauth2-oidc-sdk-5.24.1.jar;C:\Users\rujche\.m2\repository\javax\mail\mail\1.4.7\mail-1.4.7.jar;C:\Users\rujche\.m2\repository\javax\activation\activation\1.1\activation-1.1.jar;C:\Users\rujche\.m2\repository\com\github\stephenc\jcip\jcip-annotations\1.0-1\jcip-annotations-1.0-1.jar;C:\Users\rujche\.m2\repository\org\apache\commons\commons-lang3\3.8.1\commons-lang3-3.8.1.jar;C:\Users\rujche\.m2\repository\org\apache\commons\commons-collections4\4.1\commons-collections4-4.1.jar;C:\Users\rujche\.m2\repository\com\nimbusds\lang-tag\1.4.4\lang-tag-1.4.4.jar;C:\Users\rujche\.m2\repository\com\nimbusds\nimbus-jose-jwt\8.16\nimbus-jose-jwt-8.16.jar;C:\Users\rujche\.m2\repository\com\google\code\gson\gson\2.8.5\gson-2.8.5.jar;C:\Users\rujche\.m2\repository\org\slf4j\slf4j-api\1.7.25\slf4j-api-1.7.25.jar;C:\Users\rujche\.m2\repository\commons-io\commons-io\2.3\commons-io-2.3.jar;C:\Users\rujche\.m2\repository\org\springframework\boot\spring-boot-devtools\2.1.3.RELEASE\spring-boot-devtools-2.1.3.RELEASE.jar;C:\Users\rujche\.m2\repository\org\springframework\boot\spring-boot\2.1.3.RELEASE\spring-boot-2.1.3.RELEASE.jar;C:\Users\rujche\.m2\repository\org\springframework\boot\spring-boot-autoconfigure\2.1.3.RELEASE\spring-boot-autoconfigure-2.1.3.RELEASE.jar;C:\Users\rujche\.m2\repository\net\minidev\json-smart\2.3\json-smart-2.3.jar;C:\Users\rujche\.m2\repository\net\minidev\accessors-smart\1.2\accessors-smart-1.2.jar;C:\Users\rujche\.m2\repository\org\ow2\asm\asm\5.0.4\asm-5.0.4.jar;C:\Users\rujche\.m2\repository\org\springframework\spring-core\5.1.5.RELEASE\spring-core-5.1.5.RELEASE.jar;C:\Users\rujche\.m2\repository\org\springframework\spring-jcl\5.1.5.RELEASE\spring-jcl-5.1.5.RELEASE.jar com.example.demo.DemoApplication

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.1.3.RELEASE)

2020-05-11 14:25:41.797  INFO 20784 --- [  restartedMain] com.example.demo.DemoApplication         : Starting DemoApplication on cn-rujche-0507 with PID 20784 (C:\Users\rujche\Work\key-vault-demo\demo\target\classes started by rujche in C:\Users\rujche\Work\key-vault-demo\demo)
2020-05-11 14:25:41.800  INFO 20784 --- [  restartedMain] com.example.demo.DemoApplication         : No active profile set, falling back to default profiles: default
2020-05-11 14:25:41.833  INFO 20784 --- [  restartedMain] .e.DevToolsPropertyDefaultsPostProcessor : Devtools property defaults active! Set 'spring.devtools.add-properties' to 'false' to disable
2020-05-11 14:25:41.833  INFO 20784 --- [  restartedMain] .e.DevToolsPropertyDefaultsPostProcessor : For additional web related logging consider setting the 'logging.level.web' property to 'DEBUG'
2020-05-11 14:25:43.258  INFO 20784 --- [  restartedMain] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2020-05-11 14:25:43.286  INFO 20784 --- [  restartedMain] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2020-05-11 14:25:43.286  INFO 20784 --- [  restartedMain] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.16]
2020-05-11 14:25:43.297  INFO 20784 --- [  restartedMain] o.a.catalina.core.AprLifecycleListener   : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [C:\Program Files\Java\jdk-11.0.7\bin;C:\windows\Sun\Java\bin;C:\windows\system32;C:\windows;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\Program Files\Common Files\Microsoft Shared\Microsoft Online Services;C:\Program Files (x86)\Common Files\Microsoft Shared\Microsoft Online Services;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\windows\System32\OpenSSH\;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\dotnet\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\Git\cmd;C:\Users\rujche\AppData\Local\Microsoft\WindowsApps;C:\Users\rujche\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\rujche\exe;C:\Users\rujche\exe\apache-maven-3.6.3-bin\apache-maven-3.6.3\bin;;.]
2020-05-11 14:25:43.432  INFO 20784 --- [  restartedMain] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2020-05-11 14:25:43.433  INFO 20784 --- [  restartedMain] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 1600 ms
2020-05-11 14:25:43.789  INFO 20784 --- [pool-3-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 80809e48-aa17-418d-b704-b340a87d7013] Instance discovery was successful
2020-05-11 14:25:44.463  INFO 20784 --- [pool-4-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: a061aecc-a9bf-4009-84e4-0b26c47a69e5] Instance discovery was successful
2020-05-11 14:25:44.738  INFO 20784 --- [pool-5-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: e10ea216-b7c2-4151-be24-3ec6da5f9b66] Instance discovery was successful
2020-05-11 14:25:45.006  INFO 20784 --- [pool-6-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 4a3d29c9-99bf-4f88-bbd2-6d177b813940] Instance discovery was successful
2020-05-11 14:25:45.263  INFO 20784 --- [pool-7-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 6537ab1d-32c1-4cce-a1de-903f617fdb9b] Instance discovery was successful
2020-05-11 14:25:45.537  INFO 20784 --- [pool-8-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 05d9e7d7-4fa9-48b1-b263-1a290a8b6d31] Instance discovery was successful
2020-05-11 14:25:45.932  INFO 20784 --- [pool-9-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: fd9f1919-877e-4b74-a9cc-22986f81dad7] Instance discovery was successful
2020-05-11 14:25:46.220  INFO 20784 --- [ool-10-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 46e0a84f-4d40-418d-b203-1805d7f641a4] Instance discovery was successful
2020-05-11 14:25:46.467  INFO 20784 --- [ool-11-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 4306e850-fcb9-4e67-90d7-67b6fc4b5bf9] Instance discovery was successful
2020-05-11 14:25:46.846  INFO 20784 --- [ool-12-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: d0629555-5685-4e16-a6d4-1bc12c117dba] Instance discovery was successful
2020-05-11 14:25:47.117  INFO 20784 --- [ool-13-thread-1] c.m.aad.adal4j.AuthenticationAuthority   : [Correlation ID: 6e0ff61f-4c94-4928-8cda-8137f0e14719] Instance discovery was successful
2020-05-11 14:25:47.578  INFO 20784 --- [  restartedMain] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2020-05-11 14:25:47.861  INFO 20784 --- [  restartedMain] o.s.b.d.a.OptionalLiveReloadServer       : LiveReload server is running on port 35729
2020-05-11 14:25:47.869  INFO 20784 --- [  restartedMain] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 2 endpoint(s) beneath base path '/actuator'
2020-05-11 14:25:47.942  INFO 20784 --- [  restartedMain] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2020-05-11 14:25:47.945  INFO 20784 --- [  restartedMain] com.example.demo.DemoApplication         : Started DemoApplication in 10.907 seconds (JVM running for 11.51)

[VikramNisarga]

@chenrujun I need some time as I don't have the setup handy(it's been almost an year since I had this issue) nevertheless I'll try your demo project in sometime. If I recall correctly the issue was only with Eclipse OpenJ9 JVM, Are you running with OpenJ9 ? This is just my observation. @jpnaidu07 are you facing this issue on hotspot?

[jpnaidu07]

Hi @chenrujun as i already mentioned,

Along with those dependencies it seems it is explicitly fetching all secrets one by one (Normally it should fetch the secret whenever we use that secret in the project using @value or @Environment and that is how it is expected to work even along with those dependencies).

when there are no dependencies (actuator and spring cloud ) it is working fine as usual for eg: if im using 4 secrets from keyvault in my project using @value it will show only "Instance discovery was successful" 4 times.

But with those dependencies it is showing Instance discovery was successful more than 150 times, since in my keyvault i have 150 secrets and im in proxy (VPN) which makes it even slow.

Im not sure, when ever if we add these dependencies (actuator and spring cloud ) the KeyVaultEnvironmentPostProcessor or any other functionality is messed up and fetches all the secrets, just my thought. since spring cloud config also works with Environments and propertySources. please correct me if my understanding is wrong

To replicate what im saying from your demo project
1.Remove all @value annotated members from your DemoApplication.java
2.Remove azure.keyvault.secret.keys from application yml
3.Run your application without dependencies (actuator and spring cloud) and see the logs you wont find any Instance discovery was successful string.
4.Run your application with dependencies (actuator and spring cloud) and see the logs you will find Instance discovery was successful as many as secrets in the keyvault.

and im not using azure.keyvault.secret.keys in application my yml/properties.

Thanks for the support.