Replace hierarchical metrics with micrometer-based dimensional metrics #9970

jkschneider · 2017-08-07T22:20:44Z

Let this be a starting point for discussion. Currently:

I'm doing pre 1.0 releases of micrometer every several days. The first milestone release of micrometer can correspond to the Boot 2 M4 release.
See the Micrometer manual for documentation about what Spring integration looks like. These docs should eventually work their way into the Spring Boot reference manual as well.
This favors @EnableAtlasMetrics, etc instead of classpath-based autoconfiguration because support for several monitoring systems are based on the same dependency (spectator-api).
I'll add back in the Webflux support soon! @smaldini
Micrometer itself is lacking a few pieces of instrumentation that existed in actuator before (for example caffeine caches), but this gap will be closed before M4.

I expect many changes, but look forward to your feedback!

cc / @philwebb @mbhave

jkschneider · 2017-08-07T22:24:33Z

Dependencies and autoconfiguration

The "quickstart" for Spring Boot is currently:

Add @EnablePrometheusMetrics to your app.
Add io.micrometer:micrometer-prometheus-starter as a dependency.

We could drop the annotation and add a marker interface in each starter module for use in autoconfiguration, but I'm a bit concerned about what happens if more than one starter winds up on the classpath. Open to suggestions on this.

spencergibb

I think activation was done on a user provided bean.

spencergibb · 2017-08-07T22:26:52Z

spring-boot-actuator/pom.xml

 		<!-- Optional -->
 		<dependency>
+			<groupId>io.prometheus</groupId>
+			<artifactId>simpleclient_common</artifactId>
+			<version>0.0.26</version>


Versions probably go in spring-boot-dependencies

spencergibb · 2017-08-07T22:27:44Z

.../src/main/java/org/springframework/boot/actuate/autoconfigure/EndpointAutoConfiguration.java

-import org.springframework.boot.actuate.endpoint.RequestMappingEndpoint;
-import org.springframework.boot.actuate.endpoint.ShutdownEndpoint;
-import org.springframework.boot.actuate.endpoint.TraceEndpoint;
+import org.springframework.boot.actuate.endpoint.*;


* imports :-)

wilkinsona · 2017-08-09T19:07:47Z

Thanks, @jkschneider. This is great stuff. I'm excited to get it into Boot. It's going to take a while to digest, particularly as half the team are on holiday. In the meantime, here's a few general comments:

All copyright headers should look like this:

/*
 * Copyright 2012-2017 the original author or authors.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

We use constructor injection everywhere, even in configuration classes
If you want to inject something that's @Autowired(required=false) we constructor inject an ObjectProvider<Whatever> instead
All new public classes should be @since 2.0.0

You can fix these if you're so inclined, or we can take care of them when we merge this. I just mention them now as I noticed them on a first read through of the changes.

deki · 2017-08-11T15:53:59Z

Will this be part of 2.0.0?
As #5875 was dropped I would appreciate it :-)

wilkinsona · 2017-08-11T15:55:47Z

@deki Yes, it will. I wouldn't have closed #5875 if we didn't plan to get this into 2.0

wilkinsona · 2017-08-30T14:12:11Z

I've started a more in-depth review and polishing of these changes here. It's still very much a work in progress.

The biggest change and potential issue that I've seen thus far is that with the Micrometer-based metrics we lose the metrics endpoint and, I think, the ability for an app to be self-sufficient in terms of storing and "serving" its own metrics. That functionality isn't much use in a serious production environment, but I suspect that people will miss it elsewhere.

@jkschneider You've described Micrometer as "think SLF4J, but for metrics". Is there an equivalent of slf4j-simple that I've missed?

jkschneider · 2017-08-30T22:00:13Z

@wilkinsona The boot configuration has changed quite a bit in preparation for Micrometer release 0.11.0. Once that release goes out, I'll update this PR.

There is a SimpleMeterRegistry, yes.

I've talked with a number of folks about the metrics endpoint. My opinion currently is it isn't going to really be possible to create a meaningful information dump in JSON from dimensional metrics. For the single instance self-service route, Netflix uses JMX, and Micrometer packs with a JMX registry that flattens dimensions to a hierarchical name and ships them there. We should talk more about this!

jkschneider · 2017-08-30T22:09:18Z

It may be more productive when reviewing to review the contents of micrometer-spring-legacy and imagine that reactive support is added.

wilkinsona · 2017-08-31T05:43:25Z

The boot configuration has changed quite a bit in preparation for Micrometer release 0.11.0. Once that release goes out, I'll update this PR

Cool, thank you. If possible, can you update the PR on top of my branch please?

wilkinsona · 2017-08-31T05:45:07Z

For the single instance self-service route, Netflix uses JMX, and Micrometer packs with a JMX registry that flattens dimensions to a hierarchical name and ships them there. We should talk more about this!

My, perhaps naive, reaction to that is that if we can flatten the dimensions to a hierarchical name and ship them over JMX then we should be able to do the same for HTTP/JSON. Failing that, a single endpoint that returns a JSON payload containing all the metrics may be better than nothing.

See gh-9970

Update `MetricsAutoConfiguration` to use an `ObjectProvider` to guard against missing beans. See gh-9970

Update `MetricsClientHttpRequestInterceptor` to use `percentilesTime` rather than `percentiles`. See gh-9970

Update configuration property classes used with micrometer so that they no longer directly implement `Config` interfaces. Properties are now adapted to Config implementations independently. See spring-projectsgh-9970

Update configuration property classes used with micrometer so that they no longer directly implement `Config` interfaces. Properties are now adapted to Config implementations independently. See gh-9970

manderson23 · 2017-09-22T13:17:23Z

After this change are the Micrometer metrics reported via the /metrics actuator endpoint?

jkschneider · 2017-09-22T13:21:01Z

@manderson23 Yes, though because of the switch to a dimensional system, the format necessarily changed. There are some inadequacies in the /metrics endpoint that shipped with Boot 2 M4 that will be addressed in the M5 release (micrometer 1.0.0-rc.2) next week. See micrometer-metrics/micrometer#124.

ryonday · 2017-10-18T03:43:31Z

Is there going to be a migration guide for those of us who have, say, CounterService injected throughout our applications? It looks like right now we're being left to twist in the wind, especially since the snapshot docs still refer to CounterService and GaugeService everywhere.

wilkinsona · 2017-10-18T06:26:42Z

@ryonday Please comment on #10313 if there's anything specific that you'd like to see in the documentation on upgrading from 1.x.

especially since the snapshot docs still refer to CounterService and GaugeService everywhere.

I can't find any instances of CounterService or GaugeService in https://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/. Can you please link to some examples?

ryonday · 2017-10-18T16:21:18Z

@wilkinsona Thanks for the quick response!

For example: https://github.com/spring-projects/spring-boot/blob/master/spring-boot-project/spring-boot-actuator/README.adoc line 41 maintains the CounterService and GaugeService references (though not under these names).

It's unclear from the 2.0 documentation the prescribed method of recording one's own metrics (the section was removed from the docs). Is the expectation that we're use Micrometer directly within our applications? the CounterService and GaugeService interfaces offered a way to replace implementations while using only Spring interfaces.

wilkinsona · 2017-10-18T16:28:59Z

Thanks. I forgot about the README when updating the documentation. I've opened #10686.

Is the expectation that we're use Micrometer directly within our applications?

Yes, that's right. Micrometer's API is a replacement for CounterService, GaugeService, and friends.

the CounterService and GaugeService interfaces offered a way to replace implementations while using only Spring interfaces.

Micrometer has the same goal. It's led by a member of the Spring team (@jkschneider) and aims to be SLF4J but for metrics. You can use the MeterRegistry interface to register things and Micrometer will take care of mapping that onto whatever metrics backend(s) you're using.

jkschneider · 2017-10-18T18:32:14Z

@ryonday There wasn't much we could do to preserve the original CounterService and GaugeService interfaces, since they assumed a hierarchical representation of metric names and there is no reliable mapping from hierarchical names onto a dimensional scheme. The converse is possible, however.

spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label Aug 7, 2017

spencergibb reviewed Aug 7, 2017

View reviewed changes

jkschneider mentioned this pull request Aug 11, 2017

Add Support for Prometheus Metrics #8130

Closed

jkschneider mentioned this pull request Aug 22, 2017

Provide autocompletable @ConfigurationProperties for registry configs micrometer-metrics/micrometer#90

Closed

snicoll mentioned this pull request Aug 25, 2017

Need option for actuator+dropwizard timers for short timings #8770

Closed

wilkinsona mentioned this pull request Aug 30, 2017

Added increment/decrement by delta to CounterService and implementations #10107

Closed

wilkinsona self-assigned this Aug 30, 2017

wilkinsona mentioned this pull request Aug 30, 2017

Allow an operation on an endpoint to specify the media type that it produces #10118

Closed

micheljung mentioned this pull request Sep 8, 2017

Use multi-dimensional metrics, as soon as available FAForever/faf-java-server#34

Closed

jkschneider added 2 commits September 10, 2017 22:59

Remove old metrics code and dependencies

cae634a

Add micrometer metrics

0108338

wilkinsona added this to the 2.0.0.M4 milestone Sep 14, 2017

wilkinsona added priority: high type: enhancement A general enhancement and removed status: waiting-for-triage An issue we've not yet triaged labels Sep 14, 2017

wilkinsona mentioned this pull request Sep 14, 2017

open DataSourcePublicMetrics for easy extension (#10288) #10289

Closed

wilkinsona added the theme: metrics label Sep 14, 2017

wilkinsona closed this in c2958c2 Sep 14, 2017

snicoll added a commit that referenced this pull request Sep 14, 2017

Add error metadata for the Micrometer migration

ddde3c5

See gh-9970

wilkinsona added a commit that referenced this pull request Sep 14, 2017

Consistently use the spring.metrics configuration property prefix

3942ba2

See gh-9970

wilkinsona added a commit that referenced this pull request Sep 14, 2017

Add dependencies required for javadoc generation

ddca0b7

See gh-9970

philwebb added a commit that referenced this pull request Sep 14, 2017

Add missing ObjectProvider for missing beans

21b645f

Update `MetricsAutoConfiguration` to use an `ObjectProvider` to guard against missing beans. See gh-9970

philwebb added a commit that referenced this pull request Sep 14, 2017

Use correct percentilesTime call

796d400

Update `MetricsClientHttpRequestInterceptor` to use `percentilesTime` rather than `percentiles`. See gh-9970

This was referenced Sep 18, 2017

Add InfluxDB exporter to Actuator #7351

Closed

Produce Dropwizard timer metrics instead of gauges in MetricsFilter #9555

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace hierarchical metrics with micrometer-based dimensional metrics #9970

Replace hierarchical metrics with micrometer-based dimensional metrics #9970

jkschneider commented Aug 7, 2017 •

edited by wilkinsona

Loading

jkschneider commented Aug 7, 2017

spencergibb left a comment

spencergibb Aug 7, 2017

spencergibb Aug 7, 2017

wilkinsona commented Aug 9, 2017 •

edited

Loading

deki commented Aug 11, 2017

wilkinsona commented Aug 11, 2017

wilkinsona commented Aug 30, 2017

jkschneider commented Aug 30, 2017 •

edited

Loading

jkschneider commented Aug 30, 2017

wilkinsona commented Aug 31, 2017

wilkinsona commented Aug 31, 2017 •

edited

Loading

manderson23 commented Sep 22, 2017

jkschneider commented Sep 22, 2017 •

edited

Loading

ryonday commented Oct 18, 2017

wilkinsona commented Oct 18, 2017

ryonday commented Oct 18, 2017

wilkinsona commented Oct 18, 2017

jkschneider commented Oct 18, 2017

Replace hierarchical metrics with micrometer-based dimensional metrics #9970

Replace hierarchical metrics with micrometer-based dimensional metrics #9970

Conversation

jkschneider commented Aug 7, 2017 • edited by wilkinsona Loading

jkschneider commented Aug 7, 2017

Dependencies and autoconfiguration

spencergibb left a comment

Choose a reason for hiding this comment

spencergibb Aug 7, 2017

Choose a reason for hiding this comment

spencergibb Aug 7, 2017

Choose a reason for hiding this comment

wilkinsona commented Aug 9, 2017 • edited Loading

deki commented Aug 11, 2017

wilkinsona commented Aug 11, 2017

wilkinsona commented Aug 30, 2017

jkschneider commented Aug 30, 2017 • edited Loading

jkschneider commented Aug 30, 2017

wilkinsona commented Aug 31, 2017

wilkinsona commented Aug 31, 2017 • edited Loading

manderson23 commented Sep 22, 2017

jkschneider commented Sep 22, 2017 • edited Loading

ryonday commented Oct 18, 2017

wilkinsona commented Oct 18, 2017

ryonday commented Oct 18, 2017

wilkinsona commented Oct 18, 2017

jkschneider commented Oct 18, 2017

jkschneider commented Aug 7, 2017 •

edited by wilkinsona

Loading

wilkinsona commented Aug 9, 2017 •

edited

Loading

jkschneider commented Aug 30, 2017 •

edited

Loading

wilkinsona commented Aug 31, 2017 •

edited

Loading

jkschneider commented Sep 22, 2017 •

edited

Loading