From fad6d726fea1cc841389d69daa67bfa79e5f4772 Mon Sep 17 00:00:00 2001 From: Santiago Pericasgeertsen Date: Mon, 22 Nov 2021 13:47:07 -0500 Subject: [PATCH] Implementation of Fault tolerance 4.0 (#3664) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Fault Tolerance 3.0 Support (#2680) * Initial changes to implement new metrics layer. Moving from complex names to simpler names and tags. Signed-off-by: Santiago Pericasgeertsen * More metric updates. Signed-off-by: Santiago Pericasgeertsen * Migration of most unit tests to new metrics. Signed-off-by: Santiago Pericasgeertsen * Completed migration of metrics test. Signed-off-by: Santiago Pericasgeertsen * New exception to discern timeouts during retries. Signed-off-by: Santiago Pericasgeertsen * Implementation of retry metrics. Signed-off-by: Santiago Pericasgeertsen * Cleanup metrics between tests. Signed-off-by: Santiago Pericasgeertsen * Several changes related to execution of FT 3.0 TCKs. Adjusted initial size of executors and fixed a few other problems. Signed-off-by: Santiago Pericasgeertsen * Copyright and checkstyle updates. Signed-off-by: Santiago Pericasgeertsen * Fixed copyright year. Signed-off-by: Santiago Pericasgeertsen * Fixed typos and some cleanup. Signed-off-by: Santiago Pericasgeertsen * Created exclude file as a workaround for a sportbugs' bug. Signed-off-by: Santiago Pericasgeertsen * Updated copyright year. Signed-off-by: Santiago Pericasgeertsen * MicroProfile Opentracing 2.0 (#2676) * Microprofile Opentracing uprgated to 2.0 * Unused dependences removed * Obsolete excludes removed * Sync up of microprofile-4.0 with master branch (#2757) * Upgrade Netty to 4.1.58 (#2678) Signed-off-by: Tomas Langer * Added overall timeout to evictable cache (#2659) Signed-off-by: Tomas Langer * Fix copyright year for commits broken by squashing. (#2687) Signed-off-by: Tomas Langer * Concat array enhancement (#2508) * Concat array enhancement Signed-off-by: Daniel Kec * Update Jackson to 2.12.1 (#2690) * Update Jackson to 2.12.1 * Upgrade to latest Junit5 to get fix for https://github.com/junit-team/junit5/issues/2198 * Manage junit4 version * PokemonService template fixed in SE Database Archetype. (#2701) Signed-off-by: Tomas Kraus * Fixed different output in DbClient SE archetype (#2703) Signed-off-by: Tomas Kraus * Fix TODO application: (#2708) - WebSecurity needs to be passed config.get("security") to take the "security.web-server" configuration - Added outbound configuration for the google login - Upgraded cassandra driver to fix issues with old guava dependencies - Removed metrics to avoid issues with cassandra driver. Fixes #2707 * Update k8s descriptors to avoid using deprecated APIs. (#2719) * Separate execution of DataChunkReleaseTest in its own VM to prevent leak messages in other test's logs. (#2716) Signed-off-by: Santiago Pericasgeertsen * Changes in this commit: (#2727) 1. Upgrade to Jersey 2.33 2. Configuration via system properties for the Jersey Client API. Any response in an exception will be mapped to an empty one to prevent data leaks. See https://github.com/eclipse-ee4j/jersey/pull/4641. Signed-off-by: Santiago Pericasgeertsen * Properly release underlying buffer before passing it to WebSocket handler (#2715) * Properly release underlying buffer before passing it to handler. * Releases data chunks after passing them to Tyrus without any copying. Reports an error and closes connection if Tyrus is unable to handle the data. Finally, fixed a problem related to subscription requests. Signed-off-by: Santiago Pericasgeertsen * Removed unused logger. Signed-off-by: Santiago Pericasgeertsen * Fixed checkstyle. Signed-off-by: Santiago Pericasgeertsen * Fix issue with null value in JSON. (#2723) Signed-off-by: Tomas Langer * Upgrade grpc to v1.35.0 (#2713) * Upgrade grpc to v1.35.0 * Update copyright * Upgrades OCI SDK to version 1.31.0 (#2699) * Updated OCI to 1.31.0 Signed-off-by: Laird Nelson * Fix null array values in HOCON/JSON config parser. (#2731) Resolves #2720 (follow-up) * Performance improvements to queue(s) management in Webserver (#2704) * Initial patch. Signed-off-by: Santiago Pericasgeertsen * Fixed some type params and improved comments. Signed-off-by: Santiago Pericasgeertsen * More cleanup and make sure to fail publisher on an error condition. Signed-off-by: Santiago Pericasgeertsen * Suppress warnings. Signed-off-by: Santiago Pericasgeertsen * Call clearQueues on every new request for proper cleanup of keep-alive connections. Some copyright fixes. Signed-off-by: Santiago Pericasgeertsen * Fixed checkstyle issues. Signed-off-by: Santiago Pericasgeertsen * Force logging of LEAK error even if finalize does not get called on a DataChunk. Signed-off-by: Santiago Pericasgeertsen * Upgrade Weld (#2668) Signed-off-by: Tomas Langer * Rest client async header propagation with usage of Helidon Context (#2735) Rest client header propagation with usage of Helidon Context Signed-off-by: David Kral * Allow override of Jersey property via config (#2737) * Allow the default value of property jersey.config.client.ignoreExceptionResponse to be overridden via config. New test. Signed-off-by: Santiago Pericasgeertsen * Fixed copyright year. Signed-off-by: Santiago Pericasgeertsen * New implementation of LazyValue (#2738) * New implementation of LazyValue that lazily initializes a Semaphore instead of eagerly creating a ReentrantLock. Makes use of volatile guarantees and atomicity of VarHandle updates. Signed-off-by: Santiago Pericasgeertsen * New test for LazyValueImpl. Signed-off-by: Santiago Pericasgeertsen * Reduced sleep time in test. Signed-off-by: Santiago Pericasgeertsen * Update CHANGELOG for 2.2.1 release (#2743) * 2.2.1 THIRD_PARTY_LICENSES update (#2746) * Update THIRD_PARTY_LICENSES * Support async invocations using optional synthetic SimplyTimed behavior (#2745) * Add support for async invocations for optional inferred SimplyTimed behavior on JAX-RS endpoints Signed-off-by: tim.quinn@oracle.com * Do not attempt to access the request context in Fallback callback. If used together with Retry, it is possible for the fallback to be called in a fresh thread for which there is no current request scope. Instead just use the original value obtained in this class' constructor. Updated functional test (with some class renaming) to cover this use case. (#2748) Signed-off-by: Santiago Pericasgeertsen * Fix for native image. (#2753) Signed-off-by: Tomas Langer * Fixed checkstyle issues. Signed-off-by: Santiago Pericasgeertsen Co-authored-by: Tomas Langer Co-authored-by: Daniel Kec Co-authored-by: Joe DiPol Co-authored-by: Tomáš Kraus Co-authored-by: Romain Grecourt Co-authored-by: Jonathan Knight Co-authored-by: Laird Nelson Co-authored-by: David Král Co-authored-by: Tim Quinn * Fixed problems in RetryImpl after merge. Signed-off-by: Santiago Pericasgeertsen * Fixed problems with metrics after merge. Signed-off-by: Santiago Pericasgeertsen * Updated version in suite file. Signed-off-by: Santiago Pericasgeertsen * Fixed problem retrieving registry for metrics. Signed-off-by: Santiago Pericasgeertsen * Fixed more problems after merge. All tests are passing now. Signed-off-by: Santiago Pericasgeertsen * Fixed checkstyle errors. Signed-off-by: Santiago Pericasgeertsen * Fixed TODO. Signed-off-by: Santiago Pericasgeertsen * Enabled TCK's by default and removed generated file. Signed-off-by: Santiago Pericasgeertsen * One more checkstyle violation. Signed-off-by: Santiago Pericasgeertsen * Removed duplicate test after merge. Signed-off-by: Santiago Pericasgeertsen Co-authored-by: Dmitry Aleksandrov Co-authored-by: Tomas Langer Co-authored-by: Daniel Kec Co-authored-by: Joe DiPol Co-authored-by: Tomáš Kraus Co-authored-by: Romain Grecourt Co-authored-by: Jonathan Knight Co-authored-by: Laird Nelson Co-authored-by: David Král Co-authored-by: Tim Quinn --- .../io/helidon/faulttolerance/RetryImpl.java | 15 +- .../faulttolerance/RetryTimeoutException.java | 51 + .../native-image.properties | 17 + .../fault-tolerance/etc/spotbugs/exclude.xml | 34 + microprofile/fault-tolerance/pom.xml | 4 + ...mmandFallback.java => FallbackHelper.java} | 19 +- .../FaultToleranceExtension.java | 19 +- .../faulttolerance/FaultToleranceMetrics.java | 1050 ++++++++++++----- .../faulttolerance/JavaMethodFinder.java | 4 +- .../faulttolerance/MethodIntrospector.java | 30 + .../faulttolerance/MethodInvoker.java | 308 ++--- .../faulttolerance/RetryAntn.java | 5 +- .../faulttolerance/ThrowableMapper.java | 5 +- .../faulttolerance/FaultToleranceTest.java | 10 +- .../faulttolerance/MetricsBean.java | 14 - .../faulttolerance/MetricsTest.java | 482 +++++--- .../HelidonDeployableContainer.java | 34 +- .../tests/tck/tck-fault-tolerance/pom.xml | 10 +- .../src/test/tck-suite.xml | 4 +- 19 files changed, 1461 insertions(+), 654 deletions(-) create mode 100644 fault-tolerance/src/main/java/io/helidon/faulttolerance/RetryTimeoutException.java create mode 100644 jersey/client/src/main/resources/META-INF/native-image/io.helidon.jersey/helidon-jersey-client/native-image.properties create mode 100644 microprofile/fault-tolerance/etc/spotbugs/exclude.xml rename microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/{CommandFallback.java => FallbackHelper.java} (88%) diff --git a/fault-tolerance/src/main/java/io/helidon/faulttolerance/RetryImpl.java b/fault-tolerance/src/main/java/io/helidon/faulttolerance/RetryImpl.java index 79a47acb1b0..e14153ce280 100644 --- a/fault-tolerance/src/main/java/io/helidon/faulttolerance/RetryImpl.java +++ b/fault-tolerance/src/main/java/io/helidon/faulttolerance/RetryImpl.java @@ -73,9 +73,10 @@ private Single retrySingle(RetryContext> con long nanos = System.nanoTime() - context.startedNanos; if (nanos > maxTimeNanos) { - TimeoutException te = new TimeoutException("Execution took too long. Already executing: " - + TimeUnit.NANOSECONDS.toMillis(nanos) + " ms, must timeout after: " - + TimeUnit.NANOSECONDS.toMillis(maxTimeNanos) + " ms."); + TimeoutException te = new RetryTimeoutException(context.throwable(), + "Execution took too long. Already executing: " + + TimeUnit.NANOSECONDS.toMillis(nanos) + " ms, must timeout after: " + + TimeUnit.NANOSECONDS.toMillis(maxTimeNanos) + " ms."); if (context.hasThrowable()) { te.initCause(context.throwable()); } @@ -115,9 +116,10 @@ private Multi retryMulti(RetryContext> contex long nanos = System.nanoTime() - context.startedNanos; if (nanos > maxTimeNanos) { - return Multi.error(new TimeoutException("Execution took too long. Already executing: " - + TimeUnit.NANOSECONDS.toMillis(nanos) + " ms, must timeout after: " - + TimeUnit.NANOSECONDS.toMillis(maxTimeNanos) + " ms.")); + return Multi.error(new RetryTimeoutException(context.throwable(), + "Execution took too long. Already executing: " + + TimeUnit.NANOSECONDS.toMillis(nanos) + " ms, must timeout after: " + + TimeUnit.NANOSECONDS.toMillis(maxTimeNanos) + " ms.")); } if (currentCallIndex > 0) { @@ -190,3 +192,4 @@ Throwable throwable() { } } } + diff --git a/fault-tolerance/src/main/java/io/helidon/faulttolerance/RetryTimeoutException.java b/fault-tolerance/src/main/java/io/helidon/faulttolerance/RetryTimeoutException.java new file mode 100644 index 00000000000..fc62c476e35 --- /dev/null +++ b/fault-tolerance/src/main/java/io/helidon/faulttolerance/RetryTimeoutException.java @@ -0,0 +1,51 @@ +/* + * Copyright (c) 2021 Oracle and/or its affiliates. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.helidon.faulttolerance; + +import java.util.concurrent.TimeoutException; + +/** + * Subclass of {@link TimeoutException} to discern exceptions thrown by a {@link Retry} + * when its overall timeout is reached versus those thrown by a {@link Timeout}. + */ +public class RetryTimeoutException extends TimeoutException { + private static final long serialVersionUID = 1900926677490550714L; + + private final Throwable lastRetryException; + + /** + * Constructs a {@code RetryTimeoutException} with the specified detail + * message. + * + * @param throwable last retry exception + * @param message the detail message + */ + public RetryTimeoutException(Throwable throwable, String message) { + super(message); + lastRetryException = throwable; + } + + /** + * Last exception thrown in {@code Retry} before the overall timeout reached. + * + * @return last exception thrown + */ + public Throwable lastRetryException() { + return lastRetryException; + } +} + diff --git a/jersey/client/src/main/resources/META-INF/native-image/io.helidon.jersey/helidon-jersey-client/native-image.properties b/jersey/client/src/main/resources/META-INF/native-image/io.helidon.jersey/helidon-jersey-client/native-image.properties new file mode 100644 index 00000000000..ecfa41b70b2 --- /dev/null +++ b/jersey/client/src/main/resources/META-INF/native-image/io.helidon.jersey/helidon-jersey-client/native-image.properties @@ -0,0 +1,17 @@ +# +# Copyright (c) 2021 Oracle and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +Args=--initialize-at-run-time=org.glassfish.jersey.client.internal.HttpUrlConnector diff --git a/microprofile/fault-tolerance/etc/spotbugs/exclude.xml b/microprofile/fault-tolerance/etc/spotbugs/exclude.xml new file mode 100644 index 00000000000..3bd3f33cb33 --- /dev/null +++ b/microprofile/fault-tolerance/etc/spotbugs/exclude.xml @@ -0,0 +1,34 @@ + + + + + + + + + + + + diff --git a/microprofile/fault-tolerance/pom.xml b/microprofile/fault-tolerance/pom.xml index 6c4a64ba64a..49432b86d3e 100644 --- a/microprofile/fault-tolerance/pom.xml +++ b/microprofile/fault-tolerance/pom.xml @@ -33,6 +33,10 @@ Microprofile fault tolerance implementation + + etc/spotbugs/exclude.xml + + diff --git a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/CommandFallback.java b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/FallbackHelper.java similarity index 88% rename from microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/CommandFallback.java rename to microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/FallbackHelper.java index 8c77aa3c9c3..9944a014c65 100644 --- a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/CommandFallback.java +++ b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/FallbackHelper.java @@ -1,5 +1,5 @@ /* - * Copyright (c) 2018, 2020 Oracle and/or its affiliates. + * Copyright (c) 2018, 2021 Oracle and/or its affiliates. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -26,9 +26,9 @@ import org.eclipse.microprofile.faulttolerance.FallbackHandler; /** - * Class CommandFallback. + * Implements invocation callback logic. */ -class CommandFallback { +class FallbackHelper { private final InvocationContext context; @@ -45,7 +45,7 @@ class CommandFallback { * @param introspector Method introspector. * @param throwable Throwable that caused execution of fallback */ - CommandFallback(InvocationContext context, MethodIntrospector introspector, Throwable throwable) { + FallbackHelper(InvocationContext context, MethodIntrospector introspector, Throwable throwable) { this.context = context; this.throwable = throwable; @@ -103,8 +103,6 @@ public Throwable getFailure() { result = fallbackMethod.invoke(context.getTarget(), context.getParameters()); } } catch (Throwable t) { - updateMetrics(); - // If InvocationTargetException, then unwrap underlying cause if (t instanceof InvocationTargetException) { t = t.getCause(); @@ -112,15 +110,6 @@ public Throwable getFailure() { throw t instanceof Exception ? (Exception) t : new RuntimeException(t); } - updateMetrics(); return result; } - - /** - * Updates fallback metrics. - */ - private void updateMetrics() { - Method method = context.getMethod(); - FaultToleranceMetrics.getCounter(method, FaultToleranceMetrics.FALLBACK_CALLS_TOTAL).inc(); - } } diff --git a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/FaultToleranceExtension.java b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/FaultToleranceExtension.java index 56bb30f6481..157ca18d1c1 100644 --- a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/FaultToleranceExtension.java +++ b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/FaultToleranceExtension.java @@ -229,38 +229,29 @@ private void registerFaultToleranceMethods(AnnotatedType type) { } /** - * Registers metrics for all FT methods and init executors. + * Validates annotations. * * @param validation Event information. */ - void registerMetricsAndInitExecutors(@Observes AfterDeploymentValidation validation) { + void validateAnnotations(@Observes AfterDeploymentValidation validation) { if (FaultToleranceMetrics.enabled()) { getRegisteredMethods().stream().forEach(beanMethod -> { final Method method = beanMethod.method(); final Class beanClass = beanMethod.beanClass(); - // Counters for all methods - FaultToleranceMetrics.registerMetrics(method); - - // Metrics depending on the annotationSet present if (MethodAntn.isAnnotationPresent(beanClass, method, Retry.class)) { - FaultToleranceMetrics.registerRetryMetrics(method); new RetryAntn(beanClass, method).validate(); } if (MethodAntn.isAnnotationPresent(beanClass, method, CircuitBreaker.class)) { - FaultToleranceMetrics.registerCircuitBreakerMetrics(method); new CircuitBreakerAntn(beanClass, method).validate(); } if (MethodAntn.isAnnotationPresent(beanClass, method, Timeout.class)) { - FaultToleranceMetrics.registerTimeoutMetrics(method); new TimeoutAntn(beanClass, method).validate(); } if (MethodAntn.isAnnotationPresent(beanClass, method, Bulkhead.class)) { - FaultToleranceMetrics.registerBulkheadMetrics(method); new BulkheadAntn(beanClass, method).validate(); } if (MethodAntn.isAnnotationPresent(beanClass, method, Fallback.class)) { - FaultToleranceMetrics.registerFallbackMetrics(method); new FallbackAntn(beanClass, method).validate(); } if (MethodAntn.isAnnotationPresent(beanClass, method, Asynchronous.class)) { @@ -269,17 +260,17 @@ void registerMetricsAndInitExecutors(@Observes AfterDeploymentValidation validat }); } - // Initialize executors for MP FT - default size of 16 + // Initialize executors for MP FT - default size of 20 io.helidon.config.Config config = MpConfig.toHelidonConfig(ConfigProvider.getConfig()); scheduledThreadPoolSupplier = ScheduledThreadPoolSupplier.builder() .threadNamePrefix("ft-mp-schedule-") - .corePoolSize(16) + .corePoolSize(20) .config(config.get("scheduled-executor")) .build(); FaultTolerance.scheduledExecutor(scheduledThreadPoolSupplier); threadPoolSupplier = ThreadPoolSupplier.builder() .threadNamePrefix("ft-mp-") - .corePoolSize(16) + .corePoolSize(20) .config(config.get("executor")) .build(); FaultTolerance.executor(threadPoolSupplier); diff --git a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/FaultToleranceMetrics.java b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/FaultToleranceMetrics.java index 26394571199..79c45310dd3 100644 --- a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/FaultToleranceMetrics.java +++ b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/FaultToleranceMetrics.java @@ -17,11 +17,14 @@ package io.helidon.microprofile.faulttolerance; import java.lang.reflect.Method; +import java.util.Objects; import java.util.concurrent.locks.ReentrantLock; +import java.util.function.Supplier; import io.helidon.common.LazyValue; import jakarta.enterprise.inject.spi.CDI; +import jakarta.enterprise.util.AnnotationLiteral; import org.eclipse.microprofile.metrics.Counter; import org.eclipse.microprofile.metrics.Gauge; import org.eclipse.microprofile.metrics.Histogram; @@ -31,12 +34,13 @@ import org.eclipse.microprofile.metrics.MetricRegistry; import org.eclipse.microprofile.metrics.MetricType; import org.eclipse.microprofile.metrics.MetricUnits; +import org.eclipse.microprofile.metrics.Tag; +import org.eclipse.microprofile.metrics.annotation.RegistryType; import static io.helidon.microprofile.faulttolerance.FaultToleranceExtension.getRealClass; -import static io.helidon.microprofile.faulttolerance.FaultToleranceExtension.isFaultToleranceMetricsEnabled; /** - * Class FaultToleranceMetrics. + * Utility class to register and fetch FT metrics. */ class FaultToleranceMetrics { @@ -44,7 +48,7 @@ class FaultToleranceMetrics { private static final ReentrantLock LOCK = new ReentrantLock(); private static final LazyValue METRIC_REGISTRY = LazyValue.create( - () -> CDI.current().select(MetricRegistry.class).get()); + () -> CDI.current().select(MetricRegistry.class, new BaseRegistryTypeLiteral()).get()); private FaultToleranceMetrics() { } @@ -57,310 +61,511 @@ static MetricRegistry getMetricRegistry() { return METRIC_REGISTRY.get(); } - @SuppressWarnings("unchecked") - static T getMetric(Method method, String name) { - MetricID metricID = newMetricID(String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), name)); - return (T) getMetricRegistry().getMetrics().get(metricID); + /** + * Annotation literal to inject base registry. + */ + static class BaseRegistryTypeLiteral extends AnnotationLiteral implements RegistryType { + + @Override + public MetricRegistry.Type type() { + return MetricRegistry.Type.BASE; + } } - static Counter getCounter(Method method, String name) { - return getMetric(method, name); + /** + * Base class for Fault Tolerance metrics. Shares common logic for registration + * and lookup of metrics. + */ + abstract static class FaultToleranceMetric { + + abstract String name(); + + abstract String description(); + + abstract MetricType metricType(); + + abstract String unit(); + + protected Counter getCounter(Tag... tags) { + MetricID metricID = new MetricID(name(), tags); + return (Counter) getMetricRegistry().getMetrics().get(metricID); + } + + protected Counter registerCounter(Tag... tags) { + Counter counter = getCounter(tags); + if (counter == null) { + Metadata metadata = Metadata.builder() + .withName(name()) + .withDisplayName(name()) + .withDescription(description()) + .withType(metricType()) + .withUnit(unit()) + .build(); + try { + counter = getMetricRegistry().counter(metadata, tags); + } catch (IllegalArgumentException e) { + // Looks like we lost registration race + counter = getCounter(tags); + Objects.requireNonNull(counter); + } + } + return counter; + } + + protected Histogram getHistogram(Tag... tags) { + MetricID metricID = new MetricID(name(), tags); + return (Histogram) getMetricRegistry().getMetrics().get(metricID); + } + + protected Histogram registerHistogram(Tag... tags) { + Histogram histogram = getHistogram(tags); + if (histogram == null) { + Metadata metadata = Metadata.builder() + .withName(name()) + .withDisplayName(name()) + .withDescription(description()) + .withType(metricType()) + .withUnit(unit()) + .build(); + try { + histogram = getMetricRegistry().histogram(metadata, tags); + } catch (IllegalArgumentException e) { + // Looks like we lost the registration race + histogram = getHistogram(tags); + Objects.requireNonNull(histogram); + } + } + return histogram; + } + + @SuppressWarnings("unchecked") + protected Gauge getGauge(Tag... tags) { + MetricID metricID = new MetricID(name(), tags); + return (Gauge) getMetricRegistry().getMetrics().get(metricID); + } + + @SuppressWarnings("unchecked") + static T getMetric(Method method, String name) { + MetricID metricID = new MetricID(String.format(METRIC_NAME_TEMPLATE, + method.getDeclaringClass().getName(), + method.getName(), name)); + return (T) getMetricRegistry().getMetrics().get(metricID); + } + + static Counter getCounter(Method method, String name) { + return getMetric(method, name); + } + + static Histogram getHistogram(Method method, String name) { + return getMetric(method, name); + } + + @SuppressWarnings("unchecked") + static Gauge getGauge(Method method, String name) { + return getMetric(method, name); + } + + static long getCounter(Object bean, String methodName, String name, + Class... params) throws Exception { + Method method = findMethod(getRealClass(bean), methodName, params); + return getCounter(method, name).getCount(); + } + + static Histogram getHistogram(Object bean, String methodName, String name, + Class... params) throws Exception { + Method method = findMethod(getRealClass(bean), methodName, params); + return getHistogram(method, name); + } + + static Gauge getGauge(Object bean, String methodName, String name, + Class... params) throws Exception { + Method method = findMethod(getRealClass(bean), methodName, params); + return getGauge(method, name); + } + + /** + * Attempts to find a method even if not accessible. + * + * @param beanClass bean class. + * @param methodName name of method. + * @param params param types. + * @return method found. + * @throws NoSuchMethodException if not found. + */ + private static Method findMethod(Class beanClass, String methodName, + Class... params) throws NoSuchMethodException { + try { + Method method = beanClass.getDeclaredMethod(methodName, params); + method.setAccessible(true); + return method; + } catch (Exception e) { + return beanClass.getMethod(methodName, params); + } + } + + @SuppressWarnings("unchecked") + protected Gauge registerGauge(Gauge newGauge, Tag... tags) { + Gauge gauge = getGauge(tags); + if (gauge == null) { + Metadata metadata = Metadata.builder() + .withName(name()) + .withDisplayName(name()) + .withDescription(description()) + .withType(metricType()) + .withUnit(unit()) + .build(); + try { + gauge = getMetricRegistry().register(metadata, newGauge, tags); + } catch (IllegalArgumentException e) { + // Looks like we lost the registration race + gauge = getGauge(tags); + Objects.requireNonNull(gauge); + } + } + return gauge; + } } - static Histogram getHistogram(Method method, String name) { - return getMetric(method, name); + // -- Invocations --------------------------------------------------------- + + enum InvocationResult implements Supplier { + VALUE_RETURNED("valueReturned"), + EXCEPTION_THROWN("exceptionThrown"); + + private final Tag metricTag; + + InvocationResult(String value) { + metricTag = new Tag("result", value); + } + + @Override + public Tag get() { + return metricTag; + } } - @SuppressWarnings("unchecked") - static Gauge getGauge(Method method, String name) { - return getMetric(method, name); + enum InvocationFallback implements Supplier { + APPLIED("applied"), + NOT_APPLIED("notApplied"), + NOT_DEFINED("notDefined"); + + private final Tag metricTag; + + InvocationFallback(String value) { + metricTag = new Tag("fallback", value); + } + + @Override + public Tag get() { + return metricTag; + } } - static long getCounter(Object bean, String methodName, String name, - Class... params) throws Exception { - Method method = findMethod(getRealClass(bean), methodName, params); - return getCounter(method, name).getCount(); + /** + * Class for "ft.invocations.total" counters. + */ + static class InvocationsTotal extends FaultToleranceMetric { + + static final InvocationsTotal INSTANCE = new InvocationsTotal(); + + private InvocationsTotal() { + } + + @Override + String name() { + return "ft.invocations.total"; + } + + @Override + String description() { + return "The number of times the method was called"; + } + + @Override + MetricType metricType() { + return MetricType.COUNTER; + } + + @Override + String unit() { + return MetricUnits.NONE; + } + + static Counter get(Tag... tags) { + return INSTANCE.registerCounter(tags); + } + + static Counter register(Tag... tags) { + return INSTANCE.registerCounter(tags); + } } - static Histogram getHistogram(Object bean, String methodName, String name, - Class... params) throws Exception { - Method method = findMethod(getRealClass(bean), methodName, params); - return getHistogram(method, name); + // -- Retries ------------------------------------------------------------- + + enum RetryResult implements Supplier { + VALUE_RETURNED("valueReturned"), + EXCEPTION_NOT_RETRYABLE("exceptionNotRetryable"), + MAX_RETRIES_REACHED("maxRetriesReached"), + MAX_DURATION_REACHED("maxDurationReached"); + + private final Tag metricTag; + + RetryResult(String value) { + metricTag = new Tag("retryResult", value); + } + + @Override + public Tag get() { + return metricTag; + } } - static Gauge getGauge(Object bean, String methodName, String name, - Class... params) throws Exception { - Method method = findMethod(getRealClass(bean), methodName, params); - return getGauge(method, name); + enum RetryRetried implements Supplier { + TRUE("true"), + FALSE("false"); + + private final Tag metricTag; + + RetryRetried(String value) { + metricTag = new Tag("retried", value); + } + + @Override + public Tag get() { + return metricTag; + } } /** - * Attempts to find a method even if not accessible. - * - * @param beanClass bean class. - * @param methodName name of method. - * @param params param types. - * @return method found. - * @throws NoSuchMethodException if not found. + * Class for "ft.retry.calls.total" counters. */ - private static Method findMethod(Class beanClass, String methodName, - Class... params) throws NoSuchMethodException { - try { - Method method = beanClass.getDeclaredMethod(methodName, params); - method.setAccessible(true); - return method; - } catch (Exception e) { - return beanClass.getMethod(methodName, params); + static class RetryCallsTotal extends FaultToleranceMetric { + + static final RetryCallsTotal INSTANCE = new RetryCallsTotal(); + + private RetryCallsTotal() { + } + + @Override + String name() { + return "ft.retry.calls.total"; + } + + @Override + String description() { + return "The number of times the retry logic was run. This will always be once per method call."; + } + + @Override + MetricType metricType() { + return MetricType.COUNTER; } - } - // -- Global -------------------------------------------------------------- + @Override + String unit() { + return MetricUnits.NONE; + } + + static Counter get(Tag... tags) { + return INSTANCE.registerCounter(tags); + } - static final String INVOCATIONS_TOTAL = "invocations.total"; - static final String INVOCATIONS_FAILED_TOTAL = "invocations.failed.total"; + static Counter register(Tag... tags) { + return INSTANCE.registerCounter(tags); + } + } /** - * Register global method counters for a method. - * - * @param method The method. + * Class for "ft.retry.retries.total" counters. */ - static void registerMetrics(Method method) { - if (!isFaultToleranceMetricsEnabled()) { - return; - } - - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - INVOCATIONS_TOTAL), - "The number of times the method was called"); - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - INVOCATIONS_FAILED_TOTAL), - "The number of times the method was called and, " - + "after all Fault Tolerance actions had been processed, " - + "threw a Throwable"); - } + static class RetryRetriesTotal extends FaultToleranceMetric { - // -- Retry --------------------------------------------------------------- - - static final String RETRY_CALLS_SUCCEEDED_NOT_RETRIED_TOTAL = "retry.callsSucceededNotRetried.total"; - static final String RETRY_CALLS_SUCCEEDED_RETRIED_TOTAL = "retry.callsSucceededRetried.total"; - static final String RETRY_CALLS_FAILED_TOTAL = "retry.callsFailed.total"; - static final String RETRY_RETRIES_TOTAL = "retry.retries.total"; - - static void registerRetryMetrics(Method method) { - if (!isFaultToleranceMetricsEnabled()) { - return; - } - - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - RETRY_CALLS_SUCCEEDED_NOT_RETRIED_TOTAL), - "The number of times the method was called and succeeded without retrying"); - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - RETRY_CALLS_SUCCEEDED_RETRIED_TOTAL), - "The number of times the method was called and succeeded after retrying at least once"); - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - RETRY_CALLS_FAILED_TOTAL), - "The number of times the method was called and ultimately failed after retrying"); - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - RETRY_RETRIES_TOTAL), - "The total number of times the method was retried"); - } + static final RetryRetriesTotal INSTANCE = new RetryRetriesTotal(); + private RetryRetriesTotal() { + } - // -- Timeout --------------------------------------------------------------- - - static final String TIMEOUT_EXECUTION_DURATION = "timeout.executionDuration"; - static final String TIMEOUT_CALLS_TIMED_OUT_TOTAL = "timeout.callsTimedOut.total"; - static final String TIMEOUT_CALLS_NOT_TIMED_OUT_TOTAL = "timeout.callsNotTimedOut.total"; - - static void registerTimeoutMetrics(Method method) { - if (!isFaultToleranceMetricsEnabled()) { - return; - } - - registerHistogram( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - TIMEOUT_EXECUTION_DURATION), - "Histogram of execution times for the method"); - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - TIMEOUT_CALLS_TIMED_OUT_TOTAL), - "The number of times the method timed out"); - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - TIMEOUT_CALLS_NOT_TIMED_OUT_TOTAL), - "The number of times the method completed without timing out"); - } + @Override + String name() { + return "ft.retry.retries.total"; + } - // -- CircuitBreaker ----------------------------------------------------- - - static final String BREAKER_CALLS_SUCCEEDED_TOTAL = "circuitbreaker.callsSucceeded.total"; - static final String BREAKER_CALLS_FAILED_TOTAL = "circuitbreaker.callsFailed.total"; - static final String BREAKER_CALLS_PREVENTED_TOTAL = "circuitbreaker.callsPrevented.total"; - static final String BREAKER_OPENED_TOTAL = "circuitbreaker.opened.total"; - - static final String BREAKER_OPEN_TOTAL = "circuitbreaker.open.total"; - static final String BREAKER_CLOSED_TOTAL = "circuitbreaker.closed.total"; - static final String BREAKER_HALF_OPEN_TOTAL = "circuitbreaker.halfOpen.total"; - - static void registerCircuitBreakerMetrics(Method method) { - if (!isFaultToleranceMetricsEnabled()) { - return; - } - - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - BREAKER_CALLS_SUCCEEDED_TOTAL), - "Number of calls allowed to run by the circuit breaker that " - + "returned successfully"); - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - BREAKER_CALLS_FAILED_TOTAL), - "Number of calls allowed to run by the circuit breaker that then failed"); - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - BREAKER_CALLS_PREVENTED_TOTAL), - "Number of calls prevented from running by an open circuit breaker"); - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - BREAKER_OPENED_TOTAL), - "Number of times the circuit breaker has moved from closed state to open state"); - } + @Override + String description() { + return "The number of times the method was retried"; + } - // -- Fallback ----------------------------------------------------------- + @Override + MetricType metricType() { + return MetricType.COUNTER; + } - static final String FALLBACK_CALLS_TOTAL = "fallback.calls.total"; + @Override + String unit() { + return MetricUnits.NONE; + } - static void registerFallbackMetrics(Method method) { - if (!isFaultToleranceMetricsEnabled()) { - return; + static Counter get(Tag... tags) { + return INSTANCE.registerCounter(tags); } - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - FALLBACK_CALLS_TOTAL), - "Number of times the fallback handler or method was called"); + static Counter register(Tag... tags) { + return INSTANCE.registerCounter(tags); + } } - // -- Bulkhead ----------------------------------------------------------- - - static final String BULKHEAD_CONCURRENT_EXECUTIONS = "bulkhead.concurrentExecutions"; - static final String BULKHEAD_CALLS_ACCEPTED_TOTAL = "bulkhead.callsAccepted.total"; - static final String BULKHEAD_CALLS_REJECTED_TOTAL = "bulkhead.callsRejected.total"; - static final String BULKHEAD_EXECUTION_DURATION = "bulkhead.executionDuration"; - static final String BULKHEAD_WAITING_QUEUE_POPULATION = "bulkhead.waitingQueue.population"; - static final String BULKHEAD_WAITING_DURATION = "bulkhead.waiting.duration"; - - static void registerBulkheadMetrics(Method method) { - if (!isFaultToleranceMetricsEnabled()) { - return; - } - - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - BULKHEAD_CALLS_ACCEPTED_TOTAL), - "Number of calls accepted by the bulkhead"); - registerCounter( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - BULKHEAD_CALLS_REJECTED_TOTAL), - "Number of calls rejected by the bulkhead"); - registerHistogram( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - BULKHEAD_EXECUTION_DURATION), - "Histogram of method execution times. This does not include any " - + "time spent waiting in the bulkhead queue."); - } + // -- Timeouts ------------------------------------------------------------ + + enum TimeoutTimedOut implements Supplier { + TRUE("true"), + FALSE("false"); + + private final Tag metricTag; + + TimeoutTimedOut(String value) { + this.metricTag = new Tag("timedOut", value); + } - // -- Utility methods ---------------------------------------------------- + public Tag get() { + return metricTag; + } + } /** - * Register a single counter. - * - * @param name Name of counter. - * @param description Description of counter. - * @return The counter created. + * Class for "ft.timeout.calls.total" counters. */ - private static Counter registerCounter(String name, String description) { - return getMetricRegistry().counter( - newMetadata(name, name, description, MetricType.COUNTER, MetricUnits.NONE, - true)); + static class TimeoutCallsTotal extends FaultToleranceMetric { + + static final TimeoutCallsTotal INSTANCE = new TimeoutCallsTotal(); + + private TimeoutCallsTotal() { + } + + @Override + String name() { + return "ft.timeout.calls.total"; + } + + @Override + String description() { + return "The number of times the timeout logic was run. This will usually be once " + + "per method call, but may be zero times if the circuit breaker prevents " + + "execution or more than once if the method is retried."; + } + + @Override + MetricType metricType() { + return MetricType.COUNTER; + } + + @Override + String unit() { + return MetricUnits.NONE; + } + + static Counter get(Tag... tags) { + return INSTANCE.registerCounter(tags); + } + + static Counter register(Tag... tags) { + return INSTANCE.registerCounter(tags); + } } /** - * Register a histogram with nanos as unit. - * - * @param name Name of histogram. - * @param description Description of histogram. - * @return The histogram created. + * Class for "ft.timeout.executionDuration" histograms. */ - static Histogram registerHistogram(String name, String description) { - return getMetricRegistry().histogram( - newMetadata(name, name, description, MetricType.HISTOGRAM, MetricUnits.NANOSECONDS, - true)); + static class TimeoutExecutionDuration extends FaultToleranceMetric { + + static final TimeoutExecutionDuration INSTANCE = new TimeoutExecutionDuration(); + + private TimeoutExecutionDuration() { + } + + @Override + String name() { + return "ft.timeout.executionDuration"; + } + + @Override + String description() { + return "Histogram of execution times for the method"; + } + + @Override + MetricType metricType() { + return MetricType.HISTOGRAM; + } + + @Override + String unit() { + return MetricUnits.NANOSECONDS; + } + + static Histogram get(Tag... tags) { + return INSTANCE.registerHistogram(tags); + } + + static Histogram register(Tag... tags) { + return INSTANCE.registerHistogram(tags); + } + } + + // --- CircuitBreakers ---------------------------------------------------- + + enum CircuitBreakerResult implements Supplier { + SUCCESS("success"), + FAILURE("failure"), + CIRCUIT_BREAKER_OPEN("circuitBreakerOpen"); + + private final Tag metricTag; + + CircuitBreakerResult(String value) { + metricTag = new Tag("circuitBreakerResult", value); + } + + @Override + public Tag get() { + return metricTag; + } + } + + enum CircuitBreakerState implements Supplier { + OPEN("open"), + CLOSED("closed"), + HALF_OPEN("halfOpen"); + + private final Tag metricTag; + + CircuitBreakerState(String value) { + metricTag = new Tag("state", value); + } + + @Override + public Tag get() { + return metricTag; + } } /** - * Register a gauge with nanos as unit. Checks if gauge is already registered - * using synchronization. - * - * @param metricName Name of metric. - * @param description Description of gauge. - * @return The gauge created or existing if already created. + * Class for "ft.circuitbreaker.calls.total" counters. */ @SuppressWarnings("unchecked") static Gauge registerGauge(Method method, String metricName, String description, Gauge gauge) { LOCK.lock(); try { - MetricID metricID = newMetricID(String.format(METRIC_NAME_TEMPLATE, + MetricID metricID = new MetricID(String.format(METRIC_NAME_TEMPLATE, method.getDeclaringClass().getName(), method.getName(), metricName)); Gauge existing = getMetricRegistry().getGauges().get(metricID); if (existing == null) { - getMetricRegistry().register( - newMetadata(metricID.getName(), metricID.getName(), description, MetricType.GAUGE, - MetricUnits.NANOSECONDS, true), + getMetricRegistry().register(Metadata.builder() + .withName(metricID.getName()) + .withDisplayName(metricID.getName()) + .withDescription(description) + .withType(MetricType.GAUGE) + .withUnit(MetricUnits.NANOSECONDS).build(), gauge); } return existing; @@ -369,19 +574,334 @@ static Gauge registerGauge(Method method, String metricName, String descr } } - private static MetricID newMetricID(String name) { - return new MetricID(name); + static class CircuitBreakerCallsTotal extends FaultToleranceMetric { + + static final CircuitBreakerCallsTotal INSTANCE = new CircuitBreakerCallsTotal(); + + private CircuitBreakerCallsTotal() { + } + + @Override + String name() { + return "ft.circuitbreaker.calls.total"; + } + + @Override + String description() { + return "The number of times the circuit breaker logic was run. This will usually be once " + + "per method call, but may be more than once if the method call is retried."; + } + + @Override + MetricType metricType() { + return MetricType.COUNTER; + } + + @Override + String unit() { + return MetricUnits.NONE; + } + + static Counter get(Tag... tags) { + return INSTANCE.registerCounter(tags); + } + + static Counter register(Tag... tags) { + return INSTANCE.registerCounter(tags); + } + } + + /** + * Class for "ft.circuitbreaker.state.total" gauges. + */ + static class CircuitBreakerStateTotal extends FaultToleranceMetric { + + static final CircuitBreakerStateTotal INSTANCE = new CircuitBreakerStateTotal(); + + private CircuitBreakerStateTotal() { + } + + @Override + String name() { + return "ft.circuitbreaker.state.total"; + } + + @Override + String description() { + return "Amount of time the circuit breaker has spent in each state"; + } + + @Override + MetricType metricType() { + return MetricType.GAUGE; + } + + @Override + String unit() { + return MetricUnits.NANOSECONDS; + } + + + static Gauge get(Tag... tags) { + return INSTANCE.getGauge(tags); + } + + static Gauge register(Gauge gauge, Tag... tags) { + return INSTANCE.registerGauge(gauge, tags); + } + } + + /** + * Class for "ft.circuitbreaker.opened.total" counters. + */ + static class CircuitBreakerOpenedTotal extends FaultToleranceMetric { + + static final CircuitBreakerOpenedTotal INSTANCE = new CircuitBreakerOpenedTotal(); + + private CircuitBreakerOpenedTotal() { + } + + @Override + String name() { + return "ft.circuitbreaker.opened.total"; + } + + @Override + String description() { + return "Number of times the circuit breaker has moved from closed state to open state"; + } + + @Override + MetricType metricType() { + return MetricType.COUNTER; + } + + @Override + String unit() { + return MetricUnits.NONE; + } + + static Counter get(Tag... tags) { + return INSTANCE.registerCounter(tags); + } + + static Counter register(Tag... tags) { + return INSTANCE.registerCounter(tags); + } + } + + // --- Bulkheads ---------------------------------------------------------- + + enum BulkheadResult implements Supplier { + ACCEPTED("accepted"), + REJECTED("rejected"); + + private final Tag metricTag; + + BulkheadResult(String value) { + metricTag = new Tag("bulkheadResult", value); + } + + @Override + public Tag get() { + return metricTag; + } + } + + /** + * Class for "ft.bulkhead.calls.total" counters. + */ + static class BulkheadCallsTotal extends FaultToleranceMetric { + + static final BulkheadCallsTotal INSTANCE = new BulkheadCallsTotal(); + + private BulkheadCallsTotal() { + } + + @Override + String name() { + return "ft.bulkhead.calls.total"; + } + + @Override + String description() { + return "The number of times the bulkhead logic was run. This will usually be once per " + + "method call, but may be zero times if the circuit breaker prevented execution " + + "or more than once if the method call is retried."; + } + + @Override + MetricType metricType() { + return MetricType.COUNTER; + } + + @Override + String unit() { + return MetricUnits.NONE; + } + + static Counter get(Tag... tags) { + return INSTANCE.registerCounter(tags); + } + + static Counter register(Tag... tags) { + return INSTANCE.registerCounter(tags); + } + } + + /** + * Class for "ft.bulkhead.executionsRunning" gauges. + */ + static class BulkheadExecutionsRunning extends FaultToleranceMetric { + + static final BulkheadExecutionsRunning INSTANCE = new BulkheadExecutionsRunning(); + + private BulkheadExecutionsRunning() { + } + + @Override + String name() { + return "ft.bulkhead.executionsRunning"; + } + + @Override + String description() { + return "Number of currently running executions"; + } + + @Override + MetricType metricType() { + return MetricType.GAUGE; + } + + @Override + String unit() { + return MetricUnits.NONE; + } + + static Gauge get(Tag... tags) { + return INSTANCE.getGauge(tags); + } + + static Gauge register(Gauge gauge, Tag... tags) { + return INSTANCE.registerGauge(gauge, tags); + } } - // TODO 3.0.0-JAKARTA - private static Metadata newMetadata(String name, String displayName, String description, MetricType metricType, - String metricUnits, boolean isReusable) { - return Metadata.builder() - .withName(name) - .withDisplayName(displayName) - .withDescription(description) - .withType(metricType) - .withUnit(metricUnits) - .build(); + /** + * Class for "ft.bulkhead.executionsWaiting" gauges. + */ + static class BulkheadExecutionsWaiting extends FaultToleranceMetric { + + static final BulkheadExecutionsWaiting INSTANCE = new BulkheadExecutionsWaiting(); + + private BulkheadExecutionsWaiting() { + } + + @Override + String name() { + return "ft.bulkhead.executionsWaiting"; + } + + @Override + String description() { + return "Number of executions currently waiting in the queue"; + } + + @Override + MetricType metricType() { + return MetricType.GAUGE; + } + + @Override + String unit() { + return MetricUnits.NONE; + } + + static Gauge get(Tag... tags) { + return INSTANCE.getGauge(tags); + } + + static Gauge register(Gauge gauge, Tag... tags) { + return INSTANCE.registerGauge(gauge, tags); + } + } + + /** + * Class for "ft.bulkhead.runningDuration" histograms. + */ + static class BulkheadRunningDuration extends FaultToleranceMetric { + + static final BulkheadRunningDuration INSTANCE = new BulkheadRunningDuration(); + + private BulkheadRunningDuration() { + } + + @Override + String name() { + return "ft.bulkhead.runningDuration"; + } + + @Override + String description() { + return "Histogram of the time that method executions spent running"; + } + + @Override + MetricType metricType() { + return MetricType.HISTOGRAM; + } + + @Override + String unit() { + return MetricUnits.NANOSECONDS; + } + + static Histogram get(Tag... tags) { + return INSTANCE.registerHistogram(tags); + } + + static Histogram register(Tag... tags) { + return INSTANCE.registerHistogram(tags); + } + } + + /** + * Class for "ft.bulkhead.waitingDuration" histograms. + */ + static class BulkheadWaitingDuration extends FaultToleranceMetric { + + static final BulkheadWaitingDuration INSTANCE = new BulkheadWaitingDuration(); + + private BulkheadWaitingDuration() { + } + + @Override + String name() { + return "ft.bulkhead.waitingDuration"; + } + + @Override + String description() { + return "Histogram of the time that method executions spent waiting in the queue"; + } + + @Override + MetricType metricType() { + return MetricType.HISTOGRAM; + } + + @Override + String unit() { + return MetricUnits.NANOSECONDS; + } + + static Histogram get(Tag... tags) { + return INSTANCE.registerHistogram(tags); + } + + static Histogram register(Tag... tags) { + return INSTANCE.registerHistogram(tags); + } } } diff --git a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/JavaMethodFinder.java b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/JavaMethodFinder.java index 11a1bee5c92..bfc14f30007 100644 --- a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/JavaMethodFinder.java +++ b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/JavaMethodFinder.java @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019 Oracle and/or its affiliates. + * Copyright (c) 2019, 2021 Oracle and/or its affiliates. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -45,7 +45,7 @@ private JavaMethodFinder() { * @return The method found. * @throws NoSuchMethodException If not found. */ - static Method findMethod(Class clazz, String methodName, Type[] paramTypes) + static Method findMethod(Class clazz, String methodName, Type... paramTypes) throws NoSuchMethodException { // Initialize queue with first class Queue> queue = new LinkedBlockingQueue<>(); diff --git a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/MethodIntrospector.java b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/MethodIntrospector.java index 9ad99c8001a..d353151a776 100644 --- a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/MethodIntrospector.java +++ b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/MethodIntrospector.java @@ -27,7 +27,11 @@ import org.eclipse.microprofile.faulttolerance.Fallback; import org.eclipse.microprofile.faulttolerance.Retry; import org.eclipse.microprofile.faulttolerance.Timeout; +import org.eclipse.microprofile.metrics.Tag; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.InvocationFallback.APPLIED; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.InvocationFallback.NOT_APPLIED; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.InvocationFallback.NOT_DEFINED; import static io.helidon.microprofile.faulttolerance.FaultToleranceParameter.getParameter; import static io.helidon.microprofile.faulttolerance.MethodAntn.lookupAnnotation; @@ -50,6 +54,8 @@ class MethodIntrospector { private final Bulkhead bulkhead; + private Tag methodNameTag; + /** * Constructor. * @@ -150,6 +156,30 @@ Bulkhead getBulkhead() { return bulkhead; } + /** + * Returns a metric's tag with the fully qualified method name. + * + * @return the tag + */ + Tag getMethodNameTag() { + if (methodNameTag == null) { + String name = method.getDeclaringClass().getName() + "." + method.getName(); + methodNameTag = new Tag("method", name); + } + return methodNameTag; + } + + /** + * Returns a fallback tag based on the {@code fallbackCalled} parameter. + * + * @param fallbackCalled indicates if fallback logic was called or not + * @return the tag + */ + Tag getFallbackTag(boolean fallbackCalled) { + return !hasFallback() ? NOT_DEFINED.get() + : fallbackCalled ? APPLIED.get() : NOT_APPLIED.get(); + } + /** * Determines if annotation type is present and enabled. * diff --git a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/MethodInvoker.java b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/MethodInvoker.java index f63cfce0d3b..0fa72f8a2df 100644 --- a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/MethodInvoker.java +++ b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/MethodInvoker.java @@ -1,5 +1,5 @@ /* - * Copyright (c) 2021 Oracle and/or its affiliates. + * Copyright (c) 2020, 2021 Oracle and/or its affiliates. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -17,6 +17,7 @@ import java.lang.reflect.Method; import java.time.Duration; +import java.util.Arrays; import java.util.Objects; import java.util.concurrent.CancellationException; import java.util.concurrent.CompletableFuture; @@ -40,45 +41,38 @@ import io.helidon.faulttolerance.FaultTolerance; import io.helidon.faulttolerance.FtHandlerTyped; import io.helidon.faulttolerance.Retry; +import io.helidon.faulttolerance.RetryTimeoutException; import io.helidon.faulttolerance.Timeout; import jakarta.interceptor.InvocationContext; import org.eclipse.microprofile.faulttolerance.exceptions.BulkheadException; import org.eclipse.microprofile.faulttolerance.exceptions.CircuitBreakerOpenException; -import org.eclipse.microprofile.faulttolerance.exceptions.TimeoutException; import org.eclipse.microprofile.metrics.Counter; import static io.helidon.microprofile.faulttolerance.FaultToleranceExtension.isFaultToleranceMetricsEnabled; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BREAKER_CALLS_FAILED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BREAKER_CALLS_PREVENTED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BREAKER_CALLS_SUCCEEDED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BREAKER_CLOSED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BREAKER_HALF_OPEN_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BREAKER_OPENED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BREAKER_OPEN_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BULKHEAD_CALLS_ACCEPTED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BULKHEAD_CALLS_REJECTED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BULKHEAD_CONCURRENT_EXECUTIONS; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BULKHEAD_EXECUTION_DURATION; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BULKHEAD_WAITING_DURATION; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BULKHEAD_WAITING_QUEUE_POPULATION; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.INVOCATIONS_FAILED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.INVOCATIONS_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.METRIC_NAME_TEMPLATE; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.RETRY_CALLS_FAILED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.RETRY_CALLS_SUCCEEDED_NOT_RETRIED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.RETRY_CALLS_SUCCEEDED_RETRIED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.RETRY_RETRIES_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.TIMEOUT_CALLS_NOT_TIMED_OUT_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.TIMEOUT_CALLS_TIMED_OUT_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.TIMEOUT_EXECUTION_DURATION; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.getCounter; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.getHistogram; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.registerGauge; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.registerHistogram; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BulkheadCallsTotal; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BulkheadExecutionsRunning; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BulkheadExecutionsWaiting; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BulkheadResult; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BulkheadRunningDuration; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BulkheadWaitingDuration; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.CircuitBreakerCallsTotal; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.CircuitBreakerOpenedTotal; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.CircuitBreakerResult; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.CircuitBreakerState; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.CircuitBreakerStateTotal; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.InvocationResult.EXCEPTION_THROWN; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.InvocationResult.VALUE_RETURNED; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.InvocationsTotal; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.RetryCallsTotal; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.RetryResult; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.RetryRetried; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.RetryRetriesTotal; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.TimeoutCallsTotal; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.TimeoutExecutionDuration; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.TimeoutTimedOut; import static io.helidon.microprofile.faulttolerance.ThrowableMapper.map; import static io.helidon.microprofile.faulttolerance.ThrowableMapper.mapTypes; - /** * Invokes a FT method applying semantics based on method annotations. An instance * of this class is created for each method invocation. Some state is shared across @@ -134,6 +128,12 @@ class MethodInvoker implements FtSupplier { */ private Thread asyncInterruptThread; + /** + * A boolean value indicates whether the fallback logic was called or not + * on this invocation. + */ + private AtomicBoolean fallbackCalled = new AtomicBoolean(false); + /** * Helper to properly propagate active request scope to other threads. */ @@ -244,7 +244,7 @@ public T get() throws InterruptedException, ExecutionException { @Override public T get(long timeout, TimeUnit unit) throws InterruptedException, ExecutionException, java.util.concurrent.TimeoutException { - T value = super.get(); + T value = super.get(timeout, unit); if (method.getReturnType() == Future.class) { return ((Future) value).get(timeout, unit); } @@ -301,36 +301,41 @@ public boolean cancel(boolean mayInterruptIfRunning) { requestScopeHelper = new RequestScopeHelper(); requestScopeHelper.saveScope(); - // Gauges and other metrics for bulkhead and circuit breakers - if (isFaultToleranceMetricsEnabled()) { - if (introspector.hasCircuitBreaker()) { - registerGauge(method, BREAKER_OPEN_TOTAL, - "Amount of time the circuit breaker has spent in open state", - () -> methodState.breakerTimerOpen); - registerGauge(method, BREAKER_HALF_OPEN_TOTAL, - "Amount of time the circuit breaker has spent in half-open state", - () -> methodState.breakerTimerHalfOpen); - registerGauge(method, BREAKER_CLOSED_TOTAL, - "Amount of time the circuit breaker has spent in closed state", - () -> methodState.breakerTimerClosed); - } - if (introspector.hasBulkhead()) { - registerGauge(method, BULKHEAD_CONCURRENT_EXECUTIONS, - "Number of currently running executions", - () -> methodState.bulkhead.stats().concurrentExecutions()); - if (introspector.isAsynchronous()) { - registerGauge(method, BULKHEAD_WAITING_QUEUE_POPULATION, - "Number of executions currently waiting in the queue", - () -> methodState.bulkhead.stats().waitingQueueSize()); - registerHistogram( - String.format(METRIC_NAME_TEMPLATE, - method.getDeclaringClass().getName(), - method.getName(), - BULKHEAD_WAITING_DURATION), - "Histogram of the time executions spend waiting in the queue."); - } + registerMetrics(); + } + + private void registerMetrics() { + if (!isFaultToleranceMetricsEnabled()) { + return; + } + + if (introspector.hasCircuitBreaker()) { + CircuitBreakerStateTotal.register( + () -> methodState.breakerTimerOpen, + introspector.getMethodNameTag(), + CircuitBreakerState.OPEN.get()); + CircuitBreakerStateTotal.register( + () -> methodState.breakerTimerHalfOpen, + introspector.getMethodNameTag(), + CircuitBreakerState.HALF_OPEN.get()); + CircuitBreakerStateTotal.register( + () -> methodState.breakerTimerClosed, + introspector.getMethodNameTag(), + CircuitBreakerState.CLOSED.get()); + CircuitBreakerOpenedTotal.register( + introspector.getMethodNameTag()); + } + if (introspector.hasBulkhead()) { + BulkheadExecutionsRunning.register( + () -> methodState.bulkhead.stats().concurrentExecutions(), + introspector.getMethodNameTag()); + if (introspector.isAsynchronous()) { + BulkheadExecutionsWaiting.register( + () -> methodState.bulkhead.stats().waitingQueueSize(), + introspector.getMethodNameTag()); } } + } @Override @@ -400,7 +405,8 @@ public Object get() throws Throwable { cause = map(throwable); } updateMetricsAfter(cause); - resultFuture.completeExceptionally(cause); + resultFuture.completeExceptionally(cause instanceof RetryTimeoutException + ? ((RetryTimeoutException) cause).lastRetryException() : cause); } else { updateMetricsAfter(null); resultFuture.complete(result); @@ -433,6 +439,9 @@ public Object get() throws Throwable { requestScopeHelper.clearScope(); } updateMetricsAfter(cause); + if (cause instanceof RetryTimeoutException) { + throw ((RetryTimeoutException) cause).lastRetryException(); + } if (cause != null) { throw cause; } @@ -473,22 +482,6 @@ private void initMethodHandler(MethodState methodState) { .skipOn(mapTypes(introspector.getCircuitBreaker().skipOn())) .build(); } - - if (introspector.hasRetry()) { - methodState.retry = Retry.builder() - .retryPolicy(Retry.JitterRetryPolicy.builder() - .calls(introspector.getRetry().maxRetries() + 1) - .delay(Duration.of(introspector.getRetry().delay(), - introspector.getRetry().delayUnit())) - .jitter(Duration.of(introspector.getRetry().jitter(), - introspector.getRetry().jitterDelayUnit())) - .build()) - .overallTimeout(Duration.of(introspector.getRetry().maxDuration(), - introspector.getRetry().durationUnit())) - .applyOn(mapTypes(introspector.getRetry().retryOn())) - .skipOn(mapTypes(introspector.getRetry().abortOn())) - .build(); - } } /** @@ -516,7 +509,27 @@ private FtHandlerTyped createMethodHandler(MethodState methodState) { builder.addBreaker(methodState.breaker); } - if (methodState.retry != null) { + // Create a retry for this invocation only + if (introspector.hasRetry()) { + int maxRetries = introspector.getRetry().maxRetries(); + if (maxRetries == -1) { + maxRetries = Integer.MAX_VALUE; + } else { + maxRetries++; // add 1 for initial call + } + methodState.retry = Retry.builder() + .retryPolicy(Retry.JitterRetryPolicy.builder() + .calls(maxRetries) + .delay(Duration.of(introspector.getRetry().delay(), + introspector.getRetry().delayUnit())) + .jitter(Duration.of(introspector.getRetry().jitter(), + introspector.getRetry().jitterDelayUnit())) + .build()) + .overallTimeout(Duration.of(introspector.getRetry().maxDuration(), + introspector.getRetry().durationUnit())) + .applyOn(mapTypes(introspector.getRetry().retryOn())) + .skipOn(mapTypes(introspector.getRetry().abortOn())) + .build(); builder.addRetry(methodState.retry); } @@ -524,8 +537,8 @@ private FtHandlerTyped createMethodHandler(MethodState methodState) { if (introspector.hasFallback()) { Fallback fallback = Fallback.builder() .fallback(throwable -> { - // Execute callback logic - CommandFallback cfb = new CommandFallback(context, introspector, throwable); + fallbackCalled.set(true); + FallbackHelper cfb = new FallbackHelper(context, introspector, throwable); return toCompletionStageSupplier(cfb::execute).get(); }) .applyOn(mapTypes(introspector.getFallback().applyOn())) @@ -635,7 +648,7 @@ private void updateMetricsBefore() { /** * Update metrics after method is called and depending on outcome. * - * @param cause Exception cause or {@code null} if execution successful. + * @param cause Mapped cause or {@code null} if successful. */ private void updateMetricsAfter(Throwable cause) { if (!isFaultToleranceMetricsEnabled()) { @@ -647,58 +660,83 @@ private void updateMetricsAfter(Throwable cause) { // Calculate execution time long executionTime = System.nanoTime() - handlerStartNanos; - // Metrics for retries + // Retries if (introspector.hasRetry()) { - // Have retried the last call? - long newValue = methodState.retry.retryCounter(); - if (updateCounter(method, RETRY_RETRIES_TOTAL, newValue)) { - if (cause == null) { - getCounter(method, RETRY_CALLS_SUCCEEDED_RETRIED_TOTAL).inc(); - } - } else { - getCounter(method, RETRY_CALLS_SUCCEEDED_NOT_RETRIED_TOTAL).inc(); + long retryCounter = methodState.retry.retryCounter(); + boolean wasRetried = retryCounter > 0; + Counter retryRetriesTotal = RetryRetriesTotal.get(introspector.getMethodNameTag()); + + // Update retry counter + if (wasRetried) { + retryRetriesTotal.inc(retryCounter); } - // Update failed calls - if (cause != null) { - getCounter(method, RETRY_CALLS_FAILED_TOTAL).inc(); + // Update retry metrics based on outcome + if (cause == null) { + RetryCallsTotal.get(introspector.getMethodNameTag(), + wasRetried ? RetryRetried.TRUE.get() : RetryRetried.FALSE.get(), + RetryResult.VALUE_RETURNED.get()).inc(); + } else if (cause instanceof RetryTimeoutException) { + RetryCallsTotal.get(introspector.getMethodNameTag(), + wasRetried ? RetryRetried.TRUE.get() : RetryRetried.FALSE.get(), + RetryResult.MAX_DURATION_REACHED.get()).inc(); + } else { + // Exception thrown but not RetryTimeoutException + int maxRetries = introspector.getRetry().maxRetries(); + if (maxRetries == -1) { + maxRetries = Integer.MAX_VALUE; + } + if (retryCounter == maxRetries) { + RetryCallsTotal.get(introspector.getMethodNameTag(), + wasRetried ? RetryRetried.TRUE.get() : RetryRetried.FALSE.get(), + RetryResult.MAX_RETRIES_REACHED.get()).inc(); + } else if (retryCounter < maxRetries) { + RetryCallsTotal.get(introspector.getMethodNameTag(), + wasRetried ? RetryRetried.TRUE.get() : RetryRetried.FALSE.get(), + RetryResult.EXCEPTION_NOT_RETRYABLE.get()).inc(); + } } } // Timeout if (introspector.hasTimeout()) { - getHistogram(method, TIMEOUT_EXECUTION_DURATION).update(executionTime); - getCounter(method, cause instanceof TimeoutException - ? TIMEOUT_CALLS_TIMED_OUT_TOTAL - : TIMEOUT_CALLS_NOT_TIMED_OUT_TOTAL).inc(); + if (cause instanceof org.eclipse.microprofile.faulttolerance.exceptions.TimeoutException) { + TimeoutCallsTotal.get(introspector.getMethodNameTag(), + TimeoutTimedOut.TRUE.get()).inc(); + } else { + TimeoutCallsTotal.get(introspector.getMethodNameTag(), + TimeoutTimedOut.FALSE.get()).inc(); + } + TimeoutExecutionDuration.get(introspector.getMethodNameTag()).update(executionTime); } - // Circuit breaker + // CircuitBreaker if (introspector.hasCircuitBreaker()) { Objects.requireNonNull(methodState.breaker); - // Update counters based on state changes if (methodState.lastBreakerState == State.OPEN) { - getCounter(method, BREAKER_CALLS_PREVENTED_TOTAL).inc(); + CircuitBreakerCallsTotal.get(introspector.getMethodNameTag(), + CircuitBreakerResult.CIRCUIT_BREAKER_OPEN.get()).inc(); } else if (methodState.breaker.state() == State.OPEN) { // closed -> open - getCounter(method, BREAKER_OPENED_TOTAL).inc(); + CircuitBreakerOpenedTotal.get(introspector.getMethodNameTag()).inc(); } - // Update succeeded and failed if (cause == null) { - getCounter(method, BREAKER_CALLS_SUCCEEDED_TOTAL).inc(); + CircuitBreakerCallsTotal.get(introspector.getMethodNameTag(), + CircuitBreakerResult.SUCCESS.get()).inc(); } else if (!(cause instanceof CircuitBreakerOpenException)) { - boolean failure = false; - Class[] failOn = introspector.getCircuitBreaker().failOn(); - for (Class c : failOn) { - if (c.isAssignableFrom(cause.getClass())) { - failure = true; - break; - } + boolean skipOnThrowable = Arrays.stream(introspector.getCircuitBreaker().skipOn()) + .anyMatch(c -> c.isAssignableFrom(cause.getClass())); + boolean failOnThrowable = Arrays.stream(introspector.getCircuitBreaker().failOn()) + .anyMatch(c -> c.isAssignableFrom(cause.getClass())); + + if (skipOnThrowable || !failOnThrowable) { + CircuitBreakerCallsTotal.get(introspector.getMethodNameTag(), + CircuitBreakerResult.SUCCESS.get()).inc(); + } else { + CircuitBreakerCallsTotal.get(introspector.getMethodNameTag(), + CircuitBreakerResult.FAILURE.get()).inc(); } - - getCounter(method, failure ? BREAKER_CALLS_FAILED_TOTAL - : BREAKER_CALLS_SUCCEEDED_TOTAL).inc(); } // Update times for gauges @@ -725,44 +763,40 @@ private void updateMetricsAfter(Throwable cause) { if (introspector.hasBulkhead()) { Objects.requireNonNull(methodState.bulkhead); Bulkhead.Stats stats = methodState.bulkhead.stats(); - updateCounter(method, BULKHEAD_CALLS_ACCEPTED_TOTAL, stats.callsAccepted()); - updateCounter(method, BULKHEAD_CALLS_REJECTED_TOTAL, stats.callsRejected()); + Counter bulkheadAccepted = BulkheadCallsTotal.get(introspector.getMethodNameTag(), + BulkheadResult.ACCEPTED.get()); + if (stats.callsAccepted() > bulkheadAccepted.getCount()) { + bulkheadAccepted.inc(stats.callsAccepted() - bulkheadAccepted.getCount()); + } + Counter bulkheadRejected = BulkheadCallsTotal.get(introspector.getMethodNameTag(), + BulkheadResult.REJECTED.get()); + if (stats.callsRejected() > bulkheadRejected.getCount()) { + bulkheadRejected.inc(stats.callsRejected() - bulkheadRejected.getCount()); + } // Update histograms if task accepted if (!(cause instanceof BulkheadException)) { long waitingTime = invocationStartNanos - handlerStartNanos; - getHistogram(method, BULKHEAD_EXECUTION_DURATION).update(executionTime - waitingTime); + BulkheadRunningDuration.get(introspector.getMethodNameTag()) + .update(executionTime - waitingTime); if (introspector.isAsynchronous()) { - getHistogram(method, BULKHEAD_WAITING_DURATION).update(waitingTime); + BulkheadWaitingDuration.get(introspector.getMethodNameTag()).update(waitingTime); } } } // Global method counters - getCounter(method, INVOCATIONS_TOTAL).inc(); - if (cause != null) { - getCounter(method, INVOCATIONS_FAILED_TOTAL).inc(); + if (cause == null) { + InvocationsTotal.get(introspector.getMethodNameTag(), + VALUE_RETURNED.get(), + introspector.getFallbackTag(fallbackCalled.get())).inc(); + } else { + InvocationsTotal.get(introspector.getMethodNameTag(), + EXCEPTION_THROWN.get(), + introspector.getFallbackTag(fallbackCalled.get())).inc(); } } finally { methodState.lock.unlock(); } } - - /** - * Sets the value of a monotonically increasing counter using {@code inc()}. - * - * @param method The method. - * @param name The counter's name. - * @param newValue The new value. - * @return A value of {@code true} if counter updated, {@code false} otherwise. - */ - private static boolean updateCounter(Method method, String name, long newValue) { - Counter counter = getCounter(method, name); - long oldValue = counter.getCount(); - if (newValue > oldValue) { - counter.inc(newValue - oldValue); - return true; - } - return false; - } } diff --git a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/RetryAntn.java b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/RetryAntn.java index b210eda4020..4d35b3a0dc3 100644 --- a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/RetryAntn.java +++ b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/RetryAntn.java @@ -17,6 +17,7 @@ package io.helidon.microprofile.faulttolerance; import java.lang.reflect.Method; +import java.time.Duration; import java.time.temporal.ChronoUnit; import org.eclipse.microprofile.faulttolerance.Retry; @@ -47,7 +48,9 @@ public void validate() { throw new FaultToleranceDefinitionException("Invalid @Retry annotation, " + "delay must be >= 0"); } - if (maxDuration() < delay()) { + Duration delay = Duration.of(delay(), delayUnit()); + Duration maxDuration = Duration.of(maxDuration(), durationUnit()); + if (maxDuration.compareTo(delay) < 0) { throw new FaultToleranceDefinitionException("Invalid @Retry annotation, " + "maxDuration must be >= delay"); } diff --git a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/ThrowableMapper.java b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/ThrowableMapper.java index 154e69c7eee..32b0e74d1af 100644 --- a/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/ThrowableMapper.java +++ b/microprofile/fault-tolerance/src/main/java/io/helidon/microprofile/faulttolerance/ThrowableMapper.java @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020 Oracle and/or its affiliates. + * Copyright (c) 2021 Oracle and/or its affiliates. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -48,6 +48,9 @@ static Throwable map(Throwable t) { if (t instanceof io.helidon.faulttolerance.BulkheadException) { return new BulkheadException(t.getMessage(), t.getCause()); } + if (t instanceof io.helidon.faulttolerance.RetryTimeoutException) { + return t; // the cause is handled elsewhere + } if (t instanceof java.util.concurrent.TimeoutException) { return new TimeoutException(t.getMessage(), t.getCause()); } diff --git a/microprofile/fault-tolerance/src/test/java/io/helidon/microprofile/faulttolerance/FaultToleranceTest.java b/microprofile/fault-tolerance/src/test/java/io/helidon/microprofile/faulttolerance/FaultToleranceTest.java index 93d74b2d581..8260d01112a 100644 --- a/microprofile/fault-tolerance/src/test/java/io/helidon/microprofile/faulttolerance/FaultToleranceTest.java +++ b/microprofile/fault-tolerance/src/test/java/io/helidon/microprofile/faulttolerance/FaultToleranceTest.java @@ -1,5 +1,5 @@ /* - * Copyright (c) 2018, 2020 Oracle and/or its affiliates. + * Copyright (c) 2018, 2021 Oracle and/or its affiliates. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -31,15 +31,17 @@ import jakarta.enterprise.inject.literal.NamedLiteral; import jakarta.enterprise.inject.spi.CDI; +import org.eclipse.microprofile.metrics.Tag; import org.junit.jupiter.api.BeforeEach; +import static io.helidon.microprofile.faulttolerance.FaultToleranceExtension.getRealClass; import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.instanceOf; import static org.hamcrest.Matchers.is; import static org.junit.jupiter.api.Assertions.fail; /** - * Class FaultToleranceTest. + * Base class for FT tests. Mostly utility methods used by subclasses. */ @HelidonTest abstract class FaultToleranceTest { @@ -129,4 +131,8 @@ static void assertCompleteOk(CompletionStage future, String expectedMess fail("Unexpected exception" + e); } } + + static Tag getMethodTag(Object bean, String methodName) { + return new Tag("method", getRealClass(bean).getName() + "." + methodName); + } } diff --git a/microprofile/fault-tolerance/src/test/java/io/helidon/microprofile/faulttolerance/MetricsBean.java b/microprofile/fault-tolerance/src/test/java/io/helidon/microprofile/faulttolerance/MetricsBean.java index e472961c047..546e18b0803 100644 --- a/microprofile/fault-tolerance/src/test/java/io/helidon/microprofile/faulttolerance/MetricsBean.java +++ b/microprofile/fault-tolerance/src/test/java/io/helidon/microprofile/faulttolerance/MetricsBean.java @@ -31,14 +31,6 @@ import org.eclipse.microprofile.metrics.Counter; import org.eclipse.microprofile.metrics.annotation.Metric; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BULKHEAD_CONCURRENT_EXECUTIONS; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BULKHEAD_WAITING_QUEUE_POPULATION; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.getGauge; -import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.greaterThanOrEqualTo; -import static org.hamcrest.Matchers.is; -import static org.hamcrest.Matchers.not; - /** * A bean with methods that update metrics. */ @@ -171,9 +163,6 @@ String onFailure() { CompletableFuture concurrent(long sleepMillis) { FaultToleranceTest.printStatus("MetricsBean::concurrent()", "success"); try { - assertThat(getGauge(this, - "concurrent", - BULKHEAD_CONCURRENT_EXECUTIONS, long.class).getValue(), is(not(0))); Thread.sleep(sleepMillis); } catch (Exception e) { // falls through @@ -186,9 +175,6 @@ CompletableFuture concurrent(long sleepMillis) { CompletableFuture concurrentAsync(long sleepMillis) { FaultToleranceTest.printStatus("MetricsBean::concurrentAsync()", "success"); try { - assertThat((long) getGauge(this, "concurrentAsync", - BULKHEAD_WAITING_QUEUE_POPULATION, long.class).getValue(), - is(greaterThanOrEqualTo(0L))); Thread.sleep(sleepMillis); } catch (Exception e) { // falls through diff --git a/microprofile/fault-tolerance/src/test/java/io/helidon/microprofile/faulttolerance/MetricsTest.java b/microprofile/fault-tolerance/src/test/java/io/helidon/microprofile/faulttolerance/MetricsTest.java index 1d667855f3a..cd654cf7875 100644 --- a/microprofile/fault-tolerance/src/test/java/io/helidon/microprofile/faulttolerance/MetricsTest.java +++ b/microprofile/fault-tolerance/src/test/java/io/helidon/microprofile/faulttolerance/MetricsTest.java @@ -1,5 +1,5 @@ /* - * Copyright (c) 2018, 2020 Oracle and/or its affiliates. + * Copyright (c) 2018, 2021 Oracle and/or its affiliates. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -18,45 +18,28 @@ import java.util.concurrent.CompletableFuture; -import org.eclipse.microprofile.faulttolerance.exceptions.CircuitBreakerOpenException; +import org.eclipse.microprofile.metrics.Counter; +import org.eclipse.microprofile.metrics.Gauge; +import org.eclipse.microprofile.metrics.Histogram; import org.eclipse.microprofile.metrics.Metadata; import org.eclipse.microprofile.metrics.MetricRegistry; import org.eclipse.microprofile.metrics.MetricType; import org.eclipse.microprofile.metrics.MetricUnits; import org.junit.jupiter.api.Test; +import org.eclipse.microprofile.faulttolerance.exceptions.CircuitBreakerOpenException; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BREAKER_CALLS_FAILED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BREAKER_CALLS_PREVENTED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BREAKER_CALLS_SUCCEEDED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BREAKER_CLOSED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BREAKER_HALF_OPEN_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BREAKER_OPENED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BREAKER_OPEN_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BULKHEAD_CALLS_ACCEPTED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BULKHEAD_CALLS_REJECTED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BULKHEAD_CONCURRENT_EXECUTIONS; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.BULKHEAD_EXECUTION_DURATION; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.FALLBACK_CALLS_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.INVOCATIONS_FAILED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.INVOCATIONS_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.RETRY_CALLS_FAILED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.RETRY_CALLS_SUCCEEDED_NOT_RETRIED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.RETRY_CALLS_SUCCEEDED_RETRIED_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.RETRY_RETRIES_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.TIMEOUT_CALLS_NOT_TIMED_OUT_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.TIMEOUT_CALLS_TIMED_OUT_TOTAL; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.TIMEOUT_EXECUTION_DURATION; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.enabled; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.getCounter; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.getGauge; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.getHistogram; -import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.getMetricRegistry; import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.greaterThan; import static org.hamcrest.Matchers.is; import static org.hamcrest.Matchers.not; import static org.hamcrest.Matchers.notNullValue; import static org.junit.jupiter.api.Assertions.assertThrows; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.*; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.InvocationFallback.NOT_DEFINED; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.InvocationResult.EXCEPTION_THROWN; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.InvocationResult.VALUE_RETURNED; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.enabled; +import static io.helidon.microprofile.faulttolerance.FaultToleranceMetrics.getMetricRegistry; /** * Tests for bean metrics. @@ -89,216 +72,278 @@ void testInjectCounterProgrammatically() { } @Test - void testGlobalCountersSuccess() throws Exception { + void testGlobalCountersSuccess() { MetricsBean bean = newBean(MetricsBean.class); bean.retryOne(5); - assertThat(getCounter(bean, "retryOne", - INVOCATIONS_TOTAL, int.class), - is(1L)); - assertThat(getCounter(bean, "retryOne", - INVOCATIONS_FAILED_TOTAL, int.class), - is(0L)); + + Counter total = InvocationsTotal.get( + getMethodTag(bean, "retryOne"), + VALUE_RETURNED.get(), + NOT_DEFINED.get()); + assertThat(total.getCount(), is(1L)); + + Counter failedTotal = InvocationsTotal.get( + getMethodTag(bean, "retryOne"), + EXCEPTION_THROWN.get(), + NOT_DEFINED.get()); + assertThat(failedTotal.getCount(), is(0L)); } @Test - void testGlobalCountersFailure() throws Exception { + void testGlobalCountersFailure() { MetricsBean bean = newBean(MetricsBean.class); try { bean.retryTwo(10); } catch (Exception e) { // falls through } - assertThat(getCounter(bean, "retryTwo", - INVOCATIONS_TOTAL, int.class), - is(1L)); - assertThat(getCounter(bean, "retryTwo", - INVOCATIONS_FAILED_TOTAL, int.class), - is(1L)); + + Counter total = InvocationsTotal.get( + getMethodTag(bean, "retryTwo"), + VALUE_RETURNED.get(), + NOT_DEFINED.get()); + assertThat(total.getCount(), is(0L)); + + Counter failedTotal = InvocationsTotal.get( + getMethodTag(bean, "retryTwo"), + EXCEPTION_THROWN.get(), + NOT_DEFINED.get()); + assertThat(failedTotal.getCount(), is(1L)); } @Test - void testRetryCounters() throws Exception { + void testRetryCounters() { MetricsBean bean = newBean(MetricsBean.class); bean.retryThree(5); - assertThat(getCounter(bean, "retryThree", - RETRY_CALLS_SUCCEEDED_NOT_RETRIED_TOTAL, int.class), - is(0L)); - assertThat(getCounter(bean, "retryThree", - RETRY_CALLS_SUCCEEDED_RETRIED_TOTAL, int.class), - is(1L)); - assertThat(getCounter(bean, "retryThree", - RETRY_CALLS_FAILED_TOTAL, int.class), - is(0L)); - assertThat(getCounter(bean, "retryThree", - RETRY_RETRIES_TOTAL, int.class), - is(5L)); + + Counter retryRetriesTotal = RetryRetriesTotal.get( + getMethodTag(bean, "retryThree")); + assertThat(retryRetriesTotal.getCount(), is(5L)); + + Counter retryCallsTotal = RetryCallsTotal.get( + getMethodTag(bean, "retryThree"), + RetryRetried.FALSE.get(), + RetryResult.VALUE_RETURNED.get()); + assertThat(retryCallsTotal.getCount(), is(0L)); + + retryCallsTotal = RetryCallsTotal.get( + getMethodTag(bean, "retryThree"), + RetryRetried.TRUE.get(), + RetryResult.VALUE_RETURNED.get()); + assertThat(retryCallsTotal.getCount(), is(1L)); + + retryCallsTotal = RetryCallsTotal.get( + getMethodTag(bean, "retryThree"), + RetryRetried.TRUE.get(), + RetryResult.MAX_RETRIES_REACHED.get()); + assertThat(retryCallsTotal.getCount(), is(0L)); } @Test - void testRetryCountersFailure() throws Exception { + void testRetryCountersFailure() { MetricsBean bean = newBean(MetricsBean.class); try { bean.retryFour(10); } catch (Exception e) { // falls through } - assertThat(getCounter(bean, "retryFour", - RETRY_CALLS_SUCCEEDED_NOT_RETRIED_TOTAL, int.class), - is(0L)); - assertThat(getCounter(bean, "retryFour", - RETRY_CALLS_SUCCEEDED_RETRIED_TOTAL, int.class), - is(0L)); - assertThat(getCounter(bean, "retryFour", - RETRY_CALLS_FAILED_TOTAL, int.class), - is(1L)); - assertThat(getCounter(bean, "retryFour", - RETRY_RETRIES_TOTAL, int.class), - is(5L)); + + Counter retryRetriesTotal = RetryRetriesTotal.get( + getMethodTag(bean, "retryFour")); + assertThat(retryRetriesTotal.getCount(), is(5L)); + + Counter retryCallsTotal = RetryCallsTotal.get( + getMethodTag(bean, "retryFour"), + RetryRetried.FALSE.get(), + RetryResult.VALUE_RETURNED.get()); + assertThat(retryCallsTotal.getCount(), is(0L)); + + retryCallsTotal = RetryCallsTotal.get( + getMethodTag(bean, "retryFour"), + RetryRetried.TRUE.get(), + RetryResult.VALUE_RETURNED.get()); + assertThat(retryCallsTotal.getCount(), is(0L)); + + retryCallsTotal = RetryCallsTotal.get( + getMethodTag(bean, "retryFour"), + RetryRetried.TRUE.get(), + RetryResult.MAX_RETRIES_REACHED.get()); + assertThat(retryCallsTotal.getCount(), is(1L)); } @Test - void testRetryCountersSuccess() throws Exception { + void testRetryCountersSuccess() { MetricsBean bean = newBean(MetricsBean.class); bean.retryFive(0); - assertThat(getCounter(bean, "retryFive", - RETRY_CALLS_SUCCEEDED_NOT_RETRIED_TOTAL, int.class), - is(1L)); - assertThat(getCounter(bean, "retryFive", - RETRY_CALLS_SUCCEEDED_RETRIED_TOTAL, int.class), - is(0L)); - assertThat(getCounter(bean, "retryFive", - RETRY_CALLS_FAILED_TOTAL, int.class), - is(0L)); - assertThat(getCounter(bean, "retryFive", - RETRY_RETRIES_TOTAL, int.class), - is(0L)); + + Counter retryRetriesTotal = RetryRetriesTotal.get( + getMethodTag(bean, "retryFive")); + assertThat(retryRetriesTotal.getCount(), is(0L)); + + Counter retryCallsTotal = RetryCallsTotal.get( + getMethodTag(bean, "retryFive"), + RetryRetried.FALSE.get(), + RetryResult.VALUE_RETURNED.get()); + assertThat(retryCallsTotal.getCount(), is(1L)); + + retryCallsTotal = RetryCallsTotal.get( + getMethodTag(bean, "retryFive"), + RetryRetried.TRUE.get(), + RetryResult.VALUE_RETURNED.get()); + assertThat(retryCallsTotal.getCount(), is(0L)); + + retryCallsTotal = RetryCallsTotal.get( + getMethodTag(bean, "retryFive"), + RetryRetried.TRUE.get(), + RetryResult.MAX_RETRIES_REACHED.get()); + assertThat(retryCallsTotal.getCount(), is(0L)); } @Test void testTimeoutSuccess() throws Exception { MetricsBean bean = newBean(MetricsBean.class); bean.noTimeout(); - assertThat(getHistogram(bean, "noTimeout", - TIMEOUT_EXECUTION_DURATION).getCount(), - is(1L)); - assertThat(getCounter(bean, "noTimeout", - TIMEOUT_CALLS_NOT_TIMED_OUT_TOTAL), - is(1L)); - assertThat(getCounter(bean, "noTimeout", - TIMEOUT_CALLS_TIMED_OUT_TOTAL), - is(0L)); + + Counter timeoutCallsTotal = TimeoutCallsTotal.get( + getMethodTag(bean, "noTimeout"), + TimeoutTimedOut.TRUE.get()); + assertThat(timeoutCallsTotal.getCount(), is(0L)); + + timeoutCallsTotal = TimeoutCallsTotal.get( + getMethodTag(bean, "noTimeout"), + TimeoutTimedOut.FALSE.get()); + assertThat(timeoutCallsTotal.getCount(), is(1L)); + + Histogram timeoutExecutionDuration = TimeoutExecutionDuration.get( + getMethodTag(bean, "noTimeout")); + assertThat(timeoutExecutionDuration.getCount(), is(1L)); } @Test - void testTimeoutFailure() throws Exception { + void testTimeoutFailure() { MetricsBean bean = newBean(MetricsBean.class); try { bean.forceTimeout(); } catch (Exception e) { // falls through } - assertThat(getHistogram(bean, "forceTimeout", - TIMEOUT_EXECUTION_DURATION).getCount(), - is(1L)); - assertThat(getCounter(bean, "forceTimeout", - TIMEOUT_CALLS_NOT_TIMED_OUT_TOTAL), - is(0L)); - assertThat(getCounter(bean, "forceTimeout", - TIMEOUT_CALLS_TIMED_OUT_TOTAL), - is(1L)); + + Counter timeoutCallsTotal = TimeoutCallsTotal.get( + getMethodTag(bean, "forceTimeout"), + TimeoutTimedOut.TRUE.get()); + assertThat(timeoutCallsTotal.getCount(), is(1L)); + + timeoutCallsTotal = TimeoutCallsTotal.get( + getMethodTag(bean, "forceTimeout"), + TimeoutTimedOut.FALSE.get()); + assertThat(timeoutCallsTotal.getCount(), is(0L)); + + Histogram timeoutExecutionDuration = TimeoutExecutionDuration.get( + getMethodTag(bean, "forceTimeout")); + assertThat(timeoutExecutionDuration.getCount(), is(1L)); } @Test - void testBreakerTrip() throws Exception { + void testBreakerTrip() { MetricsBean bean = newBean(MetricsBean.class); - for (int i = 0; i < CircuitBreakerBean.REQUEST_VOLUME_THRESHOLD ; i++) { + for (int i = 0; i < CircuitBreakerBean.REQUEST_VOLUME_THRESHOLD; i++) { assertThrows(RuntimeException.class, () -> bean.exerciseBreaker(false)); } + assertThrows(CircuitBreakerOpenException.class, () -> bean.exerciseBreaker(false)); - assertThat(getCounter(bean, "exerciseBreaker", - BREAKER_OPENED_TOTAL, boolean.class), - is(1L)); - assertThat(getCounter(bean, "exerciseBreaker", - BREAKER_CALLS_SUCCEEDED_TOTAL, boolean.class), - is(0L)); - assertThat(getCounter(bean, "exerciseBreaker", - BREAKER_CALLS_FAILED_TOTAL, boolean.class), - is((long) CircuitBreakerBean.REQUEST_VOLUME_THRESHOLD)); - assertThat(getCounter(bean, "exerciseBreaker", - BREAKER_CALLS_PREVENTED_TOTAL, boolean.class), - is(1L)); + Counter circuitBreakerOpenedTotal = CircuitBreakerOpenedTotal.get( + getMethodTag(bean, "exerciseBreaker")); + assertThat(circuitBreakerOpenedTotal.getCount(), is(1L)); + + Counter circuitBreakerCallsTotal = CircuitBreakerCallsTotal.get( + getMethodTag(bean, "exerciseBreaker"), + CircuitBreakerResult.SUCCESS.get()); + assertThat(circuitBreakerCallsTotal.getCount(), is(0L)); + + circuitBreakerCallsTotal = CircuitBreakerCallsTotal.get( + getMethodTag(bean, "exerciseBreaker"), + CircuitBreakerResult.FAILURE.get()); + assertThat(circuitBreakerCallsTotal.getCount(), is((long) CircuitBreakerBean.REQUEST_VOLUME_THRESHOLD)); + + circuitBreakerCallsTotal = CircuitBreakerCallsTotal.get( + getMethodTag(bean, "exerciseBreaker"), + CircuitBreakerResult.CIRCUIT_BREAKER_OPEN.get()); + assertThat(circuitBreakerCallsTotal.getCount(), is(1L)); } @Test - void testBreakerGauges() throws Exception { + void testBreakerGauges() { MetricsBean bean = newBean(MetricsBean.class); + + Gauge closedStateTotal = null; + Gauge openStateTotal = null; + Gauge halfOpenStateTotal = null; + for (int i = 0; i < CircuitBreakerBean.REQUEST_VOLUME_THRESHOLD - 1; i++) { assertThrows(RuntimeException.class, () -> bean.exerciseGauges(false)); - assertThat(getGauge(bean, "exerciseGauges", - BREAKER_CLOSED_TOTAL, boolean.class).getValue(), - is(not(0L))); - assertThat(getGauge(bean, "exerciseGauges", - BREAKER_OPEN_TOTAL, boolean.class).getValue(), - is(0L)); - assertThat(getGauge(bean, "exerciseGauges", - BREAKER_HALF_OPEN_TOTAL, boolean.class).getValue(), - is(0L)); + closedStateTotal = CircuitBreakerStateTotal.get( + getMethodTag(bean, "exerciseGauges"), + CircuitBreakerState.CLOSED.get()); + assertThat(closedStateTotal.getValue(), is(not(0L))); + + openStateTotal = CircuitBreakerStateTotal.get( + getMethodTag(bean, "exerciseGauges"), + CircuitBreakerState.OPEN.get()); + assertThat(openStateTotal.getValue(), is(0L)); + halfOpenStateTotal = CircuitBreakerStateTotal.get( + getMethodTag(bean, "exerciseGauges"), + CircuitBreakerState.HALF_OPEN.get()); + assertThat(halfOpenStateTotal.getValue(), is(0L)); } assertThrows(RuntimeException.class, () -> bean.exerciseGauges(false)); assertThrows(CircuitBreakerOpenException.class, () -> bean.exerciseGauges(false)); - assertThat(getGauge(bean, "exerciseGauges", - BREAKER_CLOSED_TOTAL, boolean.class).getValue(), - is(not(0L))); - assertThat(getGauge(bean, "exerciseGauges", - BREAKER_OPEN_TOTAL, boolean.class).getValue(), - is(not(0L))); - assertThat(getGauge(bean, "exerciseGauges", - BREAKER_HALF_OPEN_TOTAL, boolean.class).getValue(), - is(0L)); + assertThat(closedStateTotal.getValue(), is(not(0L))); + assertThat(openStateTotal.getValue(), is(not(0L))); + assertThat(halfOpenStateTotal.getValue(), is(0L)); } @Test void testBreakerExceptionCounters() throws Exception { MetricsBean bean = newBean(MetricsBean.class); + Counter successCallsTotal = CircuitBreakerCallsTotal.get( + getMethodTag(bean, "exerciseBreakerException"), + CircuitBreakerResult.SUCCESS.get()); + + Counter failureCallsTotal = CircuitBreakerCallsTotal.get( + getMethodTag(bean, "exerciseBreakerException"), + CircuitBreakerResult.FAILURE.get()); + + Counter circuitBreakerOpenTotal = CircuitBreakerCallsTotal.get( + getMethodTag(bean, "exerciseBreakerException"), + CircuitBreakerResult.CIRCUIT_BREAKER_OPEN.get()); + // First failure assertThrows(MetricsBean.TestException.class, () -> bean.exerciseBreakerException(false)); // failure - assertThat(getCounter(bean, "exerciseBreakerException", - BREAKER_CALLS_SUCCEEDED_TOTAL, boolean.class), - is(0L)); - assertThat(getCounter(bean, "exerciseBreakerException", - BREAKER_CALLS_FAILED_TOTAL, boolean.class), - is(1L)); - assertThat(getCounter(bean, "exerciseBreakerException", - BREAKER_OPENED_TOTAL, boolean.class), - is(0L)); + assertThat(successCallsTotal.getCount(), is(0L)); + assertThat(failureCallsTotal.getCount(), is(1L)); + assertThat(circuitBreakerOpenTotal.getCount(), is(0L)); // Second failure assertThrows(MetricsBean.TestException.class, () -> bean.exerciseBreakerException(false)); // failure - assertThat(getCounter(bean, "exerciseBreakerException", - BREAKER_CALLS_SUCCEEDED_TOTAL, boolean.class), - is(0L)); - assertThat(getCounter(bean, "exerciseBreakerException", - BREAKER_CALLS_FAILED_TOTAL, boolean.class), - is(2L)); - assertThat(getCounter(bean, "exerciseBreakerException", - BREAKER_OPENED_TOTAL, boolean.class), - is(1L)); + assertThat(successCallsTotal.getCount(), is(0L)); + assertThat(failureCallsTotal.getCount(), is(2L)); + assertThat(circuitBreakerOpenTotal.getCount(), is(0L)); assertThrows(Exception.class, () -> bean.exerciseBreakerException(true)); // failure - assertThat(getCounter(bean, "exerciseBreakerException", - BREAKER_CALLS_SUCCEEDED_TOTAL, boolean.class), - is(0L)); + assertThat(successCallsTotal.getCount(), is(0L)); + assertThat(failureCallsTotal.getCount(), is(2L)); + assertThat(circuitBreakerOpenTotal.getCount(), is(1L)); // Sleep longer than circuit breaker delay Thread.sleep(1500); - // Following calls should succeed due to FailOn + // Following calls should succeed for (int i = 0; i < 2; i++) { try { bean.exerciseBreakerException(true); // success @@ -306,66 +351,119 @@ void testBreakerExceptionCounters() throws Exception { // expected } } - - // Check counters after successful calls - assertThat(getCounter(bean, "exerciseBreakerException", - BREAKER_CALLS_SUCCEEDED_TOTAL, boolean.class), - is(2L)); + assertThat(successCallsTotal.getCount(), is(2L)); + assertThat(failureCallsTotal.getCount(), is(2L)); + assertThat(circuitBreakerOpenTotal.getCount(), is(1L)); try { bean.exerciseBreakerException(true); // success } catch (RuntimeException e) { // expected } - - // Check counters after successful calls - assertThat(getCounter(bean, "exerciseBreakerException", - BREAKER_CALLS_SUCCEEDED_TOTAL, boolean.class), - is(3L)); + assertThat(successCallsTotal.getCount(), is(3L)); + assertThat(failureCallsTotal.getCount(), is(2L)); + assertThat(circuitBreakerOpenTotal.getCount(), is(1L)); } @Test - void testFallbackMetrics() throws Exception { + void testFallbackMetrics() { MetricsBean bean = newBean(MetricsBean.class); - assertThat(getCounter(bean, "fallback", FALLBACK_CALLS_TOTAL), is(0L)); + + Counter fallbackApplied = InvocationsTotal.get( + getMethodTag(bean, "fallback"), + InvocationResult.VALUE_RETURNED.get(), + InvocationFallback.APPLIED.get()); + Counter fallbackNotApplied = InvocationsTotal.get( + getMethodTag(bean, "fallback"), + InvocationResult.VALUE_RETURNED.get(), + InvocationFallback.NOT_APPLIED.get()); + Counter fallbackNotDefined = InvocationsTotal.get( + getMethodTag(bean, "fallback"), + InvocationResult.VALUE_RETURNED.get(), + InvocationFallback.NOT_DEFINED.get()); + + assertThat(fallbackApplied.getCount(), is(0L)); + assertThat(fallbackNotApplied.getCount(), is(0L)); + assertThat(fallbackNotDefined.getCount(), is(0L)); + bean.fallback(); - assertThat(getCounter(bean, "fallback", FALLBACK_CALLS_TOTAL), is(1L)); + + assertThat(fallbackApplied.getCount(), is(1L)); + assertThat(fallbackNotApplied.getCount(), is(0L)); + assertThat(fallbackNotDefined.getCount(), is(0L)); } @Test - void testBulkheadMetrics() throws Exception { + void testBulkheadMetrics() { MetricsBean bean = newBean(MetricsBean.class); CompletableFuture[] calls = getAsyncConcurrentCalls( - () -> bean.concurrent(200), BulkheadBean.TOTAL_CALLS); + () -> bean.concurrent(200), BulkheadBean.TOTAL_CALLS); waitFor(calls); - assertThat(getGauge(bean, "concurrent", - BULKHEAD_CONCURRENT_EXECUTIONS, long.class).getValue(), - is(0L)); - assertThat(getCounter(bean, "concurrent", - BULKHEAD_CALLS_ACCEPTED_TOTAL, long.class), - is((long) BulkheadBean.TOTAL_CALLS)); - assertThat(getCounter(bean, "concurrent", - BULKHEAD_CALLS_REJECTED_TOTAL, long.class), - is(0L)); - assertThat(getHistogram(bean, "concurrent", - BULKHEAD_EXECUTION_DURATION, long.class).getCount(), - is(greaterThan(0L))); + + Gauge executionsRunning = BulkheadExecutionsRunning.get( + getMethodTag(bean, "concurrent")); + assertThat(executionsRunning.getValue(), is(0L)); + + Gauge executionsWaiting = BulkheadExecutionsWaiting.get( + getMethodTag(bean, "concurrent")); + assertThat(executionsWaiting.getValue(), is(0L)); + + Counter acceptedCallsTotal = BulkheadCallsTotal.get( + getMethodTag(bean, "concurrent"), + BulkheadResult.ACCEPTED.get()); + assertThat(acceptedCallsTotal.getCount(), is((long) BulkheadBean.TOTAL_CALLS)); + + Counter rejectedCallsTotal = BulkheadCallsTotal.get( + getMethodTag(bean, "concurrent"), + BulkheadResult.REJECTED.get()); + assertThat(rejectedCallsTotal.getCount(), is(0L)); + + Histogram runningDuration = BulkheadRunningDuration.get( + getMethodTag(bean, "concurrent")); + assertThat(runningDuration.getCount(), is(greaterThan(0L))); + + Histogram awaitingDuration = BulkheadWaitingDuration.get( + getMethodTag(bean, "concurrent")); + assertThat(awaitingDuration.getCount(), is(greaterThan(0L))); } @Test void testBulkheadMetricsAsync() throws Exception { MetricsBean bean = newBean(MetricsBean.class); CompletableFuture[] calls = getConcurrentCalls( - () -> { - try { - return bean.concurrentAsync(200).get(); - } catch (Exception e) { - return "failure"; - } - }, BulkheadBean.TOTAL_CALLS); + () -> { + try { + return bean.concurrentAsync(200).get(); + } catch (Exception e) { + return "failure"; + } + }, BulkheadBean.TOTAL_CALLS); CompletableFuture.allOf(calls).get(); - assertThat(getHistogram(bean, "concurrentAsync", - BULKHEAD_EXECUTION_DURATION, long.class).getCount(), - is(greaterThan(0L))); + + Gauge executionsRunning = BulkheadExecutionsRunning.get( + getMethodTag(bean, "concurrentAsync")); + assertThat(executionsRunning.getValue(), is(0L)); + + Gauge executionsWaiting = BulkheadExecutionsWaiting.get( + getMethodTag(bean, "concurrentAsync")); + assertThat(executionsWaiting.getValue(), is(0L)); + + Counter acceptedCallsTotal = BulkheadCallsTotal.get( + getMethodTag(bean, "concurrentAsync"), + BulkheadResult.ACCEPTED.get()); + assertThat(acceptedCallsTotal.getCount(), is((long) BulkheadBean.TOTAL_CALLS)); + + Counter rejectedCallsTotal = BulkheadCallsTotal.get( + getMethodTag(bean, "concurrentAsync"), + BulkheadResult.REJECTED.get()); + assertThat(rejectedCallsTotal.getCount(), is(0L)); + + Histogram runningDuration = BulkheadRunningDuration.get( + getMethodTag(bean, "concurrentAsync")); + assertThat(runningDuration.getCount(), is(greaterThan(0L))); + + Histogram awaitingDuration = BulkheadWaitingDuration.get( + getMethodTag(bean, "concurrentAsync")); + assertThat(awaitingDuration.getCount(), is(greaterThan(0L))); } } diff --git a/microprofile/tests/arquillian/src/main/java/io/helidon/microprofile/arquillian/HelidonDeployableContainer.java b/microprofile/tests/arquillian/src/main/java/io/helidon/microprofile/arquillian/HelidonDeployableContainer.java index ee7c0a9d8cd..dc279d5b91f 100644 --- a/microprofile/tests/arquillian/src/main/java/io/helidon/microprofile/arquillian/HelidonDeployableContainer.java +++ b/microprofile/tests/arquillian/src/main/java/io/helidon/microprofile/arquillian/HelidonDeployableContainer.java @@ -51,11 +51,14 @@ import jakarta.enterprise.inject.se.SeContainer; import jakarta.enterprise.inject.spi.CDI; import jakarta.enterprise.inject.spi.DefinitionException; +import jakarta.enterprise.util.AnnotationLiteral; import org.eclipse.microprofile.config.Config; import org.eclipse.microprofile.config.ConfigProvider; import org.eclipse.microprofile.config.spi.ConfigBuilder; import org.eclipse.microprofile.config.spi.ConfigProviderResolver; import org.eclipse.microprofile.config.spi.ConfigSource; +import org.eclipse.microprofile.metrics.MetricRegistry; +import org.eclipse.microprofile.metrics.annotation.RegistryType; import org.jboss.arquillian.container.spi.client.container.DeployableContainer; import org.jboss.arquillian.container.spi.client.container.DeploymentException; import org.jboss.arquillian.container.spi.client.protocol.ProtocolDescription; @@ -110,6 +113,17 @@ public HelidonContainerConfiguration getContainerConfig() { */ private final Map contexts = new HashMap<>(); + /** + * Annotation literal to inject base registry. + */ + static class BaseRegistryTypeLiteral extends AnnotationLiteral implements RegistryType { + + @Override + public MetricRegistry.Type type() { + return MetricRegistry.Type.BASE; + } + } + @Override public Class getConfigurationClass() { return HelidonContainerConfiguration.class; @@ -373,11 +387,17 @@ private void ensureBeansXml(Path classesDir) throws IOException { @Override public void undeploy(Archive archive) { + // Clean up all the base metrics for next test + cleanupBaseMetrics(); + + // Clean up contexts RunContext context = contexts.remove(archive.getId()); if (null == context) { LOGGER.severe("Undeploying an archive that was not deployed. ID: " + archive.getId()); return; } + + // Stop the server try { context.runnerClass.getDeclaredMethod("stop") .invoke(context.runner); @@ -392,8 +412,8 @@ public void undeploy(Archive archive) { Thread.currentThread().setContextClassLoader(context.oldClassLoader); } + // Try to clean up the deploy directory if (containerConfig.getDeleteTmp()) { - // Try to clean up the deploy directory if (context.deployDir != null) { try { Files.walk(context.deployDir) @@ -431,6 +451,18 @@ public void undeploy(Descriptor descriptor) { // No-Op } + /** + * Injects the base metric registry and cleans up all metrics in preparation to run another + * Arquillian test in the same VM. Without this cleanup, metrics added by a previous test + * would be available and may cause failures. + */ + private void cleanupBaseMetrics() { + MetricRegistry metricRegistry = CDI.current().select(MetricRegistry.class, + new BaseRegistryTypeLiteral()).get(); + Objects.requireNonNull(metricRegistry); + metricRegistry.removeMatching((m, v) -> true); + } + /** * Copies the given Archive to the given deployDir. For each asset copied, the callback will be invoked. * diff --git a/microprofile/tests/tck/tck-fault-tolerance/pom.xml b/microprofile/tests/tck/tck-fault-tolerance/pom.xml index ea121f6b5b8..f9919750d0e 100644 --- a/microprofile/tests/tck/tck-fault-tolerance/pom.xml +++ b/microprofile/tests/tck/tck-fault-tolerance/pom.xml @@ -30,8 +30,8 @@ Helidon Microprofile Tests TCK Fault Tolerance - - true + + 4.0.3 @@ -76,6 +76,12 @@ slf4j-simple test + + org.awaitility + awaitility + ${version.lib.awaitility} + test + diff --git a/microprofile/tests/tck/tck-fault-tolerance/src/test/tck-suite.xml b/microprofile/tests/tck/tck-fault-tolerance/src/test/tck-suite.xml index 58f4f619255..eb1ba34f09a 100644 --- a/microprofile/tests/tck/tck-fault-tolerance/src/test/tck-suite.xml +++ b/microprofile/tests/tck/tck-fault-tolerance/src/test/tck-suite.xml @@ -21,9 +21,9 @@ - +