Implementation of Fault tolerance 4.0 (#3664)

* Fault Tolerance 3.0 Support (#2680) * Initial changes to implement new metrics layer. Moving from complex names to simpler names and tags. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * More metric updates. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Migration of most unit tests to new metrics. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Completed migration of metrics test. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * New exception to discern timeouts during retries. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Implementation of retry metrics. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Cleanup metrics between tests. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Several changes related to execution of FT 3.0 TCKs. Adjusted initial size of executors and fixed a few other problems. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Copyright and checkstyle updates. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Fixed copyright year. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Fixed typos and some cleanup. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Created exclude file as a workaround for a sportbugs' bug. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Updated copyright year. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * MicroProfile Opentracing 2.0 (#2676) * Microprofile Opentracing uprgated to 2.0 * Unused dependences removed * Obsolete excludes removed * Sync up of microprofile-4.0 with master branch (#2757) * Upgrade Netty to 4.1.58 (#2678) Signed-off-by: Tomas Langer <tomas.langer@oracle.com> * Added overall timeout to evictable cache (#2659) Signed-off-by: Tomas Langer <tomas.langer@oracle.com> * Fix copyright year for commits broken by squashing. (#2687) Signed-off-by: Tomas Langer <tomas.langer@oracle.com> * Concat array enhancement (#2508) * Concat array enhancement Signed-off-by: Daniel Kec <daniel.kec@oracle.com> * Update Jackson to 2.12.1 (#2690) * Update Jackson to 2.12.1 * Upgrade to latest Junit5 to get fix for junit-team/junit5#2198 * Manage junit4 version * PokemonService template fixed in SE Database Archetype. (#2701) Signed-off-by: Tomas Kraus <Tomas.Kraus@oracle.com> * Fixed different output in DbClient SE archetype (#2703) Signed-off-by: Tomas Kraus <Tomas.Kraus@oracle.com> * Fix TODO application: (#2708) - WebSecurity needs to be passed config.get("security") to take the "security.web-server" configuration - Added outbound configuration for the google login - Upgraded cassandra driver to fix issues with old guava dependencies - Removed metrics to avoid issues with cassandra driver. Fixes #2707 * Update k8s descriptors to avoid using deprecated APIs. (#2719) * Separate execution of DataChunkReleaseTest in its own VM to prevent leak messages in other test's logs. (#2716) Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Changes in this commit: (#2727) 1. Upgrade to Jersey 2.33 2. Configuration via system properties for the Jersey Client API. Any response in an exception will be mapped to an empty one to prevent data leaks. See eclipse-ee4j/jersey#4641. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Properly release underlying buffer before passing it to WebSocket handler (#2715) * Properly release underlying buffer before passing it to handler. * Releases data chunks after passing them to Tyrus without any copying. Reports an error and closes connection if Tyrus is unable to handle the data. Finally, fixed a problem related to subscription requests. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Removed unused logger. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Fixed checkstyle. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Fix issue with null value in JSON. (#2723) Signed-off-by: Tomas Langer <tomas.langer@oracle.com> * Upgrade grpc to v1.35.0 (#2713) * Upgrade grpc to v1.35.0 * Update copyright * Upgrades OCI SDK to version 1.31.0 (#2699) * Updated OCI to 1.31.0 Signed-off-by: Laird Nelson <laird.nelson@oracle.com> * Fix null array values in HOCON/JSON config parser. (#2731) Resolves #2720 (follow-up) * Performance improvements to queue(s) management in Webserver (#2704) * Initial patch. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Fixed some type params and improved comments. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * More cleanup and make sure to fail publisher on an error condition. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Suppress warnings. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Call clearQueues on every new request for proper cleanup of keep-alive connections. Some copyright fixes. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Fixed checkstyle issues. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Force logging of LEAK error even if finalize does not get called on a DataChunk. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Upgrade Weld (#2668) Signed-off-by: Tomas Langer <tomas.langer@oracle.com> * Rest client async header propagation with usage of Helidon Context (#2735) Rest client header propagation with usage of Helidon Context Signed-off-by: David Kral <david.k.kral@oracle.com> * Allow override of Jersey property via config (#2737) * Allow the default value of property jersey.config.client.ignoreExceptionResponse to be overridden via config. New test. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Fixed copyright year. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * New implementation of LazyValue (#2738) * New implementation of LazyValue that lazily initializes a Semaphore instead of eagerly creating a ReentrantLock. Makes use of volatile guarantees and atomicity of VarHandle updates. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * New test for LazyValueImpl. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Reduced sleep time in test. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Update CHANGELOG for 2.2.1 release (#2743) * 2.2.1 THIRD_PARTY_LICENSES update (#2746) * Update THIRD_PARTY_LICENSES * Support async invocations using optional synthetic SimplyTimed behavior (#2745) * Add support for async invocations for optional inferred SimplyTimed behavior on JAX-RS endpoints Signed-off-by: tim.quinn@oracle.com <tim.quinn@oracle.com> * Do not attempt to access the request context in Fallback callback. If used together with Retry, it is possible for the fallback to be called in a fresh thread for which there is no current request scope. Instead just use the original value obtained in this class' constructor. Updated functional test (with some class renaming) to cover this use case. (#2748) Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Fix for native image. (#2753) Signed-off-by: Tomas Langer <tomas.langer@oracle.com> * Fixed checkstyle issues. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> Co-authored-by: Tomas Langer <tomas.langer@gmail.com> Co-authored-by: Daniel Kec <daniel.kec@oracle.com> Co-authored-by: Joe DiPol <joe.dipol@oracle.com> Co-authored-by: Tomáš Kraus <tomas.kraus@oracle.com> Co-authored-by: Romain Grecourt <romain.grecourt@oracle.com> Co-authored-by: Jonathan Knight <jk@thegridman.com> Co-authored-by: Laird Nelson <laird.nelson@oracle.com> Co-authored-by: David Král <david.k.kral@oracle.com> Co-authored-by: Tim Quinn <tim.quinn@oracle.com> * Fixed problems in RetryImpl after merge. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Fixed problems with metrics after merge. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Updated version in suite file. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Fixed problem retrieving registry for metrics. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Fixed more problems after merge. All tests are passing now. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Fixed checkstyle errors. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Fixed TODO. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Enabled TCK's by default and removed generated file. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * One more checkstyle violation. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> * Removed duplicate test after merge. Signed-off-by: Santiago Pericasgeertsen <santiago.pericasgeertsen@oracle.com> Co-authored-by: Dmitry Aleksandrov <mitiaalexandrov@gmail.com> Co-authored-by: Tomas Langer <tomas.langer@gmail.com> Co-authored-by: Daniel Kec <daniel.kec@oracle.com> Co-authored-by: Joe DiPol <joe.dipol@oracle.com> Co-authored-by: Tomáš Kraus <tomas.kraus@oracle.com> Co-authored-by: Romain Grecourt <romain.grecourt@oracle.com> Co-authored-by: Jonathan Knight <jk@thegridman.com> Co-authored-by: Laird Nelson <laird.nelson@oracle.com> Co-authored-by: David Král <david.k.kral@oracle.com> Co-authored-by: Tim Quinn <tim.quinn@oracle.com>
helidon-io · Nov 22, 2021 · fad6d72 · fad6d72
1 parent db35b26
commit fad6d72
Show file tree

Hide file tree

Showing 19 changed files with 1,461 additions and 654 deletions.
diff --git a/fault-tolerance/src/main/java/io/helidon/faulttolerance/RetryImpl.java b/fault-tolerance/src/main/java/io/helidon/faulttolerance/RetryImpl.java
@@ -73,9 +73,10 @@ private <T> Single<T> retrySingle(RetryContext<? extends CompletionStage<T>> con
 
         long nanos = System.nanoTime() - context.startedNanos;
         if (nanos > maxTimeNanos) {
-            TimeoutException te = new TimeoutException("Execution took too long. Already executing: "
-                    + TimeUnit.NANOSECONDS.toMillis(nanos) + " ms, must timeout after: "
-                    + TimeUnit.NANOSECONDS.toMillis(maxTimeNanos) + " ms.");
+            TimeoutException te = new RetryTimeoutException(context.throwable(),
+                    "Execution took too long. Already executing: "
+                            + TimeUnit.NANOSECONDS.toMillis(nanos) + " ms, must timeout after: "
+                            + TimeUnit.NANOSECONDS.toMillis(maxTimeNanos) + " ms.");
             if (context.hasThrowable()) {
                 te.initCause(context.throwable());
             }
@@ -115,9 +116,10 @@ private <T> Multi<T> retryMulti(RetryContext<? extends Flow.Publisher<T>> contex
 
         long nanos = System.nanoTime() - context.startedNanos;
         if (nanos > maxTimeNanos) {
-            return Multi.error(new TimeoutException("Execution took too long. Already executing: "
-                                                             + TimeUnit.NANOSECONDS.toMillis(nanos) + " ms, must timeout after: "
-                                                             + TimeUnit.NANOSECONDS.toMillis(maxTimeNanos) + " ms."));
+            return Multi.error(new RetryTimeoutException(context.throwable(),
+                    "Execution took too long. Already executing: "
+                            + TimeUnit.NANOSECONDS.toMillis(nanos) + " ms, must timeout after: "
+                            + TimeUnit.NANOSECONDS.toMillis(maxTimeNanos) + " ms."));
         }
 
         if (currentCallIndex > 0) {
@@ -190,3 +192,4 @@ Throwable throwable() {
         }
     }
 }
+
diff --git a/fault-tolerance/src/main/java/io/helidon/faulttolerance/RetryTimeoutException.java b/fault-tolerance/src/main/java/io/helidon/faulttolerance/RetryTimeoutException.java
@@ -0,0 +1,51 @@
+/*
+ * Copyright (c) 2021 Oracle and/or its affiliates.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.helidon.faulttolerance;
+
+import java.util.concurrent.TimeoutException;
+
+/**
+ * Subclass of {@link TimeoutException} to discern exceptions thrown by a {@link Retry}
+ * when its overall timeout is reached versus those thrown by a {@link Timeout}.
+ */
+public class RetryTimeoutException extends TimeoutException {
+    private static final long serialVersionUID = 1900926677490550714L;
+
+    private final Throwable lastRetryException;
+
+    /**
+     * Constructs a {@code RetryTimeoutException} with the specified detail
+     * message.
+     *
+     * @param throwable last retry exception
+     * @param message the detail message
+     */
+    public RetryTimeoutException(Throwable throwable, String message) {
+        super(message);
+        lastRetryException = throwable;
+    }
+
+    /**
+     * Last exception thrown in {@code Retry} before the overall timeout reached.
+     *
+     * @return last exception thrown
+     */
+    public Throwable lastRetryException() {
+        return lastRetryException;
+    }
+}
+
diff --git a/...ces/META-INF/native-image/io.helidon.jersey/helidon-jersey-client/native-image.properties b/...ces/META-INF/native-image/io.helidon.jersey/helidon-jersey-client/native-image.properties
@@ -0,0 +1,17 @@
+#
+# Copyright (c) 2021 Oracle and/or its affiliates.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+Args=--initialize-at-run-time=org.glassfish.jersey.client.internal.HttpUrlConnector
diff --git a/microprofile/fault-tolerance/etc/spotbugs/exclude.xml b/microprofile/fault-tolerance/etc/spotbugs/exclude.xml
@@ -0,0 +1,34 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+    Copyright (c) 2021 Oracle and/or its affiliates.
+
+    Licensed under the Apache License, Version 2.0 (the "License");
+    you may not use this file except in compliance with the License.
+    You may obtain a copy of the License at
+
+        http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+
+-->
+
+<FindBugsFilter
+        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+        xmlns="https://github.com/spotbugs/filter/3.0.0"
+        xsi:schemaLocation="https://github.com/spotbugs/filter/3.0.0 https://raw.githubusercontent.com/spotbugs/spotbugs/3.1.0/spotbugs/etc/findbugsfilter.xsd">
+
+    <Match>
+        <!--
+          - Incorrectly reported on just one enum in FaultToleranceMetrics.java
+          - https://github.com/spotbugs/spotbugs/issues/740
+          -->
+        <Field type="org.eclipse.microprofile.metrics.Tag" />
+        <Bug pattern="SE_BAD_FIELD" />
+    </Match>
+
+</FindBugsFilter>
diff --git a/microprofile/fault-tolerance/pom.xml b/microprofile/fault-tolerance/pom.xml
@@ -33,6 +33,10 @@
         Microprofile fault tolerance implementation
     </description>
 
+    <properties>
+        <spotbugs.exclude>etc/spotbugs/exclude.xml</spotbugs.exclude>
+    </properties>
+
     <build>
         <plugins>
             <plugin>

diff --git a/...ofile/faulttolerance/CommandFallback.java → ...rofile/faulttolerance/FallbackHelper.java b/...ofile/faulttolerance/CommandFallback.java → ...rofile/faulttolerance/FallbackHelper.java
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2018, 2020 Oracle and/or its affiliates.
+ * Copyright (c) 2018, 2021 Oracle and/or its affiliates.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -26,9 +26,9 @@
 import org.eclipse.microprofile.faulttolerance.FallbackHandler;
 
 /**
- * Class CommandFallback.
+ * Implements invocation callback logic.
  */
-class CommandFallback {
+class FallbackHelper {
 
     private final InvocationContext context;
 
@@ -45,7 +45,7 @@ class CommandFallback {
      * @param introspector Method introspector.
      * @param throwable Throwable that caused execution of fallback
      */
-    CommandFallback(InvocationContext context, MethodIntrospector introspector, Throwable throwable) {
+    FallbackHelper(InvocationContext context, MethodIntrospector introspector, Throwable throwable) {
         this.context = context;
         this.throwable = throwable;
 
@@ -103,24 +103,13 @@ public Throwable getFailure() {
                 result = fallbackMethod.invoke(context.getTarget(), context.getParameters());
             }
         } catch (Throwable t) {
-            updateMetrics();
-
             // If InvocationTargetException, then unwrap underlying cause
             if (t instanceof InvocationTargetException) {
                 t = t.getCause();
             }
             throw t instanceof Exception ? (Exception) t : new RuntimeException(t);
         }
 
-        updateMetrics();
         return result;
     }
-
-    /**
-     * Updates fallback metrics.
-     */
-    private void updateMetrics() {
-        Method method = context.getMethod();
-        FaultToleranceMetrics.getCounter(method, FaultToleranceMetrics.FALLBACK_CALLS_TOTAL).inc();
-    }
 }
diff --git a/...lerance/src/main/java/io/helidon/microprofile/faulttolerance/FaultToleranceExtension.java b/...lerance/src/main/java/io/helidon/microprofile/faulttolerance/FaultToleranceExtension.java
@@ -229,38 +229,29 @@ private void registerFaultToleranceMethods(AnnotatedType<?> type) {
     }
 
     /**
-     * Registers metrics for all FT methods and init executors.
+     * Validates annotations.
      *
      * @param validation Event information.
      */
-    void registerMetricsAndInitExecutors(@Observes AfterDeploymentValidation validation) {
+    void validateAnnotations(@Observes AfterDeploymentValidation validation) {
         if (FaultToleranceMetrics.enabled()) {
             getRegisteredMethods().stream().forEach(beanMethod -> {
                 final Method method = beanMethod.method();
                 final Class<?> beanClass = beanMethod.beanClass();
 
-                // Counters for all methods
-                FaultToleranceMetrics.registerMetrics(method);
-
-                // Metrics depending on the annotationSet present
                 if (MethodAntn.isAnnotationPresent(beanClass, method, Retry.class)) {
-                    FaultToleranceMetrics.registerRetryMetrics(method);
                     new RetryAntn(beanClass, method).validate();
                 }
                 if (MethodAntn.isAnnotationPresent(beanClass, method, CircuitBreaker.class)) {
-                    FaultToleranceMetrics.registerCircuitBreakerMetrics(method);
                     new CircuitBreakerAntn(beanClass, method).validate();
                 }
                 if (MethodAntn.isAnnotationPresent(beanClass, method, Timeout.class)) {
-                    FaultToleranceMetrics.registerTimeoutMetrics(method);
                     new TimeoutAntn(beanClass, method).validate();
                 }
                 if (MethodAntn.isAnnotationPresent(beanClass, method, Bulkhead.class)) {
-                    FaultToleranceMetrics.registerBulkheadMetrics(method);
                     new BulkheadAntn(beanClass, method).validate();
                 }
                 if (MethodAntn.isAnnotationPresent(beanClass, method, Fallback.class)) {
-                    FaultToleranceMetrics.registerFallbackMetrics(method);
                     new FallbackAntn(beanClass, method).validate();
                 }
                 if (MethodAntn.isAnnotationPresent(beanClass, method, Asynchronous.class)) {
@@ -269,17 +260,17 @@ void registerMetricsAndInitExecutors(@Observes AfterDeploymentValidation validat
             });
         }
 
-        // Initialize executors for MP FT - default size of 16
+        // Initialize executors for MP FT - default size of 20
         io.helidon.config.Config config = MpConfig.toHelidonConfig(ConfigProvider.getConfig());
         scheduledThreadPoolSupplier = ScheduledThreadPoolSupplier.builder()
                 .threadNamePrefix("ft-mp-schedule-")
-                .corePoolSize(16)
+                .corePoolSize(20)
                 .config(config.get("scheduled-executor"))
                 .build();
         FaultTolerance.scheduledExecutor(scheduledThreadPoolSupplier);
         threadPoolSupplier = ThreadPoolSupplier.builder()
                 .threadNamePrefix("ft-mp-")
-                .corePoolSize(16)
+                .corePoolSize(20)
                 .config(config.get("executor"))
                 .build();
         FaultTolerance.executor(threadPoolSupplier);