[JENKINS-52165] Calling Controller.watch API #63

jglick · 2018-02-09T18:21:33Z

JENKINS-52165 Split out of #21. Downstream of jenkinsci/durable-task-plugin#60. Subsumes #73. Should improve performance by switching durable task logging and exit status checking from being pull (initiated by master at intervals) to push (initiated by agent at close intervals).

@reviewbybees

jglick · 2018-02-09T18:31:19Z

By the way jenkinsci/support-core-plugin#128 should make it easier to see the impact of this on actual Remoting usage. Have not yet tried it myself.

oleg-nenashev

Merge conflict, but the code itself LGTM

oleg-nenashev

merge conflict

oleg-nenashev · 2018-04-02T15:18:56Z

Approved once the upstream PRs are integrated && the conflict is fixed

vivek · 2018-04-03T01:40:28Z

This looks good to go if merge conflicts fixed? ping @jglick.

svanoort · 2018-04-03T14:04:42Z

@vivek The upstream isn't really ready-to-go yet, and the whole thing needs some sort of proper and thorough testing in a simulated environment. Oleg's review found some issues already.

We could of course YOLO-merge all this but I make no promises on the fallout if so.

jglick · 2018-04-03T21:27:24Z

Well, the upstream “issues” are just things like Javadoc that should be improved, not user-visible problems, but yes this all deserves testing in a realistic environment. Once @svanoort is ready to start that I will polish up all the PRs.

jglick · 2018-06-25T19:35:49Z

src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java


        private static final long MIN_RECURRENCE_PERIOD = 250; // ¼s
        private static final long MAX_RECURRENCE_PERIOD = 15000; // 15s
        private static final float RECURRENCE_PERIOD_BACKOFF = 1.2f;
+        private static final long WATCHING_RECURRENCE_PERIOD = Main.isUnitTest ? /* 5s */5_000: /* 5m */300_000;


Should probably be switched to nonfinal constant initialized via system property and overridden in the relevant test only.

jglick · 2018-06-25T19:36:36Z

src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java

            this.remote = ws.getRemote();
-            setupTimer();
+            try {


Should introduce a system property to act as a kill switch and fall back to polling.

…estart.

…which do not seem to have closed or broken the channel.

jglick · 2018-08-11T13:36:47Z

src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java

+    /** If set to false, disables {@link Execution#watching} mode. */
+    @SuppressWarnings("FieldMayBeFinal")
+    private static boolean USE_WATCHING = !"false".equals(System.getProperty(DurableTaskStep.class.getName() + ".USE_WATCHING"));
+
    /**
     * Represents one task that is believed to still be running.


I will flesh this out with an explanation of the expected state changes.

dwnusbaum · 2018-08-13T21:08:43Z

src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java

+                os = ps;
+            }
+            try {
+                IOUtils.copy(stream, os);


It looks like all PrintStream#write overloads synchronize on this and null-check out before writing (since it is nulled out once the stream is closed). Do we need to do either of those here as well? This usage should always be from the same thread IIUC, but do other threads have access to listener.getLogger()?

Agreed, that sounds safer.

…tStream.write would have.

dwnusbaum · 2018-08-14T16:03:54Z

src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java

+                if (ps.getClass() == PrintStream.class) {
+                    // Try to extract the underlying stream, since swallowing exceptions is undesirable and PrintStream.checkError is useless.
+                    OutputStream os = (OutputStream) printStreamDelegate.get(ps);
+                    synchronized (os) { // like PrintStream.write overloads do


Did you mean synchronized (ps)?

Indeed that seems to be the lock, will change.

…utStream.

dwnusbaum · 2018-08-14T20:34:58Z

src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java

+                if (ps.getClass() == PrintStream.class) {
+                    // Try to extract the underlying stream, since swallowing exceptions is undesirable and PrintStream.checkError is useless.
+                    synchronized (ps) { // like PrintStream.write overloads do
+                        IOUtils.copy(stream, (OutputStream) printStreamDelegate.get(ps));


Is it possible for the PrintStream to be closed? If so, we'd get an NPE because printStreamDelegate.get(ps) will be null.

It should not be closed. If it is, we expect this method to throw an exception. I can make it throw a specific exception for better diagnostics.

Ok, I guess it would be clear to anyone investigating where the NPE came from anyway, so it probably doesn't need a specific exception, but 3f8ced4 won't hurt.

…sed.

jglick · 2018-08-17T17:54:05Z

src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java

+
+        // ditto
+        @Override public void problem(Exception x) {
+            Functions.printStackTrace(x, listener().getLogger());


Seems to work. Due to an apparent mistake in jenkinsci/pipeline-cloudwatch-logs-plugin#5 affecting agent-side, but not master-side, logging, I saw in the build log:

java.lang.NullPointerException at io.jenkins.plugins.pipeline_log_fluentd_cloudwatch.FluentdLogger$FluentdOutputStream.eol(FluentdLogger.java:128) at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60) at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56) at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:74) at java.io.PrintStream.write(PrintStream.java:480) at org.jenkinsci.plugins.credentialsbinding.impl.BindingStep$Filter$1.eol(BindingStep.java:183) at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60) at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56) at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:74) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769) at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$HandlerImpl.output(DurableTaskStep.java:571) at org.jenkinsci.plugins.durabletask.FileMonitoringTask$Watcher.run(FileMonitoringTask.java:477) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

with the build proceeding nonetheless.

svanoort · 2018-09-06T21:17:15Z

src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java

+        private static final Field printStreamDelegate;
+        static {
+            try {
+                printStreamDelegate = FilterOutputStream.class.getDeclaredField("out");


Ahh, reflection hacks: the universal panacea!

Unfortunately I could not find a better way to do this.

svanoort · 2018-09-06T21:32:28Z

src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java

+            } catch (Exception x) {
+                // Try to report it to the master.
+                try {
+                    execution.problem(x);


I predict we'll be seeing a lot of this particular report when there are network issues. Thankfully it won't be that often due to the between recurrence periods.

svanoort

I guess it's as good as it's going to get for now -- happy to see us doing something with errors besides letting the PrintStream swallow it

[JENKINS-52165] Calling Controller.watch API

[JENKINS-38381] Calling Controller.watch API.

5c20d5e

jglick requested review from abayer, rsandell, oleg-nenashev and svanoort February 9, 2018 18:21

jglick changed the title ~~[JENKINS-38381] Calling Controller.watch API~~ [JENKINS-38381] [JENKINS-37575] Calling Controller.watch API Feb 9, 2018

Merge branch 'UTF-8-JENKINS-31096' into watch-plus-UTF-8-JENKINS-31096

b4ff3a5

jglick mentioned this pull request Feb 13, 2018

Watching plus Unicode #65

Closed

jglick mentioned this pull request Mar 12, 2018

[FIXED JENKINS-26521] Add activity flag to timeout step jenkinsci/workflow-basic-steps-plugin#62

Merged

oleg-nenashev approved these changes Mar 26, 2018

View reviewed changes

oleg-nenashev requested changes Apr 2, 2018

View reviewed changes

oleg-nenashev approved these changes Apr 2, 2018

View reviewed changes

jglick added 7 commits June 8, 2018 16:28

Merge branch 'master' into watch-JENKINS-38381

cb05308

Matching jenkinsci/durable-task-plugin@6974ae2.

13885df

Merge branch 'UTF-8-JENKINS-31096' into watch-plus-UTF-8-JENKINS-31096

af31da4

Merge branch 'watch-JENKINS-38381' into watch-plus-UTF-8-JENKINS-31096

bd27dc6

Review comments from @oleg-nenashev.

5eccfbb

Merge branch 'UTF-8-JENKINS-31096' into watch-plus-UTF-8-JENKINS-31096

756a87d

Merge branch 'watch-JENKINS-38381' into watch-plus-UTF-8-JENKINS-31096

6c424e0

jglick mentioned this pull request Jun 11, 2018

[JEP-210] Integration tests with DurableTaskStep #21

Merged

7 tasks

jglick mentioned this pull request Jun 25, 2018

[JENKINS-37575] Keep track of how much content has been copied by writeLog even if the callable is interrupted jenkinsci/durable-task-plugin#74

Merged

jglick changed the title ~~[JENKINS-38381] [JENKINS-37575] Calling Controller.watch API~~ [JENKINS-38381] Calling Controller.watch API Jun 25, 2018

jglick commented Jun 25, 2018

View reviewed changes

jglick changed the title ~~[JENKINS-38381] Calling Controller.watch API~~ [JENKINS-52165] Calling Controller.watch API Jun 25, 2018

jglick requested a review from dwnusbaum June 25, 2018 21:38

jglick added 5 commits August 10, 2018 16:07

Noting a way to reproduce a loss of a small amount of output during r…

6722cdb

…estart.

Work around exception suppression from PrintStream.

62aeb45

Verifying jenkinsci/durable-task-plugin@b077ce3 in a test.

ed4551a

Comments.

c41f3ec

After discussion with @dwnusbaum, attempting to proceed after errors …

c859378

…which do not seem to have closed or broken the channel.

jglick requested review from svanoort and dwnusbaum August 10, 2018 21:48

jglick commented Aug 11, 2018

View reviewed changes

Richer Javadoc.

3673025

dwnusbaum reviewed Aug 13, 2018

View reviewed changes

When writing directly to PrintStream.out, synchronize on it like Prin…

43b6d27

…tStream.write would have.

dwnusbaum reviewed Aug 14, 2018

View reviewed changes

Synchronization should be on the PrintStream, not its underlying Outp…

24ee12b

…utStream.

dwnusbaum reviewed Aug 14, 2018

View reviewed changes

Fail with a more informative exception in case the PrintStream is clo…

3f8ced4

…sed.

dwnusbaum approved these changes Aug 14, 2018

View reviewed changes

jglick commented Aug 17, 2018

View reviewed changes

jglick added 3 commits August 22, 2018 12:15

Merge branch 'master' into watch-JENKINS-38381

d54b065

Merge branch 'master' into watch-JENKINS-38381

efb481c

Bump.

77b81c3

svanoort reviewed Sep 6, 2018

View reviewed changes

svanoort approved these changes Sep 6, 2018

View reviewed changes

svanoort merged commit 3bd3498 into jenkinsci:master Sep 13, 2018

jglick deleted the watch-JENKINS-38381 branch September 13, 2018 18:31

jglick mentioned this pull request Oct 8, 2018

[JENKINS-54078] JEP-210 compatibility jenkinsci/workflow-basic-steps-plugin#74

Merged

This was referenced Oct 19, 2018

Reverting custom thread pool from #53 in watch mode #81

Merged

[JENKINS-52165] Disable watch mode by default for now #84

Merged

orgi pushed a commit to orgi/workflow-durable-task-step-plugin that referenced this pull request Nov 13, 2018

Merge pull request jenkinsci#63 from jglick/watch-JENKINS-38381

2070f0a

[JENKINS-52165] Calling Controller.watch API

jglick mentioned this pull request Jul 7, 2023

[JENKINS-52165] Stabilize and better test USE_WATCHING #323

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JENKINS-52165] Calling Controller.watch API #63

[JENKINS-52165] Calling Controller.watch API #63

jglick commented Feb 9, 2018 •

edited

Loading

jglick commented Feb 9, 2018

oleg-nenashev left a comment

oleg-nenashev left a comment

oleg-nenashev commented Apr 2, 2018

vivek commented Apr 3, 2018

svanoort commented Apr 3, 2018

jglick commented Apr 3, 2018

jglick Jun 25, 2018

jglick Jun 25, 2018

jglick Aug 11, 2018

jglick Aug 13, 2018

dwnusbaum Aug 13, 2018 •

edited

Loading

jglick Aug 14, 2018

dwnusbaum Aug 14, 2018

jglick Aug 14, 2018

dwnusbaum Aug 14, 2018

jglick Aug 14, 2018

dwnusbaum Aug 14, 2018

jglick Aug 17, 2018

svanoort Sep 6, 2018

jglick Sep 7, 2018

svanoort Sep 6, 2018

svanoort left a comment

[JENKINS-52165] Calling Controller.watch API #63

[JENKINS-52165] Calling Controller.watch API #63

Conversation

jglick commented Feb 9, 2018 • edited Loading

jglick commented Feb 9, 2018

oleg-nenashev left a comment

Choose a reason for hiding this comment

oleg-nenashev left a comment

Choose a reason for hiding this comment

oleg-nenashev commented Apr 2, 2018

vivek commented Apr 3, 2018

svanoort commented Apr 3, 2018

jglick commented Apr 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dwnusbaum Aug 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

svanoort left a comment

Choose a reason for hiding this comment

jglick commented Feb 9, 2018 •

edited

Loading

dwnusbaum Aug 13, 2018 •

edited

Loading