Avoid infinite loops due to corrupted flow graphs in some cases and improve resumption error handling

[dwnusbaum]

Related to #347. This PR tries to improve these main issues:

1. Infinite loops in StandardGraphLookupView.bruteForceScanForEnclosingBlock and LinearBlockHoppingScanner are now detected and throw an exception. This does not fix all possible infinite loops with flow graph iteration APIs, but these are the only two cases I saw where the infinite loop happens internally rather than the caller actually iterating over nodes infinitely. Loops in StandardGraphLookupView.bruteForceScanForEnclosingBlock via PlaceholderTask.getAffinityKey brick the build queue, and loops in LinearBlockHoppingScanner brick the CpsVmExecutorService for the build in question.
2. When we try to resume a Pipeline but run into something that prevents steps from resuming, we should do what we can to ensure the build does not hang forever waiting for steps to resume.

Testing done

Submitter checklist

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests - that demonstrates feature works or fixes the issue

[dwnusbaum]

I will try to create tests for all of these cases to see what happens.

[dwnusbaum]

Well, it's kind of interesting. Specifically in the case of a missing or corrupt FlowNode, there are two cases that I see:

1. If FlowExecution.heads contains the bad node, then everything gets handled in CpsFlowExecution.onLoad, we never read program.dat, we create placeholder nodes, the Pipeline fails, great.
2. If FlowExecution.heads does not contain the bad node, then the Pipeline loads, we attempt to resume it, various warnings get logged, and it hangs forever. The problem is that this call in CpsStepContext throws right away, so the CpsThread never resumes with the outcome, and the Pipeline hangs. It seems like this log message should at least be bumped to WARNING because any errors there are likely to be the proximate cause of a build hanging forever, but I wonder if we should also attempt to call CpsFlowExecution.croak or CpsVmExecutorService.reportProblem in that catch block.

So long story short, it seems like there is no point trying to do anything special in these cases in this plugin for now.

[dwnusbaum]

jenkinsci/workflow-cps-plugin#916 updates the relevant log message to WARNING so that we have an idea if the problematic situation is ever occurring in practice.

[dwnusbaum]

Not sure if this would be better or worse than killing all of the steps. If you end up here, something is pretty wrong with your flow graph. The main causes I can think of are if a BlockEndNode.startId points to a node that either doesn't exist, is not a BlockStartNode, or (the new case I just saw) points to a BlockStartNode with a newer ID than the end node, creating a cycle in the graph.

The question though is whether flow graph corruption necessarily means that trying to resume the steps is pointless, or if there is a decent chance that if we fall back to synchronous resumption of each step execution in order, the build might be able to complete successfully.

[jglick]

Ah I was just looking for #221 the other day as a reference, because its original motivating use case is obsolete as of jenkinsci/ssh-agent-plugin#152. Of course there might still be other steps which legitimately need context from an enclosing step in their onResume, but I am not aware of any offhand. So I think it would be reasonable to fall back to resuming all steps in random order. I would not expect the build to complete successfully, but it might be able to e.g. release external resources more gracefully.

[dwnusbaum]

c6b4fa9 adjusts things so we resume all step executions as long as they have a FlowNode. #349 (comment) discusses the cases where the FlowNode is missing and/or corrupted.