Wish list: make Sieve more general and ergonomic

[marshtompsxd]

Ergonomics

There is a manual porting process to onboard a project to use Sieve. We hope to automate this step by integrating Sieve with existing frameworks like operator-sdk so that users can build controllers that are Sieve-friendly from the very beginning.

• Automatically instrument the controller deployment manifest file
• Support e2e testing framework like https://github.com/kubernetes-sigs/e2e-framework to make writing tests easier
• Integrate with https://github.com/operator-framework/operator-sdk
• Better ways to browse test results

Generality

Sieve currently places the instrumentation in controller-runtime to intercept the interactions between the controller and the Kubernetes API. That said, we also need to support controllers that are not built on top of controller-runtime but directly rely on client-go.

• For generality, move instrumentation of the controller read/write APIs to client-go
  Move instrumentation from controller write APIs to rest call APIs #85
  Move instrumentation from controller cache read APIs to store/cache APIs #96
• Allow users to specify the boundary of reconcile functions
• Instrument the user-specified reconcile function automatically

Miscellaneous

• Run Sieve server (i.e., test coordinator and trace collector) as a pod in the Kind cluster

[isubasinghe]

Hello @marshtompsxd I remember you from verus, I worked on that with Andrea. I wanted to instrument argo-workflows which uses client-go directly but wasn't really able to find any bugs. I know for certain that the controller has plenty of bugs. If I push my changes, do you think you'd have time to take a look and inform me of where I am making a mistake? Thanks.

[marshtompsxd]

Hi @isubasinghe. Thank you for reaching out. Testing argo-workflows using Sieve sounds very cool. Please feel free to open a PR with the changes you made to instrument the controller and I will help review it.

I know for certain that the controller has plenty of bugs.

What types of bugs are you looking for in this controller? Do you have some concrete examples (e.g., historical bugs)? I want to make sure Sieve is the right tool to help you find bugs.

[isubasinghe]

Thanks a lot @marshtompsxd, I will push my changes when I get a chance in that case.

I've found multiple bugs of various kinds, varying from race conditions to unhandled bits of logic resulting in workflows being stuck in Runnning even though the pods have terminated. One off the top of my head is that we failed to mark a node as Error/Failed when a pod was terminated due to a missing CRD update.