Handle build/taskrun timeout outside of resync

[vdemeester]

Capturing discussion over Slack, knative/build#543 breaks our timeout handling, because we rely on the resync to enforce timeouts. build-pipeline rely on the same for timeouts (or will with #355).

The reason to change the resync time is the following

Resyncing every 30s seems to lead us to hit rate limits that we never recover from when there are too many builds in the system at a given time. Serving also switched to resyncing every 10h for similar scaling reasons.

We need to rely on something else for build/taskrun timeouts, running a background goroutine in the controller, watching builds instead could be an option (we'll need to handle the case where the controller pod restarts, …)

/cc @imjasonh @bobcatfish @shashwathi @abayer

[nader-ziada]

@bobcatfish I think this should be part of the 0.2 milestone as we discussed

[vdemeester]

/assign

[vdemeester]

/assign @shashwathi

[shashwathi]

/assign @shashwathi

[vdemeester]

/assign @shashwathi