Deploy page for Weave Cloud (Dev) is flapping

[squaremo]

The Deploy page for https://frontend.dev.weave.works/proud-wind-05/deploy/services is continuously reporting disconnected daemon. The daemon log says it's getting cut off.

ts=2019-10-21T16:17:25.959161139Z caller=upstream.go:174 component=upstream disconnected=true
ts=2019-10-21T16:17:25.959198453Z caller=upstream.go:145 component=upstream connectionclosing=true err="websocket: close sent"
ts=2019-10-21T16:17:30.95936319Z caller=upstream.go:133 component=upstream connecting=true
ts=2019-10-21T16:17:30.97950402Z caller=upstream.go:147 component=upstream connected=true

The "websocket: close sent" indicates it was told to close the websocket, by the flux-api service.

The flux-api logs show:

ts=2019-10-21T16:32:33.437776974Z caller=server.go:692 method=ListImagesWithOptions instance=2 url="/v10/images?containerFields=Name,Current,LatestFiltered,AvailableError,AvailableImagesCount,NewAvailableImagesCount,FilteredImagesCount,NewFilteredImagesCount&service=<all>" took=455.624533ms status_code=422 error="parsing :cronjob/es-clean-cortex: invalid service ID"
ts=2019-10-21T16:32:35.198852287Z caller=server.go:692 method=ListServices instance=1010 url="/v6/services?namespace=" took=484.196194ms status_code=200
ts=2019-10-21T16:32:35.701292956Z caller=server.go:692 method=Status instance=2 url=/v6/status took=2.515219ms status_code=200
ts=2019-10-21T16:32:37.832342455Z caller=server.go:692 method=ListImagesWithOptions instance=1010 url="/v10/images?containerFields=Name,Current,LatestFiltered,AvailableError,AvailableImagesCount,NewAvailableImagesCount,FilteredImagesCount,NewFilteredImagesCount&service=<all>" took=1.227257806s status_code=200

If I port forward to the fluxd API, I can get a response for e.g., ListServices. The first bit looks like this:

[{"ID":":cronjob/es-clean-cortex", ...

So yes, the daemon is returning invalid resource IDs. Ooops. Plausibly: https://github.com/fluxcd/flux/pull/2520/files#diff-852a02c5a194efb68182b9b3fcf85b13R200

[squaremo]

I've locked it and rolled it back to docker.io/fluxcd/flux-prerelease:master-655c907c (immediately prior to the suspect PR being merged), for now. Investigating smoking things.

[squaremo]

fluxcd/flux#2539 fixes the problem with the daemon.