untarring sonobuoy retrieve tarball sometimes results in EOF in archive error

[waynr]

What steps did you take and what happened:
I have a script used in CI that:

• runs sonobuoy
• waits for its status to become complete
• retrieves the results tarball into an output directory and lets the user know where it is
• dumps the e2e test summary to stdout, ie

SUCCESS! -- 136 Passed | 0 Failed | 0 Pending | 860 Skipped 

Occasionally the last step fails with the following output:

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

Which suggests to me that sonobuoy retrieve is not waiting for the tarball to be complete before pulling it locally.

What did you expect to happen:
I expect to be able to untar the tarball and grep files of interest to display results.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Seems potentially related to the following two issues:

• Sonobuoy 0.11.4 panics when analyzing results tarball #498
• Make 'sonobuoy retrieve' cli command aware if tarball is available or not #407

Environment:

• Sonobuoy version: v0.13.0
• Kubernetes version: (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"753b2dbc622f5cc417845f0ff8a77f539a4213ea", GitTreeState:"clean", BuildDate:"2018-11-26T14:31:35Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version: kubespray master branch
• Cloud provider or hardware configuration: openstack private cloud
• OS (e.g. from /etc/os-release):

 Machine ID:                 d4a9d15df9ad4f12993ffaff95800d0c
 System UUID:                D4A9D15D-F9AD-4F12-993F-FAFF95800D0C
 Boot ID:                    b09c3841-1bba-479e-9fcd-4112497bb6ae
 Kernel Version:             4.14.88-coreos
 OS Image:                   Container Linux by CoreOS 1967.3.0 (Rhyolite)
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://18.6.1
 Kubelet Version:            v1.11.5
 Kube-Proxy Version:         v1.11.5

[timothysc]

/assign @chuckha

This isn't actually a problem with sonobuoy but an issue with streaming to tar and how messages are propagated.

This has existed for quite some time.

[waynr]

@timothysc can you recommend a workaround to deterministically download a correct results tarball? Maybe just keep downloading and extracting it locally until the tar extract command succeeds?

[johnSchnake]

I'm having trouble finding a reference to the streaming tar/msg propagation issue you mention. Could you provide a link or some more details to make it searchable?

I had thought it had to do with simply having a large tarball so I was using a custom kube-conformance image to test that theory but haven't been able to repro locally. It seemed like I would get this often though whenever I was using a cluster in the cloud and had networking/latency to contend with.

[johnSchnake]

In my analysis so far I've now been able to repro this and I think I know the issue, but a sanity check is good.

It seems just that the root issue is that the sonobuoy status command is really just relying on the pod annoations for the plugins to report that they are done. When the plugs report done, the status command will report done.

The issue is that once they report done, the master node still compresses everything into a tar.gz which takes time if it is large.

I was able to repro this by using a custom kube-conformance image (schnake/kube-conformance:big) which creates a large file of results (1.9G). As soon as the status said complete, I tried to retrieve the results. A few times I just got a missing file error, then the first time that it worked the file was corrupted as reported here.

So it seems like the fix is that the master itself needs to report another status that we monitor indicating if the roundup/prep of the results is really ready or not.