Clean: clean up stale files under /tmp

[Pennyzct]

Description of problem

Stale dir /tmp/virtc/ocibundle/ has caused recent ARM CI failure.
http://jenkins.katacontainers.io/job/kata-containers-runtime-ARM-18.04-PR/1414/
Related PR: kata-containers/runtime#2387
BTW, almost ten thousand😖 stale file/dir have been accumulated on ARM CI.
We need to clean up /tmp on all bare-metal CI. ;)

[Pennyzct]

@devimc

[Pennyzct]

/test-arm

[grahamwhaley]

Hmm, that feels a bit too 'hard' to me - the system may have other tmp files down there it is using (given this is a baremetal server). I fear you will corrupt the server long term.

Also, have we identified which bit of the ci/test scripts is leaving the rubbish in /tmp? Some time ago I spent some time making sure the test framework did not do that - if we can identify the 'dirty tests' and clean them up, that would be better :-)

[devimc]

thanks @Pennyzct , I agree with @grahamwhaley - remove all file in /tmp is prone to break something

[Pennyzct]

Hi~ guys
@devimc @grahamwhaley
Sorry for the delay~~thanks for the suggestion. ;). you're right. Cleaning up the whole /tmp is so brutal.
Here are all the stale files that may be kata-container-test-related:

drwx------   2 root    root      4096 Jan 15 16:47 cid116018226
                    .........
-rw-------   1 jenkins jenkins 183542 Jan 16 21:23 codecov.dN9l7T.gz
                   ..........
drwxr-x---   3 root    root      4096 Jan 13 21:23 containers-mapping107357574
                    ...........
-rw-------   1 jenkins jenkins      0 Jan 16 20:25 img013976482 ✔
                   ..............
drwx------   3 jenkins jenkins   4096 Jan 13 21:24 kata-050276636
                    ............
drwx------   2 jenkins jenkins   4096 Jan 13 21:24 kata-agent-test141906389 ✔
                    .............
drwx------   3 jenkins jenkins   4096 Jan 13 21:24 kata-cleanup099002422 ✔
                    ..............
drwx------   2 root    root      4096 Jan 13 21:24 kata-shim-007796300 
                    ............
-rw-------   1 root    root       384 Jan 14 02:49 kubeadm_config.0rtpew.yaml ✔
                    .............
-rw-r--r--   1 root    root        62 Jan 16 20:24 mock_hook.log
                    .............
-rw-------   1 root    root       290 Jan 14 23:08 pod_config.1fs8Z1.yaml ✔
                    ............
drwx------   3 jenkins jenkins   4096 Jan 16 20:24 shimV2-010669478
                    .............
drwx------   2 root    root      4096 Jan 14 03:24 store-tmp-186691371 ✔
                    ............
drwx------   2 root    root      4096 Jan 14 23:11 tmp.Bj5S6yV3bh
                    .............
drwx------   2 jenkins jenkins   4096 Jan 13 21:24 vmfactory-tmp-016048745 ✔
                    ............

Lately. I've been investigating which code is leading to those stale files and will try my best to clean them up from the source. ;).

Updates：
I've fired two PRs kata-containers/runtime#2399 and #2211 to deal with those with the ticks.

[grahamwhaley]

thanks @Pennyzct - I think last time I did this I used the trick of setting TMPDIR, TMP and TEMP to a separate folder (like ~/tmp), did a CI test run, and then could see exactly what files were left from a single run :-)