more multi thread coordination need to avoid panics in TestStartBuildRunFollowLog

[gabemontero]

Changes

github actions back e2e runs turned up this panic in the new unit test I added in one of its recent runs:

[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1b9e0d9]

goroutine 21 [running]:
testing.tRunner.func1.2({0x1c9bdc0, 0x2c0bd70})
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1209 +0x36c
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1212 +0x3b6
panic({0x1c9bdc0, 0x2c0bd70})
	/opt/hostedtoolcache/go/1.17.1/x64/src/runtime/panic.go:1047 +0x266
github.com/shipwright-io/cli/pkg/shp/cmd/build.(*RunCommand).onEvent(0xc000379e30, 0xc0000ec800)
	/home/runner/work/cli-1/cli-1/pkg/shp/cmd/build/run.go:121 +0xbd9
github.com/shipwright-io/cli/pkg/shp/cmd/build.TestStartBuildRunFollowLog(0xc0003644e0)
	/home/runner/work/cli-1/cli-1/pkg/shp/cmd/build/run_test.go:159 +0x147c
testing.tRunner(0xc0003644e0, 0x1ed2ee8)
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1259 +0x230
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1306 +0x727
FAIL	github.com/shipwright-io/cli/pkg/shp/cmd/build	0.206s

need to employ my watch lock a bit more in the test along with the use of Gocsched

but I still avoid a blanket, sleep always approach to guarantee Run() init has occurred

/assign @otaviof
/assign @coreydaley

/kind bug

Submitter Checklist

• [/ ] Includes tests if functionality changed/was added
• [n/a ] Includes docs if changes are user-facing
• [/ ] Set a kind label on this PR
• [/ ] Release notes block has been filled in, or marked NONE

See the contributor guide
for details on coding conventions, github and prow interactions, and the code review process.

Release Notes

NONE

[gabemontero]

hmmm I have more work to do

=== RUN   TestStartBuildRunFollowLog
    run_test.go:172: Run initialization did not complete in time
coverage: 25.7% of statements
panic: test timed out after 10m0s

goroutine 23 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1788 +0xbb
created by time.goFunc
	/opt/hostedtoolcache/go/1.17.1/x64/src/time/sleep.go:180 +0x4a

goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc00036bd40, {0x1e3718b, 0x1a}, 0x1ed3f18)
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1307 +0x752
testing.runTests.func1(0x0)
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1598 +0x9a
testing.tRunner(0xc00036bd40, 0xc0003d9bc0)
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1259 +0x230
testing.runTests(0xc00019c200, {0x2c0e060, 0x1, 0x1}, {0x0, 0xc0000d4200, 0x2c2d440})
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1596 +0x7cb
testing.(*M).Run(0xc00019c200)
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1504 +0x9d2
main.main()
	_testmain.go:101 +0x325

goroutine 19 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
	/home/runner/work/cli/cli/vendor/k8s.io/klog/v2/klog.go:1169 +0x8b
created by k8s.io/klog/v2.init.0
	/home/runner/work/cli/cli/vendor/k8s.io/klog/v2/klog.go:417 +0x1c5

goroutine 5 [semacquire, 10 minutes]:
sync.runtime_SemacquireMutex(0x4a0dd7, 0x7, 0x4a0e05)
	/opt/hostedtoolcache/go/1.17.1/x64/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0xc0001ab1e4)
	/opt/hostedtoolcache/go/1.17.1/x64/src/sync/mutex.go:138 +0x21d
sync.(*Mutex).Lock(0xc0001ab1e4)
	/opt/hostedtoolcache/go/1.17.1/x64/src/sync/mutex.go:81 +0x65
github.com/shipwright-io/cli/pkg/shp/cmd/build.(*RunCommand).onEvent(0xc0001ab180, 0xc0000f4000)
	/home/runner/work/cli/cli/pkg/shp/cmd/build/run.go:98 +0x85
github.com/shipwright-io/cli/pkg/shp/cmd/build.TestStartBuildRunFollowLog(0xc000001860)
	/home/runner/work/cli/cli/pkg/shp/cmd/build/run_test.go:177 +0x14f5
testing.tRunner(0xc000001860, 0x1ed3f18)
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1259 +0x230
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1306 +0x727

goroutine 9 [select, 10 minutes]:
github.com/shipwright-io/cli/pkg/shp/reactor.(*PodWatcher).Start(0xc00015e0c0)
	/home/runner/work/cli/cli/pkg/shp/reactor/pod_watcher.go:89 +0x185
github.com/shipwright-io/cli/pkg/shp/cmd/build.(*RunCommand).Run(0xc0001aaf50, 0xc0000d4740, 0xc00027bf50)
	/home/runner/work/cli/cli/pkg/shp/cmd/build/run.go:202 +0xcc7
github.com/shipwright-io/cli/pkg/shp/cmd/build.TestStartBuildRunFollowLog.func3()
	/home/runner/work/cli/cli/pkg/shp/cmd/build/run_test.go:148 +0x5a
created by github.com/shipwright-io/cli/pkg/shp/cmd/build.TestStartBuildRunFollowLog
	/home/runner/work/cli/cli/pkg/shp/cmd/build/run_test.go:147 +0x13ee

goroutine 11 [select, 10 minutes]:
github.com/shipwright-io/cli/pkg/shp/reactor.(*PodWatcher).Start(0xc0000a9a40)
	/home/runner/work/cli/cli/pkg/shp/reactor/pod_watcher.go:89 +0x185
github.com/shipwright-io/cli/pkg/shp/cmd/build.(*RunCommand).Run(0xc0001aafc0, 0xc0000d4940, 0xc000357e30)
	/home/runner/work/cli/cli/pkg/shp/cmd/build/run.go:202 +0xcc7
github.com/shipwright-io/cli/pkg/shp/cmd/build.TestStartBuildRunFollowLog.func3()
	/home/runner/work/cli/cli/pkg/shp/cmd/build/run_test.go:148 +0x5a
created by github.com/shipwright-io/cli/pkg/shp/cmd/build.TestStartBuildRunFollowLog
	/home/runner/work/cli/cli/pkg/shp/cmd/build/run_test.go:147 +0x13ee

goroutine 13 [semacquire, 10 minutes]:
sync.runtime_SemacquireMutex(0x0, 0x0, 0x0)
	/opt/hostedtoolcache/go/1.17.1/x64/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0xc0001ab1e4)
	/opt/hostedtoolcache/go/1.17.1/x64/src/sync/mutex.go:138 +0x21d
sync.(*Mutex).Lock(0xc0001ab1e4)
	/opt/hostedtoolcache/go/1.17.1/x64/src/sync/mutex.go:81 +0x65
github.com/shipwright-io/cli/pkg/shp/cmd/build.(*RunCommand).Run(0xc0001ab180, 0xc0000d4bc0, 0xc0003c6db0)
	/home/runner/work/cli/cli/pkg/shp/cmd/build/run.go:151 +0x7a
github.com/shipwright-io/cli/pkg/shp/cmd/build.TestStartBuildRunFollowLog.func3()
	/home/runner/work/cli/cli/pkg/shp/cmd/build/run_test.go:148 +0x5a
created by github.com/shipwright-io/cli/pkg/shp/cmd/build.TestStartBuildRunFollowLog
	/home/runner/work/cli/cli/pkg/shp/cmd/build/run_test.go:147 +0x13ee
FAIL	github.com/shipwright-io/cli/pkg/shp/cmd/build	600.206s

[gabemontero]

/approve

I've pinged @adambkaplan in shipwright slack about adjusting the permissions in this repo so I can re-run the unit tests a few more times before we potentially merge this one

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gabemontero

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [gabemontero]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gabemontero]

/approve

I've pinged @adambkaplan in shipwright slack about adjusting the permissions in this repo so I can re-run the unit tests a few more times before we potentially merge this one

thanks @adambkaplan ... rerunning test jobs now ... will do that a few times today, see how clean they are

[otaviof]

I've tested it locally as well, it looks solid!

[otaviof]

/lgtm