cluster: use WaitConditionNextExit

[tbg]

There is a chance that this will address an issue that causes spurious
failures on CI, where it looks like we're janking the file system out
from under a running process. For context, see:

#58955

Release note: None

[cockroach-teamcity]

This change is 

[tbg]

No idea what this is but it's not this PR:

@postamar git blame shows you as touching this last, is this something you could route?

------- Stdout: -------
=== RUN   TestDescriptorRepair/_SELECT_crdb_internal.unsafe_upsert_descriptor(59,_crdb_internal.json_to_pb('cockroach.sql.sqlbase.Descriptor',__'{___"table":_{_____"columns":_[_{_"id":_1,_"name":_"i",_"type":_{_"family":_"IntFamily",_"oid":_20,_"width":_64_}_}_],_____"families":_[_______{_________"columnIds":_[_1_],_________"columnNames":_[_"i"_],_________"defaultColumnId":_0,_________"id":_0,_________"name":_"primary"_______}_____],_____"formatVersion":_3,_____"id":_59,_____"name":_"foo",_____"nextColumnId":_2,_____"nextFamilyId":_1,_____"nextIndexId":_2,_____"nextMutationId":_1,_____"parentId":_52,_____"primaryIndex":_{_______"columnDirections":_[_"ASC"_],_______"columnIds":_[_1_],_______"columnNames":_[_"i"_],_______"id":_1,_______"name":_"primary",_______"type":_"FORWARD",_______"unique":_true,_______"version":_1_____},_____"privileges":_{_______"owner_proto":_"admin",_______"users":_[_________{_"privileges":_2,_"user_proto":_"admin"_},_________{_"privileges":_2,_"user_proto":_"root"_},_________{_"privile
ges":_8,_"user_proto":_"newuser1"_}_______],_______"version":_1_____},_____"state":_"PUBLIC",_____"unexposedParentSchemaId":_29,_____"version":_1___}_}_'))_
    repair_test.go:447: 
        	Error Trace:	repair_test.go:447
        	Error:      	Received unexpected error:
        	            	pq: setting updated but timed out waiting to read new value
        	Test:       	TestDescriptorRepair/_SELECT_crdb_internal.unsafe_upsert_descriptor(59,_crdb_internal.json_to_pb('cockroach.sql.sqlbase.Descriptor',__'{___"table":_{_____"columns":_[_{_"id":_1,_"name":_"i",_"type":_{_"family":_"IntFamily",_"oid":_20,_"width":_64_}_}_],_____"families":_[_______{_________"columnIds":_[_1_],_________"columnNames":_[_"i"_],_________"defaultColumnId":_0,_________"id":_0,_________"name":_"primary"_______}_____],_____"formatVersion":_3,_____"id":_59,_____"name":_"foo",_____"nextColumnId":_2,_____"nextFamilyId":_1,_____"nextIndexId":_2,_____"nextMutationId":_1,_____"parentId":_52,_____"primaryIndex":_{_______"columnDirections":_[_"ASC"_],_______"columnIds":_[_1_],_______"columnNames":_[_"i"_],_______"id":_1,_______"name":_"primary",_______"type":_"FORWARD",_______"unique":_true,_______"version":_1_____},_____"privileges":_{_______"owner_proto":_"admin",_______"users":_[_________{_"privileges":_2,_"user_proto":_"admin"_},_________{_"privileges":_2,_"user_proto":_"root"_},_______
__{_"privileges":_8,_"user_proto":_"newuser1"_}_______],_______"version":_1_____},_____"state":_"PUBLIC",_____"unexposedParentSchemaId":_29,_____"version":_1___}_}_'))_
    --- FAIL: TestDescriptorRepair/_SELECT_crdb_internal.unsafe_upsert_descriptor(59,_crdb_internal.json_to_pb('cockroach.sql.sqlbase.Descriptor',__'{___"table":_{_____"columns":_[_{_"id":_1,_"name":_"i",_"type":_{_"family":_"IntFamily",_"oid":_20,_"width":_64_}_}_],_____"families":_[_______{_________"columnIds":_[_1_],_________"columnNames":_[_"i"_],_________"defaultColumnId":_0,_________"id":_0,_________"name":_"primary"_______}_____],_____"formatVersion":_3,_____"id":_59,_____"name":_"foo",_____"nextColumnId":_2,_____"nextFamilyId":_1,_____"nextIndexId":_2,_____"nextMutationId":_1,_____"parentId":_52,_____"primaryIndex":_{_______"columnDirections":_[_"ASC"_],_______"columnIds":_[_1_],_______"columnNames":_[_"i"_],_______"id":_1,_______"name":_"primary",_______"type":_"FORWARD",_______"unique":_true,_______"version":_1_____},_____"privileges":_{_______"owner_proto":_"admin",_______"users":_[_________{_"privileges":_2,_"user_proto":_"admin"_},_________{_"privileges":_2,_"user_proto":_"root"_},_________{_"pri
vileges":_8,_"user_proto":_"newuser1"_}_______],_______"version":_1_____},_____"state":_"PUBLIC",_____"unexposedParentSchemaId":_29,_____"version":_1___}_}_'))_ (10.42s)

bors r=rickystewart

[craig]

Build failed:

• GitHub CI (Cockroach)

[tbg]

bors r=rickystewart

…

On Fri, Feb 19, 2021, 18:26 craig[bot] ***@***.***> wrote: Build failed: - GitHub CI (Cockroach) <https://teamcity.cockroachdb.com/viewLog.html?buildId=2695259&buildTypeId=Cockroach_UnitTests> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#60781 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGXPZARVXMWWZLOYAQK3LLS72NLJANCNFSM4X366TDA> .

[craig]

Build failed (retrying...):

• GitHub CI (Cockroach)

[tbg]

If this fails again due to something clearly unrelated, please merge again. I'm out for the weekend.

…

On Fri, Feb 19, 2021, 21:06 craig[bot] ***@***.***> wrote: Build failed (retrying...): - GitHub CI (Cockroach) <https://teamcity.cockroachdb.com/viewLog.html?buildId=2695716&buildTypeId=Cockroach_UnitTests> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#60781 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGXPZHOU3UVTOL7F3FNRLDS73ADBANCNFSM4X366TDA> .

[craig]

Build succeeded:

• GitHub CI (Cockroach)

[postamar]

Thanks for reporting this. This failure was ultimately caused by SET CLUSTER SETTING timing out and throwing an error, and the proximal cause is that I added assertions on the statement error in the descriptor repair test when previously there were none. I'll file an issue.