roachtest: remove stable vs unstable distinction

[petermattis]

Remove the distinction between stable vs unstable tests. That
distinction had more value in the past when it was useful to delineate
which tests were stable and which weren't and when there were more
failures due to the test infrastructure. Now that most tests are stable
and the roachtest infrastruture is mature, the value has
lessened. Rather than keeping the burden of managing the "stable" tag,
we're just removing it.

Release note: None

[cockroach-teamcity]

This change is 

[andreimatei]

LGTM

Reviewable status: complete! 0 of 0 LGTMs obtained

[petermattis]

bors r=andreimatei

[craig]

Build succeeded

• GitHub CI (Cockroach)

[danhhz]

I'm 👎 on this. In both partitioning and then changefeeds, it was quite useful to have an unstable test that would give us a daily datapoint but was ignored by anyone investigating the roachtest failures for a release. If the motivation is the burden of maintaining the tag, perhaps we could switch the default to stable and only specify it for the unstable tests.

[tbg]

@danhhz as a counterpoint, I only look at issues and issues get posted for both. Do we also want to stop issues from being posted (hopefully not)? Perhaps you can get a similar result by appending [unstable] to a failed test's issue and communicate the intent that way, without all the boilerplate of maintaining the abstraction officially. (You could also put "unstable" in the test name, but that seems like the kind of thing that would never be changed).

[petermattis]

As one of the frequent investigators of roachtest failures, it isn't onerous to remember that a particular test was recently added and thus unstable.

[tbg]

I think @danhhz might be worried about whoever is on release rotation (and thus isn't necessarily experienced). But I think noting the instability in the issue for the test is going to work out OK.

[danhhz]

The issues are useful as a convenient history, so I wouldn't want those to go away.

But I think noting the instability in the issue for the test is going to work out OK.

I think it's quite optimistic to think the release engineer is going to read enough of the issue to notice this. OTOH, the worst case here is an extra round of questions from individuals that don't frequently investigate roachtest failures. And it sounds like I've been the main (only?) person having this issue. I think the burden of maintaining this feature is being overstated here, but perhaps a test naming convention is the appropriate compromise.

[tbg]

I think it's quite optimistic to think the release engineer is going to read enough of the issue to notice this

In case that wasn't clear, I was suggesting adding [unstable] to the title (I think the issue poster still finds it, it just searches for the issue it would post). See #30832 for an example. Might not be enough 🤷‍♂️ we will soon find out.

[danhhz]

In case that wasn't clear, I was suggesting adding [unstable] to the title (I think the issue poster still finds it, it just searches for the issue it would post).

TIL. I don't have a mental model of how this thing works and have been assuming I can't change the issue title at all.