Update documentation for analyzing PR failures

[jkotas]

Document the recent engineering system improvements

[ghost]

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Document the recent engineering system improvements

Author:	jkotas
Assignees:	-
Labels:	area-Meta
Milestone:	-

[jkotas]

cc @dotnet/runtime-infrastructure

[danmoseley]

cc @markwilkie

[tommcdon]

cc @hoyosjs

[ViktorHofer]

FYI @danmoseley (EDIT: saw your comment too late).

[markwilkie]

Thanks all!

[danmoseley]

I actually didn't know about this known errors format/system. How can we increase awareness? @ViktorHofer maybe including in your next SDK update mail (not sure when that is)?

[markwilkie]

There's been some talk too of "advertising" in a PR comment. Is this something folks are interested in? (we've been slow to do this kind of thing so we're not SPAMmy)

[ViktorHofer]

@ViktorHofer maybe including in your next SDK update mail (not sure when that is)?

cc @jkoritzinsky @joperezr whoI believe will update the SDK in arcade and runtime soon.

[danmoseley]

There's been some talk too of "advertising" in a PR comment. Is this something folks are interested in? (we've been slow to do this kind of thing so we're not SPAMmy)

Can you do that only for PR's that were merged on red? If so, seems like something worth a trial. "This PR was merged with one or more failing pipelines, but it is not linked to any known failures. Our goal is for all PR's merged with failures to be linked to a known failure. Please see foo.md for information about how to create and link known failures." or something better ?

[jkotas]

I actually didn't know about this known errors format/system. How can we increase awareness?

FWIW, this was mentioned in "DevWF Team is Addressing YOUR Feedback Update!" email sent to everybody on 8/11, and a several other similar emails before that.

[markwilkie]

Can you do that only for PR's that were merged on red?

We could. Would devs go back and actually look at it? @tommcdon mentioned too the possibility of advertising the build analysis tab for each failed PR. (this obviously has the create known issue link too)

Again....not sure what folks want.

[danmoseley]

They would see the notification, either for their PR or someone else's. It would be an experiment for a temporary period, and if it was a problem we'd stop it.
We were recently discussing posting info about pipeline costs when a particularly expensive PR is merged, so it seems reasonable to try

[hoyosjs]

There was an issue with this recently and the test log search will be rolling out. After that a how we work talk could be a good forum for this.