Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publish using darc stage is not detecting failed promotion build #6403

Closed
1 of 2 tasks
epananth opened this issue Oct 14, 2020 · 12 comments
Closed
1 of 2 tasks

Publish using darc stage is not detecting failed promotion build #6403

epananth opened this issue Oct 14, 2020 · 12 comments
Assignees

Comments

@epananth
Copy link
Member

epananth commented Oct 14, 2020

  • This issue is blocking
  • This issue is causing unreasonable pain

Publish using darc queues a build in Maestro promotion pipeline but it fails to detect if the promotion failed.

Eg : https://dnceng.visualstudio.com/internal/_build/results?buildId=851119&view=logs&j=226748d0-f812-5437-d3f0-2dd291f5666e&t=bad11196-972e-5d03-45a8-9db526506073

Failed promotion pipeline : https://dnceng.visualstudio.com/internal/_build/results?buildId=851947&view=results

[Edit]
[Add a warning that Promotion pipeline may have failed till we fix this error]

  1. Add-build-to-channel should detect if the promotion pipeline failed and fail the build
@epananth epananth self-assigned this Oct 14, 2020
@epananth
Copy link
Member Author

cc : @mmitche @riarenas @MattGal

@riarenas
Copy link
Member

Add work around to not to wait of the promotion pipeline build

This already exists right? there's a --no-wait parameter

@mmitche
Copy link
Member

mmitche commented Oct 14, 2020

Yeah, the existing way builds are promoted is desirable. add-build-to-channel should not do the promotion itself unless --skip-assets-publishing is passed.

@epananth
Copy link
Member Author

Yes the work around exists but we are not using it, so I was planning to add that for time being till I fix this the correct way ..

@riarenas
Copy link
Member

Builds shouldn't pass --no-wait. It's correct behavior that they wait for the promotion build, it's just not correct that the step is saying it was successful.

@epananth
Copy link
Member Author

If we are okay to show that build passed even when it failed for time being then we don't have to add the work around. I was thinking it will reduce the confusion if maestro promotion was successful or not cos then one will be forced to check the status of the build in the maestro pipeline and not rely on the msg in the logs..

I will edit the description.

@mmitche
Copy link
Member

mmitche commented Oct 14, 2020

I think what we do today is wait for the promotion build to complete, then check that the build got assigned to the desired target channels. If it didn't, we say something went wrong and error out.

What we could do in the interim is detect the result of the promotion build. Then it would say: "the promotion build failed, but the build was still assigned to the target channels". Technically it might not be an error, so you could emit a warning. That would make this case more visible until we fix the root cause (that the symbol publishing isn't integrated into regular publishing)

@hoyosjs
Copy link
Member

hoyosjs commented Nov 4, 2020

@epananth @mmitche do you know if there's any timeline for this?

It bit us pretty silently as we weren't publishing symbols:

cc: @mikem8361 @dotnet/dotnet-diag

@mmitche
Copy link
Member

mmitche commented Nov 4, 2020

@epananth Is working on this right now and she'll have an ETA.

@epananth
Copy link
Member Author

epananth commented Nov 4, 2020

@hoyosis working on integrating symbol publishing with asset publishing, I am going to be testing my changes this week. So next week I should have this ready for you. I will keep you posted on my PR.

@hoyosjs
Copy link
Member

hoyosjs commented Nov 4, 2020

Thanks @epananth

@epananth
Copy link
Member Author

this got fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants