Skip to content
This repository has been archived by the owner on Jul 30, 2024. It is now read-only.
/ NuGet.Jobs Public archive

[Revalidation] Pause if the ingestion pipeline is unhealthy #504

Merged
merged 2 commits into from
Aug 1, 2018

Conversation

loic-sharma
Copy link
Contributor

@loic-sharma loic-sharma commented Aug 1, 2018

This integrates with @scottbommarito's status work to pause the job if the ingestion pipeline is unhealthy.

Part of https://github.com/NuGet/Engineering/issues/1128

@@ -115,7 +115,7 @@
<Version>2.27.0</Version>
</PackageReference>
<PackageReference Include="NuGetGallery.Core">
<Version>4.4.5-dev-35743</Version>
<Version>4.4.5-loshar-health-36421</Version>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will update this before merging this change.

@loic-sharma loic-sharma changed the title [Revalidation] Slow down if the ingestion pipeline is unhealthy [Revalidation] Pause if the ingestion pipeline is unhealthy Aug 1, 2018
"Could not find component path {ComponentPath} in status health blob!",
_config.ComponentPath);

throw new InvalidOperationException($"Could not find component path '{_config.ComponentPath}' in status health blob!");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if this exception is thrown, how will the caller treat it? Can we just return false in this case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this exception is thrown, it'll be caught and reported by the JobRunner. The job would then exit and restart.

Returning false if the path can't be found sounds fine to me too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

.Setup(s => s.GetFileAsync(_config.ContainerName, _config.StatusBlobName))
.ThrowsAsync(new Exception());

await Assert.ThrowsAsync<Exception>(() => _target.IsHealthyAsync());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit Assert.Same on the exception instance to verify no wrapping.

Copy link
Contributor Author

@loic-sharma loic-sharma Aug 1, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

private readonly HealthConfiguration _config;
private readonly HealthService _target;

public HealthServiceFacts()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test th missing component path?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless I'm misunderstanding you, this should be covered by ThrowsIfStatusBlobIsMissingExpectedPath.

Copy link
Contributor Author

@loic-sharma loic-sharma Aug 1, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, perfect.

Copy link
Member

@joelverhagen joelverhagen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Address comments then :shipit:

@loic-sharma
Copy link
Contributor Author

Verified this on DEV with a manually created incident.

@loic-sharma loic-sharma merged commit 0bf9d92 into dev Aug 1, 2018
@loic-sharma loic-sharma deleted the loshar-health branch August 1, 2018 21:26
joelverhagen pushed a commit that referenced this pull request Oct 26, 2020
This integrates with @scottbommarito's status work to pause the job if the ingestion pipeline is unhealthy.

Part of https://github.com/NuGet/Engineering/issues/1128
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants