Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support service principal #10043

Merged
merged 63 commits into from
Jul 2, 2024
Merged
Show file tree
Hide file tree
Changes from 57 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
018faec
WIP
agr May 21, 2024
56ec658
Unused args.
agr May 21, 2024
5dda8bf
Removing BlobRequestOptions
agr May 22, 2024
cf3c800
Less usings.
agr May 22, 2024
812662d
BlobContinuationTokens.
agr May 22, 2024
cb98c41
BLob list details wrapper.
agr May 22, 2024
ed59f66
BlobProperties
agr May 23, 2024
99b4180
CopyState
agr May 23, 2024
6d65715
AccessCondition
agr May 23, 2024
f81cc7f
SAS permissions enum
agr May 23, 2024
435d521
Method rename
agr May 23, 2024
217c7e6
ContentEncoding property
agr May 24, 2024
5526e3d
Overwrite fix.
agr May 24, 2024
379d154
Exceptions!
agr May 25, 2024
d08d4e6
Pulling back nuget.jobs changes.
agr May 28, 2024
cdf1063
WrapStorageExceptionAsync
agr May 29, 2024
bfd2449
Missed `?`
agr May 29, 2024
b919573
Catching more cases of blob/container not found.
agr May 29, 2024
c747941
Aligning exception checks with old code.
agr May 29, 2024
5b72650
NuGet.Jobs version update
agr May 29, 2024
97937be
WIP
agr May 31, 2024
20b1ca6
WIP
agr Jun 1, 2024
32428bc
WIP
agr Jun 5, 2024
617b1b4
WIP
agr Jun 5, 2024
e0f4793
WIP
agr Jun 5, 2024
1f68b56
WIP
agr Jun 6, 2024
471e8fe
WIP
agr Jun 6, 2024
e26c9bc
Compiles
agr Jun 6, 2024
f901d4a
Test "fixes"
agr Jun 6, 2024
8fdc168
Connection string fixes.
agr Jun 6, 2024
a35e574
More binding redirects.
agr Jun 6, 2024
76c954e
FetchAttributes after blob upload because calling code expects to be …
agr Jun 7, 2024
290ecc7
Using DownloadContent for DownloadText.
agr Jun 7, 2024
bba277f
Request timeout in CloudBlobClientWrapper constructor.
agr Jun 10, 2024
fc24eb3
FetchAttributes in OpenRead call as there is expectations that proper…
agr Jun 11, 2024
c6eab65
Better(?) way to get properties/metadata for OpenReadAsync.
agr Jun 11, 2024
fe65c21
The expectations are that we should always be able to set HTTP headers.
agr Jun 12, 2024
946c6de
More FetchAttributes
agr Jun 14, 2024
ce5112c
Test fix
agr Jun 14, 2024
d173bda
named parameter
agr Jun 14, 2024
b452051
Dropping Continuation token interface.
agr Jun 14, 2024
a0f87db
Preserving SAS URL for copying.
agr Jun 14, 2024
f88895d
Continuation token.
agr Jun 14, 2024
b3a543f
CloudBlobWrapper.Uri would not contain query
agr Jun 18, 2024
1af9756
PR feedback.
agr Jun 19, 2024
ae380dc
More PR feedback.
agr Jun 19, 2024
c6d0cfa
Less warnings.
agr Jun 20, 2024
e96561e
fix Priviledged typo
SimonCropp Jun 5, 2024
7c88c7e
Bringing back updated NuGet.Jobs packges.
agr Jun 21, 2024
e0174a0
Merge remote-tracking branch 'origin/dev' into agr-st-sdk
agr Jun 25, 2024
51e101b
Cosntructor argument type fix
agr Jun 25, 2024
8c6d504
Tests
agr Jun 25, 2024
a857b1d
Dropped blob client wrapper facts.
agr Jun 25, 2024
d9a1d9b
Not using HasFlag
agr Jun 25, 2024
eb0f21b
Comment update.
agr Jun 25, 2024
f4e9512
support service principal
Jun 26, 2024
8ca81a7
space
Jun 27, 2024
704bbb5
Merge branch 'dev' into arg-st-sdk-geneva-version
Jun 28, 2024
5db5e28
resolve conflict
Jun 28, 2024
dcd25b2
support sp
Jun 28, 2024
124503a
Merge branch 'dev' into arg-st-sdk-geneva-version
Jun 28, 2024
30ed904
update the sp auth
Jun 28, 2024
95c2aaf
s
Jun 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<PropertyGroup>
<NuGetClientPackageVersion>6.9.1</NuGetClientPackageVersion>
<ServerCommonPackageVersion>2.120.0</ServerCommonPackageVersion>
<NuGetJobsPackageVersion>4.3.0-main-9408418</NuGetJobsPackageVersion>
<NuGetJobsPackageVersion>4.3.0-agr-gal-stsdk-9768098</NuGetJobsPackageVersion>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.CodeAnalysis.NetAnalyzers">
Expand Down
38 changes: 12 additions & 26 deletions src/NuGetGallery.Core/Auditing/CloudAuditingService.cs
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,6 @@
using System.Globalization;
using System.IO;
using System.Threading.Tasks;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using Microsoft.WindowsAzure.Storage.Blob.Protocol;
using Newtonsoft.Json;
using NuGetGallery.Auditing.Obfuscation;

Expand Down Expand Up @@ -58,23 +55,16 @@ protected override async Task SaveAuditRecordAsync(string auditData, string reso
{
await WriteBlob(auditData, fullPath, blob);
}
catch (StorageException ex)
catch (CloudBlobContainerNotFoundException)
{
if (ex.RequestInformation?.ExtendedErrorInformation?.ErrorCode == BlobErrorCodeStrings.ContainerNotFound)
{
retry = true;
}
else
{
throw;
}
retry = true;
}

if (retry)
{
// Create the container and try again,
// this time we let exceptions bubble out
await container.CreateIfNotExistAsync(permissions: null);
await container.CreateIfNotExistAsync(enablePublicAccess: false);
await WriteBlob(auditData, fullPath, blob);
}
}
Expand All @@ -89,29 +79,25 @@ private static async Task WriteBlob(string auditData, string fullPath, ISimpleCl
{
try
{
using (var stream = await blob.OpenWriteAsync(AccessCondition.GenerateIfNoneMatchCondition("*")))
using (var stream = await blob.OpenWriteAsync(AccessConditionWrapper.GenerateIfNoneMatchCondition("*")))
using (var writer = new StreamWriter(stream))
{
await writer.WriteAsync(auditData);
}
}
catch (StorageException ex)
catch (CloudBlobConflictException ex)
{
if (ex.RequestInformation != null && ex.RequestInformation.HttpStatusCode == 409)
{
// Blob already existed!
throw new InvalidOperationException(String.Format(
CultureInfo.CurrentCulture,
CoreStrings.CloudAuditingService_DuplicateAuditRecord,
fullPath), ex);
}
throw;
// Blob already existed!
throw new InvalidOperationException(String.Format(
CultureInfo.CurrentCulture,
CoreStrings.CloudAuditingService_DuplicateAuditRecord,
fullPath), ex.InnerException);
}
}

public Task<bool> IsAvailableAsync(BlobRequestOptions options, OperationContext operationContext)
public Task<bool> IsAvailableAsync(CloudBlobLocationMode? locationMode)
{
return _auditContainerFactory().ExistsAsync(options, operationContext);
return _auditContainerFactory().ExistsAsync(locationMode);
}

public override string RenderAuditEntry(AuditEntry entry)
Expand Down
21 changes: 0 additions & 21 deletions src/NuGetGallery.Core/Extensions/StorageExceptionExtensions.cs

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
using System.Linq;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using Microsoft.WindowsAzure.Storage;
using Newtonsoft.Json;
using NuGet.Services.Entities;
using NuGet.Services.FeatureFlags;
Expand Down Expand Up @@ -117,7 +116,7 @@ private async Task<ContentSaveResult> TrySaveInternalAsync(FeatureFlags flags, s
return ContentSaveResult.Ok;
}
}
catch (StorageException e) when (e.IsPreconditionFailedException())
catch (CloudBlobPreconditionFailedException)
{
return ContentSaveResult.Conflict;
}
Expand Down
4 changes: 1 addition & 3 deletions src/NuGetGallery.Core/ICloudStorageStatusDependency.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

using System.Threading.Tasks;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;

namespace NuGetGallery
{
Expand All @@ -13,6 +11,6 @@ namespace NuGetGallery
/// </summary>
public interface ICloudStorageStatusDependency
{
Task<bool> IsAvailableAsync(BlobRequestOptions options, OperationContext operationContext);
Task<bool> IsAvailableAsync(CloudBlobLocationMode? locationMode);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,8 @@
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.InteropServices.ComTypes;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using Microsoft.WindowsAzure.Storage;
using Newtonsoft.Json;
using NuGetGallery.Shared;

Expand Down Expand Up @@ -123,7 +121,7 @@ private async Task<ContentSaveResult> TrySaveInternalAsync(LoginDiscontinuation
return ContentSaveResult.Ok;
}
}
catch (StorageException e) when (e.IsPreconditionFailedException())
catch (CloudBlobPreconditionFailedException)
{
return ContentSaveResult.Conflict;
}
Expand Down
8 changes: 5 additions & 3 deletions src/NuGetGallery.Core/NuGetGallery.Core.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -43,16 +43,18 @@
</ItemGroup>

<ItemGroup>
<PackageReference Include="WindowsAzure.Storage" Version="9.3.3" />

<PackageReference Include="Azure.Core" Version="1.39.0" />
<PackageReference Include="Azure.Identity" Version="1.11.3" />
<PackageReference Include="Azure.Storage.Blobs" Version="12.20.0" />
<PackageReference Include="EntityFramework" Version="6.4.4" />
<PackageReference Include="NuGet.Packaging">
<Version>$(NuGetClientPackageVersion)</Version>
</PackageReference>
<PackageReference Include="NuGet.Services.FeatureFlags">
<Version>$(ServerCommonPackageVersion)</Version>
</PackageReference>
<PackageReference Include="WindowsAzure.Storage">
<Version>9.3.3</Version>
</PackageReference>
</ItemGroup>

<ItemGroup Condition="'$(TargetFramework)' == 'net472'">
Expand Down
8 changes: 3 additions & 5 deletions src/NuGetGallery.Core/Services/AccessConditionWrapper.cs
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

using Microsoft.WindowsAzure.Storage;

namespace NuGetGallery
{
public class AccessConditionWrapper : IAccessCondition
Expand All @@ -28,20 +26,20 @@ public static IAccessCondition GenerateIfMatchCondition(string etag)
{
return new AccessConditionWrapper(
ifNoneMatchETag: null,
ifMatchETag: AccessCondition.GenerateIfMatchCondition(etag).IfMatchETag);
ifMatchETag: etag);
}

public static IAccessCondition GenerateIfNoneMatchCondition(string etag)
{
return new AccessConditionWrapper(
ifNoneMatchETag: AccessCondition.GenerateIfNoneMatchCondition(etag).IfNoneMatchETag,
ifNoneMatchETag: etag,
ifMatchETag: null);
}

public static IAccessCondition GenerateIfNotExistsCondition()
{
return new AccessConditionWrapper(
ifNoneMatchETag: AccessCondition.GenerateIfNotExistsCondition().IfNoneMatchETag,
ifNoneMatchETag: "*",
ifMatchETag: null);
}
}
Expand Down
15 changes: 15 additions & 0 deletions src/NuGetGallery.Core/Services/BlobListContinuationToken.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

namespace NuGetGallery
{
public class BlobListContinuationToken
{
internal BlobListContinuationToken(string continuationToken)
{
ContinuationToken = continuationToken;
}

internal string ContinuationToken { get; }
}
}
10 changes: 4 additions & 6 deletions src/NuGetGallery.Core/Services/BlobResultSegmentWrapper.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,21 @@
// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

using System.Collections.Generic;
using System.Linq;
using Microsoft.WindowsAzure.Storage.Blob;

namespace NuGetGallery
{
public class BlobResultSegmentWrapper : ISimpleBlobResultSegment
{
public BlobResultSegmentWrapper(BlobResultSegment segment)
public BlobResultSegmentWrapper(IReadOnlyList<ISimpleCloudBlob> items, string continuationToken)
{
// For now, assume all of the blobs are block blobs. This library's storage abstraction only allows
// creation of block blobs so it's good enough for now. If another caller created a non-block blob, this
// cast will fail at runtime.
Results = segment.Results.Cast<CloudBlockBlob>().Select(x => new CloudBlobWrapper(x)).ToList();
ContinuationToken = segment.ContinuationToken;
Results = items;
ContinuationToken = new BlobListContinuationToken(continuationToken);
}

public IReadOnlyList<ISimpleCloudBlob> Results { get; }
public BlobContinuationToken ContinuationToken { get; }
public BlobListContinuationToken ContinuationToken { get; }
}
}
Loading