Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using Azure.Storage.Blobs for storage access #10018

Merged
merged 64 commits into from
Jun 27, 2024
Merged

Using Azure.Storage.Blobs for storage access #10018

merged 64 commits into from
Jun 27, 2024

Conversation

agr
Copy link
Contributor

@agr agr commented Jun 7, 2024

Progress on https://github.com/NuGet/Engineering/issues/5335

Built on top of #9981 (draft PR), so all the SDK isolation work done there is present here.

Some choices made:

  • MaximumExecutionTime support that existed in WindowsAzure.Storage is dropped. Consumers would have to produce cancellation tokens with time limits if needed;
  • There are no longer ambient request timeouts in any of SDK objects (internally, all configuration objects passed in are consumed immediately to produce HTTP pipeline, so nothing is configurable after objects are created), to support overriding network timeouts for individual blob operations support for recreating whole stack of objects (service/container/blob) was added.
  • Previously ETag updates were implicitly captured by storage SDK and a lot of code assumes fresh ETags are available after many operations, new SDK has updated ETag in some responses (which are all different types incompatible with each other), so call-specific etag capturing had to be added where available.
  • Otherwise, I tried to make minimal changes to API surface we exposed, so that we don't spend time reworking downstream consumers of wrappers.

There are table wrappers using old SDK left that I haven't touched yet. Those are going to be addressed separately.

I commented out some tests and need some input on what to do with them.

The code was deployed to dev and passed E2E and functional tests.

There likely going to be more changes as this update is propagated to other repositories.

var ex = await Assert.ThrowsAsync<StorageException>(() => file.DownloadToStreamAsync(destination));
Assert.Contains("timeout", ex.Message);
}

[BlobStorageFact]
public async Task OpenWriteAsyncReturnsWritableStream()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know where these integration tests are run. If they are not run anywhere, it might be worth setting up to run them locally (I think there is some connection string setup) or deleting them. Perhaps they'll catch a bug if they are run.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed dependency on old SDK in those tests, ran/fixed some, removed one that no longer passes due to the way new SDK works, verified that we don't really rely on the behavior the test checks.

joelverhagen
joelverhagen previously approved these changes Jun 25, 2024
drewgillies
drewgillies previously approved these changes Jun 25, 2024
Copy link
Contributor

@drewgillies drewgillies left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Outstanding, @agr!

@agr agr dismissed stale reviews from drewgillies and joelverhagen via eb0f21b June 25, 2024 23:04
erdembayar
erdembayar previously approved these changes Jun 25, 2024
Copy link
Contributor

@erdembayar erdembayar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏 👏 👏 Great work.

_blobClient = new Lazy<BlobServiceClient>(CreateBlobServiceClient);
}

public static CloudBlobClientWrapper UsingMsi(string storageConnectionString, string clientId = null)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is storageConnectString is necessary if we use MSI? storageConnectionString with "SharedAccessSignature=" will work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically, we only need the host name. But I'm not sure yet how drastic I want to go with configuration changes that would require. For MSI, I'd guess, a trivial connection string like BlobEndpoint=https://storageaccount.blob.core.windows.net might be enough, I haven't tested it though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like not working, I could test it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably can't. it threw exception: No valid combination of account information found.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think at least we could remove the SAS in connection string when using MSI

@agr agr merged commit 17b24e4 into dev Jun 27, 2024
2 checks passed
@agr agr deleted the agr-st-sdk branch June 27, 2024 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants