-
Notifications
You must be signed in to change notification settings - Fork 852
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TransferUtility.UploadAsync() throws error: "content would exceed Content-Length" when update to .NET8 #3201
Comments
Thank you for reporting the issue. Unfortunately, I am unable to reproduce the exception using your code sample. Can you please elaborate on how the Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.SystemDiagnostics;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener()); Regards, |
Hello, var boundary = Request.GetBoundary();
var reader = new MultipartReader(boundary, Request.Body, _bufferSize);
MultipartSection section;
while ((section = await reader.ReadNextSectionAsync(cancellationToken)) != null)
{
var contentDisposition = section.GetContentDispositionHeader();
if (contentDisposition.IsFileDisposition())
{
var fileSection = section.AsFileSection();
_logger.LogInformation("Received file to upload: {0}", fileSection.FileName);
Validate.FormFile(fileSection.FileStream, fileSection.FileName, fileSection.Section.ContentType);
var stream = new TypedStream
{
ContentType = fileSection.Section.ContentType,
ContentStream = fileSection.FileStream,
Filename = fileSection.FileName
};
await _documentManager.UploadDocument(tenantId, new DocumentReference(documentId), name, stream);
}
} the public async Task UploadDocument(Guid tenantId, IDocumentReference document,
string name, ITypedStream stream)
{
//... other code logic
await _storageManager.UploadContentStream(tenantId, document.DocumentId, name, stream);
} the public async Task UploadContentStream(Guid tenantId, Guid id, string name, ITypedStream typedStream, CancellationToken cancellationToken = default)
{
var fullPath = $"{tenantId}/{id}/{name.ToLower()}";
await _blobStorageRepo.UploadAsync(typedStream.ContentStream, fullPath, typedStream.ContentType, cancellationToken);
_logger.LogInformation("Content stream has been uploaded");
} the |
One more thing. I altered the code var boundary = Request.GetBoundary();
var reader = new MultipartReader(boundary, Request.Body, _bufferSize);
MultipartSection section;
while ((section = await reader.ReadNextSectionAsync(cancellationToken)) != null)
{
var contentDisposition = section.GetContentDispositionHeader();
if (contentDisposition.IsFileDisposition())
{
var fileSection = section.AsFileSection();
_logger.LogInformation("Received file to upload: {0}", fileSection.FileName);
Validate.FormFile(fileSection.FileStream, fileSection.FileName, fileSection.Section.ContentType);
var stream = new TypedStream
{
ContentType = fileSection.Section.ContentType,
ContentStream = fileSection.FileStream,
Filename = fileSection.FileName
};
await _documentManager.UploadDocument(tenantId, new DocumentReference(documentId), name, stream);
}
} to this: var fStream = System.IO.File.OpenRead(@"C:\original0001.pdf");
var stream = new TypedStream
{
ContentType = "application/pdf",
ContentStream = fStream,
Filename = "original0001.pdf"
};
await _documentManager.UploadDocument(tenantId, new DocumentReference(documentId), name, stream); Basically I do not read the stream from the multipart form from the request but read a local file, and the Upload is successful. If this can help narrow the possible cause of the problem. Regards, |
I'm also seeing the same issue. |
Looks like |
Thanks for providing more details. Support for uploading unseekable streams in However, S3 (3.7.305.8) may have introduced some conflicts with the above feature release. Will discuss this with the team and keep you posted. Regards, |
Hi! I noticed that the latest version of the AWSSDK.S3 package where this - sort of - works is: 3.7.305.7 while also using .NET 8 From then onwards we get the aforementioned error. It is also worth noting that the following conditions must be met for this to work (please verify before trying them):
Hope this helps in any sort of way. |
@julian-dimitroff Good afternoon. Using the latest version of AWSSDK.S3 (version @{
ViewData["Title"] = "Home Page";
}
<div class="text-center">
<form method="post" enctype="multipart/form-data" asp-controller="FileUpload" asp-action="Index">
<div class="form-group">
<div class="col-md-10">
<p>Upload one or more files using this form:</p>
<input type="file" name="files" multiple />
</div>
</div>
<div class="form-group">
<div class="col-md-10">
<input type="submit" value="Upload" />
</div>
</div>
</form>
</div> HomeController.cs using Microsoft.AspNetCore.Mvc;
using TestWebAppNetCore.Models;
namespace TestWebAppNetCore.Controllers
{
public class HomeController : Controller
{
public IActionResult Index()
{
return View();
}
}
} FileUploadController.cs using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Transfer;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.WebUtilities;
using System;
using System.ComponentModel.DataAnnotations;
using System.Threading;
using System.Xml.Linq;
namespace TestWebAppNetCore.Controllers
{
public class FileUploadController : Controller
{
[HttpPost("FileUpload")]
public async Task<IActionResult> Index(List<IFormFile> files)
{
long size = files.Sum(f => f.Length);
foreach (var formFile in files)
{
if (formFile.Length > 0)
{
AmazonS3Client amazonS3Client = new AmazonS3Client();
await UploadAsync(amazonS3Client, "testbucket-issue3201", formFile.OpenReadStream(), formFile.FileName, formFile.ContentType);
}
}
return Ok(new { count = files.Count, size });
}
public async Task UploadAsync(AmazonS3Client amazonS3Client, string bucketName, Stream sourceStream, string fullPath, string contentType, CancellationToken cancellationToken = default(CancellationToken))
{
try
{
TransferUtility transferUtility = new TransferUtility(amazonS3Client);
await transferUtility.UploadAsync(sourceStream, bucketName, fullPath, cancellationToken);
}
catch (AmazonServiceException ex3)
{
AmazonServiceException ex2 = ex3;
throw new Exception($"Got AmazonServiceException in {"UploadAsync"} while uploading object to '{fullPath}'.", ex2);
}
catch (Exception ex4)
{
Exception ex = ex4;
throw new Exception($"Got exception in {"UploadAsync"} while uploading object to '{fullPath}'.", ex);
}
}
}
} Kindly note that in above code I'm relying on ASP.NET Model binding feature to bind incoming files to list of Could you please:
Thanks, |
Could you please retry but, this time, use a stream instead of a FormFile as a parameter. This issue happens mainly when working with the stream directly (such as in multipart/content) and then trying to upload a stream that has a Length of 0. |
Okay, I'll try to setup a sample project that is able to reproduce the problem without this internal nuget package. Have a great day! |
Hello again @JFSiller @ashishdhingra I made a simple setup using .NET 8 and AWSSDK.S3 v.3.7.307: Hope that helps reproduce the problem Best regards |
@julian-dimitroff Thanks for providing the reproduction code and steps. I was able to reproduce the issue where it gave error at System.IO.Stream.<<CopyToAsync>g__Core|27_0>d.MoveNext()
at System.Net.Http.HttpContent.<<CopyToAsync>g__WaitAsync|56_0>d.MoveNext()
at System.Net.Http.HttpConnection.<SendRequestContentAsync>d__61.MoveNext()
at System.Net.Http.HttpConnection.<SendRequestContentWithExpect100ContinueAsync>d__62.MoveNext()
at System.Net.Http.HttpConnection.<SendAsync>d__57.MoveNext()
at System.Net.Http.HttpConnectionPool.<SendWithVersionDetectionAndRetryAsync>d__89.MoveNext()
at System.Net.Http.DiagnosticsHandler.<SendAsyncCore>d__10.MoveNext()
at System.Net.Http.HttpClient.<<SendAsync>g__Core|83_0>d.MoveNext()
at Amazon.Runtime.HttpWebRequestMessage.<GetResponseAsync>d__20.MoveNext()
at Amazon.Runtime.Internal.HttpHandler`1.<InvokeAsync>d__9`1.MoveNext()
at Amazon.Runtime.Internal.RedirectHandler.<InvokeAsync>d__1`1.MoveNext()
at Amazon.Runtime.Internal.Unmarshaller.<InvokeAsync>d__3`1.MoveNext()
at Amazon.S3.Internal.AmazonS3ResponseHandler.<InvokeAsync>d__1`1.MoveNext()
at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1.MoveNext()
at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1.MoveNext()
at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
at Amazon.Runtime.Internal.Signer.<InvokeAsync>d__1`1.MoveNext()
at Amazon.S3.Internal.S3Express.S3ExpressPreSigner.<InvokeAsync>d__5`1.MoveNext()
at Amazon.Runtime.Internal.EndpointDiscoveryHandler.<InvokeAsync>d__2`1.MoveNext()
at Amazon.Runtime.Internal.EndpointDiscoveryHandler.<InvokeAsync>d__2`1.MoveNext()
at Amazon.Runtime.Internal.CredentialsRetriever.<InvokeAsync>d__7`1.MoveNext()
at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
at Amazon.S3.Internal.AmazonS3ExceptionHandler.<InvokeAsync>d__1`1.MoveNext()
at Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext()
at Amazon.Runtime.Internal.MetricsHandler.<InvokeAsync>d__1`1.MoveNext()
at Amazon.S3.Transfer.Internal.SimpleUploadCommand.<ExecuteAsync>d__10.MoveNext()
at AWS_Tests.Controllers.WeatherForecastController.<UploadDocumentWithTenantId>d__6.MoveNext() in D:\source\repros\AWS.Tests\AWS Tests\AWS.Tests\Controllers\WeatherForecastController.cs:line 89 Based on stack trace, Thanks, |
@julian-dimitroff While debugging, I noticed that One weird behavior noticed is that if I temporarily copy the contents of In other words, copy this code: var memoryStream = new MemoryStream();
fileSection.FileStream.CopyTo(memoryStream); after ValidateFormFile(fileSection.FileStream, fileSection.FileName, fileSection.Section.ContentType);
fileSection.FileStream.Position = 0;
... So there appears to be issue with Thanks, |
@ashishdhingra Thank you again for your effort. Best regards, |
@julian-dimitroff We have reviewed this created backlog item for this to be groomed for taking further course of action. |
@ashishdhingra Again thank you for your time and effort! Regards, |
we also have this issue with upload failing for non-seekable streams. From this. From the above, you are suggesting to use the low level API for non-seekable streams. But from this documentation you are saying non-seekable streams should work with the high level API So which one is it? Do you expect the high level API to work with non-seekable streams or you expect us to use the low level API for future versions? Thank you in advance |
Describe the bug
I recently migrated our project to .NET8 and updated all the packages to latest version, but I keep getting a strange error when I try to upload a stream to Amazon S3 storage. I try to upload a relatively small file ~500 KB.
Expected Behavior
Should not throw exception in
at System.IO.Stream.<CopyToAsync>
Current Behavior
It throws exception:
Reproduction Steps
Possible Solution
No response
Additional Information/Context
When I revert my project to .NET6 the issue disappears. If needed I can provide additional information
AWS .NET SDK and/or Package version used
AWSSDK.S3, Version=3.3.0.0
Targeted .NET Platform
.NET8
Operating System and version
Windows 11, Debian 12
The text was updated successfully, but these errors were encountered: