AWS SDK .NET 4.5 "Error unmarshalling response back from AWS. HTTP Status Code: 200 OK" on ListObjectsV2

[cstro2021]

I do not know if this is a bug or not, but any help would be appreciated.

I am getting this error trying to list objects in a directory on a bucket. I cannot list from the root of the bucket as it has more than 1000 objects, so I need to drill farther down into the directory list to get what I want. My code works when I display from the root of the bucket, but when I try to add directories at the end of the bucket to list their contents I get this error. "Error unmarshalling response back from AWS. HTTP Status Code: 200 OK", "Root element is missing" on the ListObjectsV2. This is a public S3 bucket so I have included my code below so others can try it. I am using AWS-SDK-NET45.zip, AWSSDK.Core.dll v3.7.12.11 and compiling as Visual Basic 2019 for .NET 4.8 within an SSIS Script task. This should work, any ideas on what I am doing wrong?

Thanks.

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime
Imports Amazon.S3
Imports Amazon.S3.Model
Imports Amazon.Runtime
Imports Amazon
Imports Amazon.S3.Util
Imports System.Collections.ObjectModel
Imports System.IO

'ScriptMain is the entry point class of the script.  Do not change the name, attributes,
'or parent of this class.
<Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute()>
<System.CLSCompliantAttribute(False)>
Partial Public Class ScriptMain
    Inherits Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase



    Public Sub Main()
        '
        ' Add your code here
        '
        
        Dim filecol As ObservableCollection(Of String)
        Try

            'filecol = ListingFiles("/gov-fpac-rma-pubfs-production/pub/References/actuarial_data_master/2023/")
            'filecol = ListingFiles("/gov-fpac-rma-pubfs-production/") 'Bucket root
            filecol = ListingFiles("/gov-fpac-rma-pubfs-production/pub/")

            Dts.TaskResult = ScriptResults.Success
        Catch ex As Exception
            Console.WriteLine(ex.Message.ToString)
            Dts.TaskResult = ScriptResults.Failure
        End Try
    


    End Sub

#Region "ScriptResults declaration"
    'This enum provides a convenient shorthand within the scope of this class for setting the
    'result of the script.

    'This code was generated automatically.
    Enum ScriptResults
        Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success
        Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
    End Enum

#End Region
    Private Function ListingFiles(bucketName As String, Optional foldername As String = "/") As ObservableCollection(Of String)
        Dim obsv As New ObservableCollection(Of String)
        Dim delimiter As String = "/"
        Dim AWS_ACCESS_KEY As String = "xxxxxxxxxxxx" 'Add your Access Key here
        Dim AWS_SECRET_KEY As String = "xxxxxxxxxxxxxxxxx" ' 'Add your Secret here

        Dim s3config As AmazonS3Config = New AmazonS3Config
        With s3config
            .ForcePathStyle = True
            .RegionEndpoint = RegionEndpoint.USEast1


        End With

        Dim s3Client As AmazonS3Client = New AmazonS3Client(AWS_ACCESS_KEY, AWS_SECRET_KEY, s3config)


        If Not foldername.EndsWith(delimiter) Then
            foldername = String.Format("{0}{1}", foldername, delimiter)
        End If
        Try
            Try
                Dim request As New ListObjectsV2Request() 
                With request
                    .BucketName = bucketName

                End With

                
                Do
                    Dim response As New ListObjectsV2Response()
                    response = s3Client.ListObjectsV2(request)

                    For i As Integer = 1 To response.S3Objects.Count - 1
                        Dim entry As S3Object = response.S3Objects(i)
                        If Not foldername = "/" Then
                            If entry.Key.ToString.StartsWith(foldername) Then
                                Dim replacementstring As String = Replace(entry.Key, foldername, "")
                                If Not replacementstring = "" Then
                                    obsv.Add(replacementstring)
                                End If
                            End If

                        Else
                            obsv.Add(Replace(entry.Key, foldername, ""))
                        End If
                        MessageBox.Show(entry.Key + "  " + entry.LastModified.ToString())
                        'Console.WriteLine("Object - " + entry.Key.ToString())
                        'Console.WriteLine(" Size - " + entry.Size.ToString())
                        'Console.WriteLine(" LastModified - " + entry.LastModified.ToString())
                        'Console.WriteLine(" Storage class - " + entry.StorageClass)
                    Next
                    If (response.IsTruncated) Then
                        request.ContinuationToken = response.NextContinuationToken

                    Else
                        request = Nothing
                    End If
                Loop Until IsNothing(request)
            Catch ex As AmazonS3Exception
                Console.WriteLine(ex.Message.ToString)
                Dts.TaskResult = ScriptResults.Failure
            End Try
        Catch ex As Exception
            Console.WriteLine(ex.Message.ToString)
            Dts.TaskResult = ScriptResults.Failure
        End Try
        Return obsv
    End Function
End Class

[ashishdhingra]

Hi @cstro2021,

Good afternoon.

Looks like you are using the API incorrectly. If you refer service API documentation for S3: ListObjectsV2, the there is a prefix parameter in addition to Bucket parameter. Bucket parameter should be the name of the bucket, not the path down to the specific key (there is no concept of folder in S3, it is represented as a key). So you need to use the Prefix property if you want to narrow down results to specific folders. In your case, below is the minimal example:

Imports System.Collections.ObjectModel
Imports System.Windows.Forms
Imports Amazon
Imports Amazon.S3
Imports Amazon.S3.Model

Module Module1

    Sub Main()
        Dim filecol As ObservableCollection(Of String)
        Try
            filecol = ListingFiles("gov-fpac-rma-pubfs-production", "pub/")
        Catch ex As Exception
            Console.WriteLine(ex.Message.ToString)
        End Try

    End Sub

    Private Function ListingFiles(bucketName As String, Optional prefix As String = "", Optional foldername As String = "/") As ObservableCollection(Of String)
        Dim obsv As New ObservableCollection(Of String)
        Dim delimiter As String = "/"

        Dim s3config As AmazonS3Config = New AmazonS3Config
        With s3config
            .ForcePathStyle = True
        End With

        Dim s3Client As AmazonS3Client = New AmazonS3Client(s3config)


        If Not foldername.EndsWith(delimiter) Then
            foldername = String.Format("{0}{1}", foldername, delimiter)
        End If
        Try
            Try
                Dim request As New ListObjectsV2Request()
                With request
                    .BucketName = bucketName
                End With
                If Not String.IsNullOrWhiteSpace(prefix) Then
                    request.Prefix = prefix
                End If

                Do
                    Dim response As New ListObjectsV2Response()
                    response = s3Client.ListObjectsV2(request)

                    For i As Integer = 1 To response.S3Objects.Count - 1
                        Dim entry As S3Object = response.S3Objects(i)
                        If Not foldername = "/" Then
                            If entry.Key.ToString.StartsWith(foldername) Then
                                Dim replacementstring As String = Replace(entry.Key, foldername, "")
                                If Not replacementstring = "" Then
                                    obsv.Add(replacementstring)
                                End If
                            End If

                        Else
                            obsv.Add(Replace(entry.Key, foldername, ""))
                        End If
                        MessageBox.Show(entry.Key + "  " + entry.LastModified.ToString())
                        'Console.WriteLine("Object - " + entry.Key.ToString())
                        'Console.WriteLine(" Size - " + entry.Size.ToString())
                        'Console.WriteLine(" LastModified - " + entry.LastModified.ToString())
                        'Console.WriteLine(" Storage class - " + entry.StorageClass)
                    Next
                    If (response.IsTruncated) Then
                        request.ContinuationToken = response.NextContinuationToken

                    Else
                        request = Nothing
                    End If
                Loop Until IsNothing(request)
            Catch ex As AmazonS3Exception
                Console.WriteLine(ex.Message.ToString)
            End Try
        Catch ex As Exception
            Console.WriteLine(ex.Message.ToString)
        End Try
        Return obsv
    End Function
End Module

Thanks,
Ashish

[cstro2021]

Thanks I will give it a try, could not find any good examples of this. I was thinking it worked like the GetObject function where you put the Bucket with the Full directory path in the Bucketname to where the file is, and the file name goes into the Entry.Key. Will let you know shortly. Thanks.

[cstro2021]

Ok I added the prefix option to the ListObjectsV2Request as follows and it worked. I was able to get list of files from just the directory I wanted. Thanks for the help, consider this one answered.

Dim request As New ListObjectsV2Request() 'With {.BucketName = bucketName}
                With request
                    .BucketName = "/gov-fpac-rma-pubfs-production"
                    .Prefix = "pub/References/actuarial_data_master/2023/2023_"
                End With

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.