Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DynamoDB rate limit error without throttling on the table #1665

Closed
pcolazurdo opened this issue Apr 9, 2022 · 16 comments
Closed

DynamoDB rate limit error without throttling on the table #1665

pcolazurdo opened this issue Apr 9, 2022 · 16 comments

Comments

@pcolazurdo
Copy link

pcolazurdo commented Apr 9, 2022 •
edited
Loading

Describe the bug

I'm trying to debug a problem where my application is crashing on leaked GoRoutines. While trying to root cause the problem I've found some issues that seem to be related with the SDK:

I'm launching 500 GoRoutines to do a PutItem (different PK on each). Only between 200-350 succeed (depending on the run), with the rest returning:

operation error DynamoDB: PutItem, failed to get rate limit token, retry quota exceeded, 0 available, 5 requested

Interestingly, the DDB table is configured as On Demand, and CloudWatch doesn't show any Throttling errors

Also, when running goleak.VerifyNone(t), the library detects lots of leaked GoRoutine in different states in some part of http libraries. I've looked into #1434 but the PutItem doesn't seem to have any response with io.ReadCloser so I'm assuming these aren't related.

Expected Behavior

  1. I'm assuming rate limit errors should be caused by service throttle errors (DynamoDB) but I can see no evidence in DynamoDB that this is the cause of these errors, so more clarity would be useful (what rate limits am I hitting?)
  2. I would expect that there should be no Goroutine leaks at the end of the operations. This doesn't seem to be the case.

Current Behavior

  1. Receiving error operation error DynamoDB: PutItem, failed to get rate limit token, retry quota exceeded, 0 available, 5 requested without any throttling evidenced in DynamoDB CloudWatch metrics
  2. Goroutine leaks

Reproduction Steps

Sample below includes commented code to customise the http setting which I used to see how that could change behaviour but couldn't find any settings that would solve the perceived problem.

package database

import (
	"context"
	"crypto/md5"
	"crypto/rand"
	"encoding/hex"
	"fmt"
	"log"

	// "net"
	"runtime"
	"sync"
	"testing"
	"time"

	"github.com/aws/aws-sdk-go-v2/aws"
	// "github.com/aws/aws-sdk-go-v2/aws/transport/http"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
	"github.com/aws/aws-sdk-go-v2/service/dynamodb"
	"github.com/stretchr/testify/assert"
	"go.uber.org/goleak"
)

// func TestMain(m *testing.M) {
// 	goleak.VerifyTestMain(m)
// }

func TestConnectionReuseForPutContentHash(t *testing.T) {
	defer goleak.VerifyNone(t)

	ctx := context.Background()
	ctx, cancel := context.WithCancel(ctx)
	svc := databaseConnection(ctx)
	table := "testtable"

	n := runtime.NumGoroutine()

	var wg sync.WaitGroup
	limit := 500
	success := make(chan int, limit)
	s := 0
	go func() {
		for i := range success {
			s = s + i
		}
	}()

	for i := 0; i < limit; i++ {
		wg.Add(1)
		go func(c chan int) {
			defer wg.Done()
			entry := NewHashEntry(
				fakeHash(),
				"175767cb005f8809beb731fd853d0343",
			)
			err := putItem(ctx, svc, table, entry)
			if err == nil {
				c <- 1
			}

		}(success)
	}
	wg.Wait()
	cancel()
	close(success)
	assert.Equal(t, n, runtime.NumGoroutine(), "Number of GoRoutines is different to expected")
	assert.Equal(t, limit, s, "Number of succesful items processed is different to requested")
	// assert.Equal(t, true, false, "forced assert to be sure the test is performing")
}

func fakeHash() string {
	hash := md5.New()
	// used to create a hard enough to repeat hash and avoid PK problems
	s := fmt.Sprintf("%d%s", time.Now().UnixNano(), pseudo_uuid())

	hash.Write([]byte(s))
	hashInBytes := hash.Sum(nil)[:16]
	returnHashString := hex.EncodeToString(hashInBytes)
	return returnHashString
}

// Note - NOT RFC4122 compliant
func pseudo_uuid() (uuid string) {
	b := make([]byte, 16)
	_, err := rand.Read(b)
	if err != nil {
		fmt.Println("Error: ", err)
		return
	}

	uuid = fmt.Sprintf("%X-%X-%X-%X-%X", b[0:4], b[4:6], b[6:8], b[8:10], b[10:])

	return
}

func databaseConnection(ctx context.Context) *dynamodb.Client {
	// Initialize a session that the SDK will use to load
	// credentials from the shared credentials file ~/.aws/credentials
	// and region from the shared configuration file ~/.aws/config.
	// httpCustomClient := http.
	// 	NewBuildableClient().
	// 	WithTimeout(time.Second * 5).
	// 	WithDialerOptions(func(d *net.Dialer) {
	// 		d.KeepAlive = -1
	// 		d.Timeout = time.Millisecond * 1000
	// 	})
	// cfg, err := config.LoadDefaultConfig(ctx, config.WithHTTPClient(httpCustomClient))
	cfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		log.Printf("unable to load SDK config, %v", err)
	}

	return dynamodb.NewFromConfig(cfg)
}

// Generic function to record a generic object into DDB
func putItem(ctx context.Context, svc *dynamodb.Client, table string, item interface{}) error {

	av, err := attributevalue.MarshalMap(item)
	if err != nil {
		log.Printf("Got error marshalling new item: %s", err)
		return err
	}

	tableName := table

	input := &dynamodb.PutItemInput{
		Item:      av,
		TableName: aws.String(tableName),
	}

	_, err = svc.PutItem(ctx, input)
	if err != nil {
		log.Printf("Got error calling PutItem: %s", err)
	}

	return err

}

func NewHashEntry(hashvalue string, filenamehash string) *HashEntry {
	return &HashEntry{
		Pk: hashvalue,
		Sk: filenamehash,
	}
}

type HashEntry struct {
	Pk string `dynamodbav:"pk"`
	Sk string `dynamodbav:"sk"`
}

Possible Solution

No response

Additional Information/Context

Error is reproducible in both Linux and MacOS

AWS Go SDK version used

Go 1.18 (and Go 1.16)

Compiler and Version used

go version go1.18 darwin/amd64

Operating System and version

MacOs 12.2.1 and Amazon Linux 2
@pcolazurdo pcolazurdo added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Apr 9, 2022
@pcolazurdo
Copy link
Author

Setting KeepAlive to -1 didn't make any difference.

Adding some pprof information:

mem.prof -raw

PeriodType: space bytes
Period: 524288
Time: 2022-04-14 15:33:00.631651 +0100 IST
Samples:
alloc_objects/count alloc_space/bytes[dflt] inuse_objects/count inuse_space/bytes
         18     538754          0          0: 1 2 3 4 
                bytes:[28672]
         64     528394          0          0: 5 2 3 4 
                bytes:[8192]
          2     666237          0          0: 6 7 8 9 10 2 3 4 
                bytes:[262144]
          1     924248          0          0: 11 8 9 10 2 3 4 
                bytes:[663552]
       2521     524392          0          0: 12 13 3 4 
                bytes:[208]
         11     549247         11     549247: 14 15 16 17 18 19 20 21 22 23 
                bytes:[49152]
         16     540842          0          0: 24 25 26 27 28 
                bytes:[32768]
         24     535243          0          0: 29 30 31 32 33 34 35 18 36 37 38 27 28 
                bytes:[21760]
        128     526338          0          0: 39 40 41 42 43 38 27 28 
                bytes:[4096]
       6554     524328          0          0: 44 45 43 38 27 28 
                bytes:[80]
        128     526338          0          0: 46 43 38 27 28 
                bytes:[4096]
        228     525440          0          0: 47 48 49 38 27 28 
                bytes:[2304]
        391    1051266          0          0: 50 48 49 38 27 28 
                bytes:[2688]
         17    1115476          0          0: 51 52 43 38 27 28 
                bytes:[65536]
       4096     524352          0          0: 53 54 55 56 57 58 57 59 60 57 61 57 62 63 64 65 66 67 66 68 66 69 66 70 63 71 72 73 74 73 75 
                bytes:[128]
        372     524992          0          0: 76 77 55 56 57 58 57 59 60 57 61 57 62 63 64 65 66 67 66 68 66 69 66 70 63 71 72 73 74 
                bytes:[1408]
        455     524864          0          0: 53 54 55 56 57 58 57 59 60 57 61 57 62 63 64 65 66 67 66 68 66 69 66 70 63 71 72 73 74 73 75 
                bytes:[1152]
       8192     524320          0          0: 78 79 55 56 57 58 57 59 60 57 61 57 62 63 64 65 66 67 66 68 66 69 66 70 63 71 72 73 74 73 75 
                bytes:[64]
       1638     524448          0          0: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 
                bytes:[320]
       6554     524328          0          0: 95 96 97 85 86 87 88 89 90 91 92 93 94 
                bytes:[80]
       1638     524448          0          0: 98 99 100 101 102 92 93 94 
                bytes:[320]
      10923     524312          0          0: 103 104 85 86 87 88 89 90 91 92 93 94 
                bytes:[48]
        910     524576          0          0: 105 106 107 108 109 110 111 112 91 92 93 94 
                bytes:[576]
       5461     524336          0          0: 113 114 115 116 117 118 119 120 121 122 123 124 
                bytes:[96]
      10923     524312          0          0: 103 125 85 86 87 88 89 90 91 92 93 94 
                bytes:[48]
        585     524736          0          0: 126 127 107 108 109 110 111 112 91 92 93 94 
                bytes:[896]
        586    1050369          0          0: 128 129 130 87 88 89 90 91 92 93 94 
                bytes:[1792]
       2731     524384          0          0: 131 104 85 132 90 91 92 93 94 
                bytes:[192]
       3277     524368          0          0: 133 134 90 91 92 93 94 
                bytes:[160]
        196    1053961          0          0: 135 136 91 92 93 94 
                bytes:[5376]
      32769    1048608          0          0: 95 137 125 85 86 87 88 89 90 91 92 93 94 
                bytes:[32]
       5461     524336          0          0: 138 
                bytes:[96]
        585     524736          0          0: 128 129 130 87 88 89 90 91 92 93 94 
                bytes:[896]
        404    2637793          0          0: 39 40 139 140 141 136 91 92 93 94 
                bytes:[6528]
      10923     524312          0          0: 131 104 85 132 90 91 92 93 94 
                bytes:[48]
       5461     524336          0          0: 142 143 136 91 92 93 94 
                bytes:[96]
      32768     524296          0          0: 144 125 85 132 90 91 92 93 94 
                bytes:[16]
        490    2634902         98     526980: 39 40 145 146 141 136 91 92 93 94 
                bytes:[5376]
        745    1049984          0          0: 147 85 86 87 88 89 90 91 92 93 94 
                bytes:[1408]
       5461     524336          0          0: 148 149 150 151 152 153 93 94 
                bytes:[96]
        910     524576          0          0: 39 40 139 154 141 155 93 94 
                bytes:[576]
      16384     524304          0          0: 156 157 93 94 
                bytes:[32]
       5461     524336          0          0: 148 158 150 151 159 150 151 152 153 93 94 
                bytes:[96]
      10923     524312      10923     524312: 160 161 115 116 117 118 119 120 121 122 123 124 
                bytes:[48]
       3641     524360          0          0: 162 163 164 165 166 167 168 167 169 167 170 167 171 167 172 167 173 167 174 167 175 167 176 63 177 178 57 58 57 59 60 
                bytes:[144]
      32768     524296          0          0: 179 54 55 56 57 58 57 59 60 57 61 57 62 63 64 65 66 67 66 68 66 69 66 70 63 71 72 73 74 73 75 73 180 
                bytes:[16]
        745     524640          0          0: 181 163 164 165 166 167 168 167 169 167 170 167 171 167 172 167 173 167 174 167 175 167 176 63 177 178 57 58 57 182 
                bytes:[704]
      16384     524304          0          0: 183 184 185 115 116 117 118 119 120 121 122 123 124 
                bytes:[32]
      10923     524312          0          0: 181 163 164 165 166 167 168 167 169 167 170 167 171 167 172 167 173 167 174 167 175 167 176 63 177 178 57 58 57 59 
                bytes:[48]
        372     524992          0          0: 186 187 55 56 57 58 57 59 60 57 61 57 62 63 64 65 66 67 66 68 66 69 66 70 63 71 72 73 74 73 75 
                bytes:[1408]
       4096     524352          0          0: 188 187 55 56 57 58 57 59 60 57 61 57 62 63 64 65 66 67 66 68 66 69 66 70 63 71 72 73 74 73 75 
                bytes:[128]
       5461     524336          0          0: 189 190 191 192 193 194 165 166 167 168 167 169 167 170 167 171 167 172 167 173 167 174 167 175 167 176 63 177 178 57 58 
                bytes:[96]
        745     524640          0          0: 195 55 56 57 58 57 59 60 57 61 57 62 63 64 65 66 67 66 68 66 69 66 70 63 71 72 73 74 73 180 
                bytes:[704]
      16384     524304          0          0: 196 197 198 199 122 123 124 
                bytes:[32]
       8192     524320          0          0: 200 201 202 203 55 56 57 58 57 59 60 57 61 57 62 63 64 65 66 67 66 68 66 69 66 70 63 71 72 73 74 73 180 
                bytes:[64]
       5461     524336          0          0: 204 205 190 191 192 193 194 165 166 167 168 167 169 167 170 167 171 167 172 167 173 167 174 167 175 167 176 63 177 178 57 182 
                bytes:[96]
       1490    1049280          0          0: 206 207 208 193 194 165 166 167 168 167 169 167 170 167 171 167 172 167 173 167 174 167 175 167 176 63 177 178 
                bytes:[704]
        586    1050369          0          0: 209 54 55 56 57 58 57 59 60 57 61 57 62 63 64 65 66 67 66 68 66 69 66 70 63 71 72 73 74 73 75 
                bytes:[1792]
       4681     524344          0          0: 210 211 212 213 214 215 216 92 93 94 
                bytes:[112]
       1489     524464          0          0: 217 73 74 73 75 73 218 63 219 220 221 222 221 223 221 224 63 225 63 226 227 228 229 
                bytes:[352]
      10923     524312          0          0: 181 230 231 73 74 73 75 73 218 63 219 220 221 222 221 223 221 224 63 225 63 226 227 228 229 
                bytes:[48]
       1820     524432          0          0: 232 233 234 227 228 229 
                bytes:[288]
      10923     524312          0          0: 235 236 237 238 239 73 218 63 219 220 221 222 221 223 221 224 63 225 63 226 227 228 229 
                bytes:[48]
      21845     524300          0          0: 240 241 242 229 
                bytes:[24]
      10923     524312          0          0: 243 244 245 246 247 248 249 250 234 227 228 229 
                bytes:[48]
       1489     524464          0          0: 251 252 234 227 228 229 
                bytes:[352]
       2048     524416          0          0: 253 254 63 219 220 221 222 221 223 221 224 63 225 63 226 227 228 229 
                bytes:[256]
       1820     524432          0          0: 232 255 234 227 228 229 
                bytes:[288]
       1820     524432          0          0: 232 256 234 227 228 229 
                bytes:[288]
      10923     524312          0          0: 257 28 
                bytes:[48]
      32768     524296          0          0: 258 259 260 
                bytes:[16]
       2979     524376          0          0: 261 262 263 
                bytes:[176]
          1    1212696          0          0: 264 265 266 267 268 269 
                bytes:[1048576]
        819     524608          0          0: 270 271 272 273 274 275 276 277 278 278 278 278 278 279 
                bytes:[640]
        455     524864        455     524864: 280 281 282 283 284 285 286 287 288 
                bytes:[1152]
       3782    1573488       3782    1573488: 289 290 291 292 
                bytes:[416]
Locations
     1: 0x1138887 M=1 runtime/pprof.(*profileBuilder).stringIndex /usr/local/Cellar/go/1.18.1/libexec/src/runtime/pprof/proto.go:137 s=0
             runtime/pprof.(*profileBuilder).emitLocation /usr/local/Cellar/go/1.18.1/libexec/src/runtime/pprof/proto.go:569 s=0
     2: 0x1137391 M=1 runtime/pprof.(*profileBuilder).appendLocsForStack /usr/local/Cellar/go/1.18.1/libexec/src/runtime/pprof/proto.go:435 s=0
     3: 0x1136c70 M=1 runtime/pprof.(*profileBuilder).build /usr/local/Cellar/go/1.18.1/libexec/src/runtime/pprof/proto.go:370 s=0
     4: 0x113317a M=1 runtime/pprof.profileWriter /usr/local/Cellar/go/1.18.1/libexec/src/runtime/pprof/pprof.go:814 s=0
     5: 0x1138a75 M=1 runtime/pprof.(*profileBuilder).stringIndex /usr/local/Cellar/go/1.18.1/libexec/src/runtime/pprof/proto.go:136 s=0
             runtime/pprof.(*profileBuilder).emitLocation /usr/local/Cellar/go/1.18.1/libexec/src/runtime/pprof/proto.go:571 s=0
     6: 0x11237c4 M=1 compress/flate.(*compressor).init /usr/local/Cellar/go/1.18.1/libexec/src/compress/flate/deflate.go:596 s=0
     7: 0x1123ea7 M=1 compress/flate.NewWriter /usr/local/Cellar/go/1.18.1/libexec/src/compress/flate/deflate.go:671 s=0
     8: 0x112dd86 M=1 compress/gzip.(*Writer).Write /usr/local/Cellar/go/1.18.1/libexec/src/compress/gzip/gzip.go:191 s=0
     9: 0x1133e4f M=1 runtime/pprof.(*profileBuilder).flush /usr/local/Cellar/go/1.18.1/libexec/src/runtime/pprof/proto.go:145 s=0
    10: 0x11386b2 M=1 runtime/pprof.(*profileBuilder).emitLocation /usr/local/Cellar/go/1.18.1/libexec/src/runtime/pprof/proto.go:575 s=0
    11: 0x1123e8e M=1 compress/flate.NewWriter /usr/local/Cellar/go/1.18.1/libexec/src/compress/flate/deflate.go:670 s=0
    12: 0x1136089 M=1 runtime/pprof.allFrames /usr/local/Cellar/go/1.18.1/libexec/src/runtime/pprof/proto.go:213 s=0
    13: 0x1137234 M=1 runtime/pprof.(*profileBuilder).appendLocsForStack /usr/local/Cellar/go/1.18.1/libexec/src/runtime/pprof/proto.go:419 s=0
    14: 0x10b9e3b M=1 fmt.(*buffer).write /usr/local/Cellar/go/1.18.1/libexec/src/fmt/print.go:78 s=0
             fmt.(*fmt).pad /usr/local/Cellar/go/1.18.1/libexec/src/fmt/format.go:92 s=0
    15: 0x10bad97 M=1 fmt.(*fmt).fmtBs /usr/local/Cellar/go/1.18.1/libexec/src/fmt/format.go:365 s=0
    16: 0x10bdee4 M=1 fmt.(*pp).fmtBytes /usr/local/Cellar/go/1.18.1/libexec/src/fmt/print.go:486 s=0
    17: 0x10bf64f M=1 fmt.(*pp).printArg /usr/local/Cellar/go/1.18.1/libexec/src/fmt/print.go:696 s=0
    18: 0x10c2307 M=1 fmt.(*pp).doPrintf /usr/local/Cellar/go/1.18.1/libexec/src/fmt/print.go:1026 s=0
    19: 0x10bc674 M=1 fmt.Fprintf /usr/local/Cellar/go/1.18.1/libexec/src/fmt/print.go:204 s=0
    20: 0x10dfc6a M=1 testing.(*common).flushToParent /usr/local/Cellar/go/1.18.1/libexec/src/testing/testing.go:710 s=0
    21: 0x10e3864 M=1 testing.(*T).report /usr/local/Cellar/go/1.18.1/libexec/src/testing/testing.go:1758 s=0
    22: 0x10e1b3e M=1 testing.tRunner.func1 /usr/local/Cellar/go/1.18.1/libexec/src/testing/testing.go:1422 s=0
    23: 0x10e167a M=1 testing.tRunner /usr/local/Cellar/go/1.18.1/libexec/src/testing/testing.go:1445 s=0
    24: 0x10bcb17 M=1 fmt.Sprintln /usr/local/Cellar/go/1.18.1/libexec/src/fmt/print.go:282 s=0
    25: 0x10e0bb8 M=1 testing.(*common).Error /usr/local/Cellar/go/1.18.1/libexec/src/testing/testing.go:926 s=0
    26: 0x1344e84 M=1 go.uber.org/goleak.VerifyNone /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/leaks.go:78 s=0
    27: 0x1345c4c M=1 github.com/pcolazurdo/sdkbug/pkg/database.TestConnectionReuseForPutContentHash /Users/pcolazurdo/SINGU/src/gosdkv2-bug/pkg/database/database_test.go:71 s=0
    28: 0x10e1641 M=1 testing.tRunner /usr/local/Cellar/go/1.18.1/libexec/src/testing/testing.go:1439 s=0
    29: 0x10ba0fb M=1 fmt.(*buffer).writeString /usr/local/Cellar/go/1.18.1/libexec/src/fmt/print.go:82 s=0
             fmt.(*fmt).padString /usr/local/Cellar/go/1.18.1/libexec/src/fmt/format.go:110 s=0
    30: 0x10bac9e M=1 fmt.(*fmt).fmtS /usr/local/Cellar/go/1.18.1/libexec/src/fmt/format.go:359 s=0
    31: 0x10bdaa4 M=1 fmt.(*pp).fmtString /usr/local/Cellar/go/1.18.1/libexec/src/fmt/print.go:446 s=0
    32: 0x10beec4 M=1 fmt.(*pp).handleMethods /usr/local/Cellar/go/1.18.1/libexec/src/fmt/print.go:626 s=0
    33: 0x10bfc79 M=1 fmt.(*pp).printValue /usr/local/Cellar/go/1.18.1/libexec/src/fmt/print.go:723 s=0
    34: 0x10c134d M=1 fmt.(*pp).printValue /usr/local/Cellar/go/1.18.1/libexec/src/fmt/print.go:865 s=0
    35: 0x10bfb0b M=1 fmt.(*pp).printArg /usr/local/Cellar/go/1.18.1/libexec/src/fmt/print.go:712 s=0
    36: 0x10b990b M=1 fmt.Errorf /usr/local/Cellar/go/1.18.1/libexec/src/fmt/errors.go:20 s=0
    37: 0x1344d8c M=1 go.uber.org/goleak.Find /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/leaks.go:69 s=0
    38: 0x1344e14 M=1 go.uber.org/goleak.VerifyNone /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/leaks.go:77 s=0
    39: 0x10ce5c4 M=1 bytes.makeSlice /usr/local/Cellar/go/1.18.1/libexec/src/bytes/buffer.go:229 s=0
    40: 0x10cdf9e M=1 bytes.(*Buffer).grow /usr/local/Cellar/go/1.18.1/libexec/src/bytes/buffer.go:142 s=0
    41: 0x10ce2a5 M=1 bytes.(*Buffer).WriteString /usr/local/Cellar/go/1.18.1/libexec/src/bytes/buffer.go:184 s=0
    42: 0x13440d3 M=1 go.uber.org/goleak/internal/stack.getStacks /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/internal/stack/stacks.go:99 s=0
    43: 0x1344d24 M=1 go.uber.org/goleak/internal/stack.All /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/internal/stack/stacks.go:113 s=0
             go.uber.org/goleak.Find /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/leaks.go:61 s=0
    44: 0x10e7a17 M=1 strings.(*Builder).grow /usr/local/Cellar/go/1.18.1/libexec/src/strings/builder.go:67 s=0
             strings.(*Builder).Grow /usr/local/Cellar/go/1.18.1/libexec/src/strings/builder.go:81 s=0
             bufio.(*Reader).ReadString /usr/local/Cellar/go/1.18.1/libexec/src/bufio/bufio.go:497 s=0
    45: 0x1343de7 M=1 go.uber.org/goleak/internal/stack.getStacks /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/internal/stack/stacks.go:75 s=0
    46: 0x1343ce6 M=1 bufio.NewReaderSize /usr/local/Cellar/go/1.18.1/libexec/src/bufio/bufio.go:57 s=0
             bufio.NewReader /usr/local/Cellar/go/1.18.1/libexec/src/bufio/bufio.go:63 s=0
             go.uber.org/goleak/internal/stack.getStacks /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/internal/stack/stacks.go:73 s=0
    47: 0x13456aa M=1 bytes.(*Buffer).String /usr/local/Cellar/go/1.18.1/libexec/src/bytes/buffer.go:65 s=0
             go.uber.org/goleak/internal/stack.Stack.Full /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/internal/stack/stacks.go:55 s=0
             go.uber.org/goleak.isTraceStack /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/tracestack_new.go:33 s=0
    48: 0x1344b8a M=1 go.uber.org/goleak.(*opts).filter /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/options.go:103 s=0
             go.uber.org/goleak.filterStacks /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/leaks.go:44 s=0
    49: 0x1344d33 M=1 go.uber.org/goleak.Find /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/leaks.go:61 s=0
    50: 0x1345565 M=1 bytes.(*Buffer).String /usr/local/Cellar/go/1.18.1/libexec/src/bytes/buffer.go:65 s=0
             go.uber.org/goleak/internal/stack.Stack.Full /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/internal/stack/stacks.go:55 s=0
             go.uber.org/goleak.isStdLibStack /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/options.go:155 s=0
    51: 0x13442d8 M=1 go.uber.org/goleak/internal/stack.getStackBuffer /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/internal/stack/stacks.go:123 s=0
    52: 0x1343c2d M=1 go.uber.org/goleak/internal/stack.getStacks /Users/pcolazurdo/go/pkg/mod/go.uber.org/goleak@v1.1.12/internal/stack/stacks.go:73 s=0
    53: 0x12bbcb6 M=1 strings.(*Builder).WriteString /usr/local/Cellar/go/1.18.1/libexec/src/strings/builder.go:123 s=0
             github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).buildCanonicalHeaders /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:450 s=0
    54: 0x12b991d M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).Build /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:174 s=0
    55: 0x12bad2e M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.Signer.SignHTTP /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:288 s=0
    56: 0x12b8ffc M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*SignHTTPRequestMiddleware).HandleFinalize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/middleware.go:304 s=0
    57: 0x129406a M=1 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_finalize.go:200 s=0
    58: 0x12a6001 M=1 github.com/aws/aws-sdk-go-v2/aws/retry.MetricsHeader.HandleFinalize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/retry/middleware.go:284 s=0
    59: 0x12a5067 M=1 github.com/aws/aws-sdk-go-v2/aws/retry.(*Attempt).handleAttempt /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/retry/middleware.go:166 s=0
    60: 0x12a45d0 M=1 github.com/aws/aws-sdk-go-v2/aws/retry.(*Attempt).HandleFinalize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/retry/middleware.go:90 s=0
    61: 0x131c618 M=1 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding.(*DisableGzip).HandleFinalize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.9.1/accept_encoding_gzip.go:67 s=0
    62: 0x1292c6e M=1 github.com/aws/smithy-go/middleware.(*FinalizeStep).HandleMiddleware /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_finalize.go:114 s=0
    63: 0x129384a M=1 github.com/aws/smithy-go/middleware.decoratedHandler.Handle /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/middleware.go:57 s=0
    64: 0x1293b62 M=1 github.com/aws/smithy-go/middleware.buildWrapHandler.HandleBuild /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_build.go:184 s=0
    65: 0x12a03b7 M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.(*requestUserAgent).HandleBuild /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/user_agent.go:217 s=0
    66: 0x1293c2a M=1 github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_build.go:200 s=0
    67: 0x12b89cb M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*computePayloadSHA256).HandleBuild /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/middleware.go:200 s=0
    68: 0x129ce94 M=1 github.com/aws/smithy-go/transport/http.(*ComputeContentLength).HandleBuild /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/middleware_content_length.go:49 s=0
    69: 0x129ec4c M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.ClientRequestID.HandleBuild /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/middleware.go:42 s=0
    70: 0x129234e M=1 github.com/aws/smithy-go/middleware.(*BuildStep).HandleMiddleware /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_build.go:114 s=0
    71: 0x12943fe M=1 github.com/aws/smithy-go/middleware.serializeWrapHandler.HandleSerialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_serialize.go:192 s=0
    72: 0x1332cf6 M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*awsAwsjson10_serializeOpPutItem).HandleSerialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/serializers.go:1887 s=0
    73: 0x12944e6 M=1 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_serialize.go:208 s=0
    74: 0x131dfb9 M=1 github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery.(*DiscoverEndpoint).HandleSerialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery@v1.7.3/middleware.go:70 s=0
    75: 0x13310ef M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*ResolveEndpoint).HandleSerialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/endpoints.go:116 s=0
    76: 0x10c79ef M=1 strings.(*Builder).grow /usr/local/Cellar/go/1.18.1/libexec/src/strings/builder.go:67 s=0
             strings.(*Builder).Grow /usr/local/Cellar/go/1.18.1/libexec/src/strings/builder.go:81 s=0
             strings.Join /usr/local/Cellar/go/1.18.1/libexec/src/strings/strings.go:438 s=0
    77: 0x12b9c44 M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).buildCanonicalString /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:464 s=0
             github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).Build /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:188 s=0
    78: 0x12bc353 M=1 encoding/hex.EncodeToString /usr/local/Cellar/go/1.18.1/libexec/src/encoding/hex/hex.go:101 s=0
             github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).buildSignature /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:491 s=0
    79: 0x12b9c91 M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).Build /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:197 s=0
    80: 0x11514c4 M=1 math/big.nat.make /usr/local/Cellar/go/1.18.1/libexec/src/math/big/nat.go:69 s=0
             math/big.nat.setBytes /usr/local/Cellar/go/1.18.1/libexec/src/math/big/nat.go:1193 s=0
    81: 0x11c5c65 M=1 math/big.(*Int).SetBytes /usr/local/Cellar/go/1.18.1/libexec/src/math/big/int.go:444 s=0
             vendor/golang.org/x/crypto/cryptobyte.(*String).readASN1BigInt /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/cryptobyte/asn1.go:331 s=0
    82: 0x11c567a M=1 vendor/golang.org/x/crypto/cryptobyte.(*String).ReadASN1Integer /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/cryptobyte/asn1.go:293 s=0
    83: 0x11f0ad3 M=1 crypto/x509.parsePublicKey /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/parser.go:253 s=0
    84: 0x11f58d1 M=1 crypto/x509.parseCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/parser.go:935 s=0
    85: 0x11f62a4 M=1 crypto/x509.ParseCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/parser.go:990 s=0
    86: 0x11f7074 M=1 crypto/x509.exportCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/root_darwin.go:108 s=0
    87: 0x11f698e M=1 crypto/x509.(*Certificate).systemVerify /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/root_darwin.go:65 s=0
    88: 0x11fa9a6 M=1 crypto/x509.(*Certificate).Verify /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/verify.go:750 s=0
    89: 0x12128d7 M=1 crypto/tls.(*Conn).verifyServerCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:868 s=0
    90: 0x120fc4c M=1 crypto/tls.(*clientHandshakeState).doFullHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:513 s=0
    91: 0x120f644 M=1 crypto/tls.(*clientHandshakeState).handshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:431 s=0
    92: 0x120e349 M=1 crypto/tls.(*Conn).clientHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:230 s=0
    93: 0x120c670 M=1 crypto/tls.(*Conn).handshakeContext /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:1452 s=0
    94: 0x127f290 M=1 crypto/tls.(*Conn).HandshakeContext /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:1402 s=0
             net/http.(*persistConn).addTLS.func2 /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:1537 s=0
    95: 0x11c60ed M=1 vendor/golang.org/x/crypto/cryptobyte.(*String).ReadASN1ObjectIdentifier /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/cryptobyte/asn1.go:439 s=0
    96: 0x11f0685 M=1 crypto/x509.parseExtension /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/parser.go:223 s=0
    97: 0x11f5f84 M=1 crypto/x509.parseCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/parser.go:965 s=0
    98: 0x11410b2 M=1 crypto/aes.(*aesCipherGCM).NewGCM /usr/local/Cellar/go/1.18.1/libexec/src/crypto/aes/aes_gcm.go:55 s=0
    99: 0x113f950 M=1 crypto/cipher.newGCMWithNonceAndTagSize /usr/local/Cellar/go/1.18.1/libexec/src/crypto/cipher/gcm.go:121 s=0
   100: 0x1202604 M=1 crypto/cipher.NewGCM /usr/local/Cellar/go/1.18.1/libexec/src/crypto/cipher/gcm.go:85 s=0
             crypto/tls.aeadAESGCM /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/cipher_suites.go:513 s=0
   101: 0x1211062 M=1 crypto/tls.(*clientHandshakeState).establishKeys /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:669 s=0
   102: 0x120f65a M=1 crypto/tls.(*clientHandshakeState).handshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:434 s=0
   103: 0x11efd33 M=1 crypto/x509.parseName /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/parser.go:140 s=0
   104: 0x11f564a M=1 crypto/x509.parseCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/parser.go:908 s=0
   105: 0x114e2e9 M=1 math/big.nat.make /usr/local/Cellar/go/1.18.1/libexec/src/math/big/nat.go:69 s=0
             math/big.nat.sqr /usr/local/Cellar/go/1.18.1/libexec/src/math/big/nat.go:582 s=0
   106: 0x114f7cf M=1 math/big.nat.expNN /usr/local/Cellar/go/1.18.1/libexec/src/math/big/nat.go:958 s=0
   107: 0x1147c38 M=1 math/big.(*Int).Exp /usr/local/Cellar/go/1.18.1/libexec/src/math/big/int.go:509 s=0
   108: 0x11d0504 M=1 crypto/rsa.encrypt /usr/local/Cellar/go/1.18.1/libexec/src/crypto/rsa/rsa.go:389 s=0
   109: 0x11cf618 M=1 crypto/rsa.VerifyPKCS1v15 /usr/local/Cellar/go/1.18.1/libexec/src/crypto/rsa/pkcs1v15.go:286 s=0
   110: 0x120084b M=1 crypto/tls.verifyHandshakeSignature /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/auth.go:45 s=0
   111: 0x122b7e4 M=1 crypto/tls.(*ecdheKeyAgreement).processServerKeyExchange /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/key_agreement.go:345 s=0
   112: 0x120fe39 M=1 crypto/tls.(*clientHandshakeState).doFullHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:534 s=0
   113: 0x116fb1b M=1 net.(*netFD).connect /usr/local/Cellar/go/1.18.1/libexec/src/net/fd_unix.go:103 s=0
   114: 0x117d3f3 M=1 net.(*netFD).dial /usr/local/Cellar/go/1.18.1/libexec/src/net/sock_posix.go:149 s=0
   115: 0x117cf51 M=1 net.socket /usr/local/Cellar/go/1.18.1/libexec/src/net/sock_posix.go:70 s=0
   116: 0x11778b7 M=1 net.internetSocket /usr/local/Cellar/go/1.18.1/libexec/src/net/ipsock_posix.go:142 s=0
   117: 0x117f584 M=1 net.(*sysDialer).doDialTCP /usr/local/Cellar/go/1.18.1/libexec/src/net/tcpsock_posix.go:65 s=0
   118: 0x117f478 M=1 net.(*sysDialer).dialTCP /usr/local/Cellar/go/1.18.1/libexec/src/net/tcpsock_posix.go:61 s=0
   119: 0x116802a M=1 net.(*sysDialer).dialSingle /usr/local/Cellar/go/1.18.1/libexec/src/net/dial.go:583 s=0
   120: 0x1167971 M=1 net.(*sysDialer).dialSerial /usr/local/Cellar/go/1.18.1/libexec/src/net/dial.go:551 s=0
   121: 0x11667f5 M=1 net.(*Dialer).DialContext /usr/local/Cellar/go/1.18.1/libexec/src/net/dial.go:428 s=0
   122: 0x127ca99 M=1 net/http.(*Transport).dial /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:1169 s=0
   123: 0x127fbbe M=1 net/http.(*Transport).dialConn /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:1607 s=0
   124: 0x127e44f M=1 net/http.(*Transport).dialConnFor /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:1449 s=0
   125: 0x11f54c9 M=1 crypto/x509.parseCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/parser.go:888 s=0
   126: 0x114cffb M=1 math/big.nat.make /usr/local/Cellar/go/1.18.1/libexec/src/math/big/nat.go:69 s=0
             math/big.nat.mul /usr/local/Cellar/go/1.18.1/libexec/src/math/big/nat.go:429 s=0
   127: 0x114f854 M=1 math/big.nat.expNN /usr/local/Cellar/go/1.18.1/libexec/src/math/big/nat.go:962 s=0
   128: 0x11ed251 M=1 crypto/x509/internal/macos.CFDataToSlice /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/internal/macos/corefoundation.go:34 s=0
   129: 0x11edfe4 M=1 crypto/x509/internal/macos.SecCertificateCopyData /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/internal/macos/security.go:230 s=0
   130: 0x11f7058 M=1 crypto/x509.exportCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/root_darwin.go:104 s=0
   131: 0x11efb13 M=1 crypto/x509.parseName /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/parser.go:143 s=0
   132: 0x121232e M=1 crypto/tls.(*Conn).verifyServerCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:849 s=0
   133: 0x11eebdb M=1 crypto/x509.(*CertPool).AddCert /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/cert_pool.go:172 s=0
   134: 0x1212864 M=1 crypto/tls.(*Conn).verifyServerCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:865 s=0
   135: 0x120a4f6 M=1 crypto/tls.(*Conn).readHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:1088 s=0
   136: 0x120f9c4 M=1 crypto/tls.(*clientHandshakeState).doFullHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:471 s=0
   137: 0x11efc64 M=1 crypto/x509.parseName /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/parser.go:127 s=0
   138: 0x1282cef M=1 net/http.(*persistConn).readLoop /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:2082 s=0
   139: 0x1208b76 M=1 bytes.(*Buffer).Grow /usr/local/Cellar/go/1.18.1/libexec/src/bytes/buffer.go:161 s=0
             crypto/tls.(*Conn).readFromUntil /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:805 s=0
   140: 0x1206bc4 M=1 crypto/tls.(*Conn).readRecordOrCCS /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:658 s=0
   141: 0x1209e2c M=1 crypto/tls.(*Conn).readRecord /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:581 s=0
             crypto/tls.(*Conn).readHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:1016 s=0
   142: 0x12266e7 M=1 crypto/tls.(*certificateMsg).unmarshal /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_messages.go:1268 s=0
   143: 0x120a538 M=1 crypto/tls.(*Conn).readHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:1090 s=0
   144: 0x11efcde M=1 crypto/x509.parseName /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/parser.go:136 s=0
   145: 0x10ce185 M=1 bytes.(*Buffer).Write /usr/local/Cellar/go/1.18.1/libexec/src/bytes/buffer.go:172 s=0
   146: 0x1207d5e M=1 crypto/tls.(*Conn).readRecordOrCCS /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:755 s=0
   147: 0x11f506c M=1 crypto/x509.parseCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/parser.go:814 s=0
   148: 0x11c7693 M=1 vendor/golang.org/x/crypto/cryptobyte.(*Builder).addLengthPrefixed /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/cryptobyte/builder.go:190 s=0
   149: 0x121742e M=1 vendor/golang.org/x/crypto/cryptobyte.(*Builder).AddUint16LengthPrefixed /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/cryptobyte/builder.go:141 s=0
             crypto/tls.(*clientHelloMsg).marshal.func1 /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_messages.go:110 s=0
   150: 0x11c7373 M=1 vendor/golang.org/x/crypto/cryptobyte.(*Builder).callContinuation /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/cryptobyte/builder.go:174 s=0
   151: 0x11c7724 M=1 vendor/golang.org/x/crypto/cryptobyte.(*Builder).addLengthPrefixed /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/cryptobyte/builder.go:199 s=0
   152: 0x1217144 M=1 vendor/golang.org/x/crypto/cryptobyte.(*Builder).AddUint24LengthPrefixed /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/cryptobyte/builder.go:146 s=0
             crypto/tls.(*clientHelloMsg).marshal /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_messages.go:104 s=0
   153: 0x120decd M=1 crypto/tls.(*Conn).clientHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:175 s=0
   154: 0x12061f5 M=1 crypto/tls.(*Conn).readRecordOrCCS /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:613 s=0
   155: 0x120df48 M=1 crypto/tls.(*Conn).clientHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:179 s=0
   156: 0x120da25 M=1 crypto/tls.(*Conn).makeClientHello /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:139 s=0
   157: 0x120dd95 M=1 crypto/tls.(*Conn).clientHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:154 s=0
   158: 0x1218a26 M=1 vendor/golang.org/x/crypto/cryptobyte.(*Builder).AddUint16LengthPrefixed /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/cryptobyte/builder.go:141 s=0
             crypto/tls.(*clientHelloMsg).marshal.func1.4 /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_messages.go:224 s=0
   159: 0x1217504 M=1 vendor/golang.org/x/crypto/cryptobyte.(*Builder).AddUint16LengthPrefixed /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/cryptobyte/builder.go:141 s=0
             crypto/tls.(*clientHelloMsg).marshal.func1 /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_messages.go:123 s=0
   160: 0x117f158 M=1 net.sockaddrToTCP /usr/local/Cellar/go/1.18.1/libexec/src/net/tcpsock_posix.go:19 s=0
   161: 0x117d51a M=1 net.(*netFD).dial /usr/local/Cellar/go/1.18.1/libexec/src/net/sock_posix.go:166 s=0
   162: 0x126bdb8 M=1 net/http.cloneURL /usr/local/Cellar/go/1.18.1/libexec/src/net/http/clone.go:26 s=0
             net/http.(*Request).Clone /usr/local/Cellar/go/1.18.1/libexec/src/net/http/request.go:379 s=0
   163: 0x129da91 M=1 github.com/aws/smithy-go/transport/http.(*Request).Build /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/request.go:153 s=0
   164: 0x129b9cd M=1 github.com/aws/smithy-go/transport/http.ClientHandler.Handle /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/client.go:50 s=0
   165: 0x1293d82 M=1 github.com/aws/smithy-go/middleware.deserializeWrapHandler.HandleDeserialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_deserialize.go:190 s=0
   166: 0x129d28f M=1 github.com/aws/smithy-go/transport/http.(*RequestResponseLogger).HandleDeserialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/middleware_http_logging.go:58 s=0
   167: 0x1293e4a M=1 github.com/aws/smithy-go/middleware.decoratedDeserializeHandler.HandleDeserialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_deserialize.go:206 s=0
   168: 0x131bade M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb/internal/customizations.(*Checksum).HandleDeserialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/internal/customizations/checksum.go:47 s=0
   169: 0x129edb9 M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.RecordResponseTiming.HandleDeserialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/middleware.go:57 s=0
   170: 0x13267aa M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*awsAwsjson10_deserializeOpPutItem).HandleDeserialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/deserializers.go:3977 s=0
   171: 0x129f527 M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.(*requestIDRetriever).HandleDeserialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/request_id_retriever.go:28 s=0
   172: 0x12a1b3e M=1 github.com/aws/aws-sdk-go-v2/aws/transport/http.(*responseErrorWrapper).HandleDeserialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/transport/http/response_error_middleware.go:29 s=0
   173: 0x129cb3e M=1 github.com/aws/smithy-go/transport/http.(*closeResponseBody).HandleDeserialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/middleware_close_response_body.go:60 s=0
   174: 0x129c93e M=1 github.com/aws/smithy-go/transport/http.(*errorCloseResponseBodyMiddleware).HandleDeserialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/middleware_close_response_body.go:29 s=0
   175: 0x129f385 M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.addRawResponse.HandleDeserialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/middleware.go:154 s=0
   176: 0x12927ce M=1 github.com/aws/smithy-go/middleware.(*DeserializeStep).HandleMiddleware /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_deserialize.go:120 s=0
   177: 0x1293fa2 M=1 github.com/aws/smithy-go/middleware.finalizeWrapHandler.HandleFinalize /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_finalize.go:184 s=0
   178: 0x12b918d M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*SignHTTPRequestMiddleware).HandleFinalize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/middleware.go:315 s=0
   179: 0x12bb13d M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).buildCanonicalHeaders /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:407 s=0
   180: 0x12944cd M=1 github.com/aws/smithy-go/middleware.(*decoratedSerializeHandler).HandleSerialize <autogenerated>:1 s=0
   181: 0x126bfa4 M=1 net/http.Header.Clone /usr/local/Cellar/go/1.18.1/libexec/src/net/http/header.go:104 s=0
             net/http.(*Request).Clone /usr/local/Cellar/go/1.18.1/libexec/src/net/http/request.go:381 s=0
   182: 0x1294051 M=1 github.com/aws/smithy-go/middleware.(*decoratedFinalizeHandler).HandleFinalize <autogenerated>:1 s=0
   183: 0x11780b1 M=1 net.ipToSockaddr /usr/local/Cellar/go/1.18.1/libexec/src/net/ipsock_posix.go:186 s=0
   184: 0x117f33a M=1 net.(*TCPAddr).sockaddr /usr/local/Cellar/go/1.18.1/libexec/src/net/tcpsock_posix.go:40 s=0
   185: 0x117d3c1 M=1 net.(*netFD).dial /usr/local/Cellar/go/1.18.1/libexec/src/net/sock_posix.go:146 s=0
   186: 0x12bbebd M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).buildStringToSign /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:479 s=0
   187: 0x12b9c71 M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).Build /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:196 s=0
   188: 0x12bbe9c M=1 crypto/sha256.New /usr/local/Cellar/go/1.18.1/libexec/src/crypto/sha256/sha256.go:162 s=0
             github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).buildStringToSign /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:479 s=0
   189: 0x124b6eb M=1 net/http.setRequestCancel /usr/local/Cellar/go/1.18.1/libexec/src/net/http/client.go:383 s=0
   190: 0x124ac45 M=1 net/http.send /usr/local/Cellar/go/1.18.1/libexec/src/net/http/client.go:250 s=0
   191: 0x124a51a M=1 net/http.(*Client).send /usr/local/Cellar/go/1.18.1/libexec/src/net/http/client.go:176 s=0
   192: 0x124c8b4 M=1 net/http.(*Client).do /usr/local/Cellar/go/1.18.1/libexec/src/net/http/client.go:725 s=0
   193: 0x12a0b04 M=1 net/http.(*Client).Do /usr/local/Cellar/go/1.18.1/libexec/src/net/http/client.go:593 s=0
             github.com/aws/aws-sdk-go-v2/aws/transport/http.(*BuildableClient).Do /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/transport/http/client.go:69 s=0
   194: 0x129ba06 M=1 github.com/aws/smithy-go/transport/http.ClientHandler.Handle /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/client.go:55 s=0
   195: 0x12b9f98 M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).Build /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:206 s=0
   196: 0x11789ab M=1 net.withUnexpiredValuesPreserved /usr/local/Cellar/go/1.18.1/libexec/src/net/lookup.go:282 s=0
             net.(*Resolver).lookupIPAddr /usr/local/Cellar/go/1.18.1/libexec/src/net/lookup.go:313 s=0
   197: 0x1176bb9 M=1 net.(*Resolver).internetAddrList /usr/local/Cellar/go/1.18.1/libexec/src/net/ipsock.go:288 s=0
   198: 0x116563a M=1 net.(*Resolver).resolveAddrList /usr/local/Cellar/go/1.18.1/libexec/src/net/dial.go:221 s=0
   199: 0x1166507 M=1 net.(*Dialer).DialContext /usr/local/Cellar/go/1.18.1/libexec/src/net/dial.go:406 s=0
   200: 0x10ce06a M=1 bytes.(*Buffer).grow /usr/local/Cellar/go/1.18.1/libexec/src/bytes/buffer.go:128 s=0
   201: 0x10ce7d1 M=1 bytes.(*Buffer).WriteByte /usr/local/Cellar/go/1.18.1/libexec/src/bytes/buffer.go:267 s=0
   202: 0x12b7fab M=1 github.com/aws/smithy-go/encoding/httpbinding.EscapePath /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/encoding/httpbinding/path_replace.go:87 s=0
   203: 0x12b9b4f M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).Build /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:185 s=0
   204: 0x10d3c44 M=1 context.WithDeadline /usr/local/Cellar/go/1.18.1/libexec/src/context/context.go:442 s=0
   205: 0x124b5d6 M=1 net/http.setRequestCancel /usr/local/Cellar/go/1.18.1/libexec/src/net/http/client.go:366 s=0
   206: 0x124e9e9 M=1 net/http.Header.Clone /usr/local/Cellar/go/1.18.1/libexec/src/net/http/header.go:104 s=0
             net/http.cloneOrMakeHeader /usr/local/Cellar/go/1.18.1/libexec/src/net/http/clone.go:69 s=0
   207: 0x124d224 M=1 net/http.(*Client).makeHeadersCopier /usr/local/Cellar/go/1.18.1/libexec/src/net/http/client.go:754 s=0
   208: 0x124c235 M=1 net/http.(*Client).do /usr/local/Cellar/go/1.18.1/libexec/src/net/http/client.go:614 s=0
   209: 0x12bbb04 M=1 strings.(*Builder).WriteString /usr/local/Cellar/go/1.18.1/libexec/src/strings/builder.go:123 s=0
             github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).buildCanonicalHeaders /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:444 s=0
   210: 0x10eab72 M=1 crypto/sha256.(*digest).MarshalBinary /usr/local/Cellar/go/1.18.1/libexec/src/crypto/sha256/sha256.go:66 s=0
   211: 0x11d2303 M=1 crypto/hmac.(*hmac).Reset /usr/local/Cellar/go/1.18.1/libexec/src/crypto/hmac/hmac.go:111 s=0
   212: 0x122d6ec M=1 crypto/tls.pHash /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/prf.go:34 s=0
   213: 0x122dc64 M=1 crypto/tls.prf12.func1 /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/prf.go:73 s=0
   214: 0x122ebb3 M=1 crypto/tls.finishedHash.serverSum /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/prf.go:212 s=0
   215: 0x121193e M=1 crypto/tls.(*clientHandshakeState).readFinished /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:784 s=0
   216: 0x120f6f6 M=1 crypto/tls.(*clientHandshakeState).handshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:447 s=0
   217: 0x13328cd M=1 github.com/aws/smithy-go/encoding/httpbinding.HeaderValue.modifyHeader /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/encoding/httpbinding/header.go:47 s=0
             github.com/aws/smithy-go/encoding/httpbinding.HeaderValue.String /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/encoding/httpbinding/header.go:53 s=0
             github.com/aws/aws-sdk-go-v2/service/dynamodb.(*awsAwsjson10_serializeOpPutItem).HandleSerialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/serializers.go:1870 s=0
   218: 0x12935b1 M=1 github.com/aws/smithy-go/middleware.(*SerializeStep).HandleMiddleware /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_serialize.go:122 s=0
   219: 0x12941c2 M=1 github.com/aws/smithy-go/middleware.initializeWrapHandler.HandleInitialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_initialize.go:184 s=0
   220: 0x1335dea M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*validateOpPutItem).HandleInitialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/validators.go:550 s=0
   221: 0x129428a M=1 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_initialize.go:200 s=0
   222: 0x128f3b4 M=1 github.com/aws/smithy-go/middleware.(*setLogger).HandleInitialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/logging.go:45 s=0
   223: 0x129e9b0 M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.RegisterServiceMetadata.HandleInitialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/metadata.go:40 s=0
   224: 0x12930ee M=1 github.com/aws/smithy-go/middleware.(*InitializeStep).HandleMiddleware /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_initialize.go:114 s=0
   225: 0x1290c9a M=1 github.com/aws/smithy-go/middleware.(*Stack).HandleMiddleware /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/stack.go:109 s=0
   226: 0x132210a M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*Client).invokeOperation /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_client.go:218 s=0
   227: 0x1324806 M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*Client).PutItem /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_op_PutItem.go:80 s=0
   228: 0x134700f M=1 github.com/pcolazurdo/sdkbug/pkg/database.putItem /Users/pcolazurdo/SINGU/src/gosdkv2-bug/pkg/database/database_test.go:177 s=0
   229: 0x1345e18 M=1 github.com/pcolazurdo/sdkbug/pkg/database.TestConnectionReuseForPutContentHash.func2 /Users/pcolazurdo/SINGU/src/gosdkv2-bug/pkg/database/database_test.go:58 s=0
   230: 0x129d8f1 M=1 github.com/aws/smithy-go/transport/http.(*Request).Clone /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/request.go:40 s=0
             github.com/aws/smithy-go/transport/http.(*Request).SetStream /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/request.go:114 s=0
   231: 0x1332b9b M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*awsAwsjson10_serializeOpPutItem).HandleSerialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/serializers.go:1878 s=0
   232: 0x128f57d M=1 github.com/aws/smithy-go/middleware.(*orderedIDs).Add /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/ordered_group.go:47 s=0
   233: 0x13249a4 M=1 github.com/aws/smithy-go/middleware.(*DeserializeStep).Add /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_deserialize.go:136 s=0
             github.com/aws/aws-sdk-go-v2/service/dynamodb.(*Client).addOperationPutItemMiddlewares /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_op_PutItem.go:292 s=0
   234: 0x1321f3c M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*Client).invokeOperation /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_client.go:206 s=0
   235: 0x12bed5e M=1 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2.Endpoint.resolve /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.4.3/endpoints.go:234 s=0
   236: 0x12be8d7 M=1 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2.Partition.ResolveEndpoint /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.4.3/endpoints.go:150 s=0
   237: 0x12bdf9e M=1 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2.Partitions.ResolveEndpoint /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.4.3/endpoints.go:109 s=0
   238: 0x131e5f1 M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb/internal/endpoints.(*Resolver).ResolveEndpoint /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/internal/endpoints/endpoints.go:77 s=0
   239: 0x1330d82 M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*ResolveEndpoint).HandleSerialize /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/endpoints.go:95 s=0
   240: 0x1316104 M=1 github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue.NewEncoder /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue@v1.8.4/encode.go:391 s=0
   241: 0x1315fed M=1 github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue.MarshalMap /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue@v1.8.4/encode.go:294 s=0
   242: 0x1346ef7 M=1 github.com/pcolazurdo/sdkbug/pkg/database.putItem /Users/pcolazurdo/SINGU/src/gosdkv2-bug/pkg/database/database_test.go:164 s=0
   243: 0x129e124 M=1 strings.(*Builder).WriteString /usr/local/Cellar/go/1.18.1/libexec/src/strings/builder.go:123 s=0
             github.com/aws/smithy-go/transport/http.(*UserAgentBuilder).appendTo /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/user_agent.go:36 s=0
   244: 0x129ff04 M=1 github.com/aws/smithy-go/transport/http.(*UserAgentBuilder).AddKey /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/user_agent.go:19 s=0
             github.com/aws/aws-sdk-go-v2/aws/middleware.(*requestUserAgent).AddSDKAgentKey /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/user_agent.go:190 s=0
   245: 0x129f8b2 M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.addSDKMetadata /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/user_agent.go:91 s=0
   246: 0x129f851 M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.newRequestUserAgent /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/user_agent.go:85 s=0
   247: 0x129fcc6 M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.getOrAddRequestUserAgent /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/user_agent.go:162 s=0
   248: 0x129fbe9 M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.AddSDKAgentKeyValue.func1 /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/user_agent.go:143 s=0
   249: 0x13230cd M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.addClientUserAgent /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_client.go:383 s=0
   250: 0x1324c5a M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*Client).addOperationPutItemMiddlewares /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_op_PutItem.go:323 s=0
   251: 0x1325233 M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.addOpPutItemDiscoverEndpointMiddleware /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_op_PutItem.go:359 s=0
   252: 0x1324d69 M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*Client).addOperationPutItemMiddlewares /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_op_PutItem.go:332 s=0
   253: 0x129d575 M=1 github.com/aws/smithy-go/transport/http.NewStackRequest /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/request.go:28 s=0
   254: 0x1293578 M=1 github.com/aws/smithy-go/middleware.(*SerializeStep).HandleMiddleware /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_serialize.go:119 s=0
   255: 0x1324954 M=1 github.com/aws/smithy-go/middleware.(*SerializeStep).Add /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_serialize.go:138 s=0
             github.com/aws/aws-sdk-go-v2/service/dynamodb.(*Client).addOperationPutItemMiddlewares /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_op_PutItem.go:288 s=0
   256: 0x1324a32 M=1 github.com/aws/smithy-go/middleware.(*BuildStep).Add /Users/pcolazurdo/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_build.go:130 s=0
             github.com/aws/aws-sdk-go-v2/aws/middleware.AddClientRequestIDMiddleware /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/middleware.go:129 s=0
             github.com/aws/aws-sdk-go-v2/service/dynamodb.(*Client).addOperationPutItemMiddlewares /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_op_PutItem.go:299 s=0
   257: 0x13459f0 M=1 github.com/pcolazurdo/sdkbug/pkg/database.TestConnectionReuseForPutContentHash /Users/pcolazurdo/SINGU/src/gosdkv2-bug/pkg/database/database_test.go:52 s=0
   258: 0x13469b9 M=1 github.com/pcolazurdo/sdkbug/pkg/database.pseudo_uuid /Users/pcolazurdo/SINGU/src/gosdkv2-bug/pkg/database/database_test.go:129 s=0
   259: 0x13467af M=1 github.com/pcolazurdo/sdkbug/pkg/database.fakeHash /Users/pcolazurdo/SINGU/src/gosdkv2-bug/pkg/database/database_test.go:119 s=0
   260: 0x1345d97 M=1 github.com/pcolazurdo/sdkbug/pkg/database.TestConnectionReuseForPutContentHash.func2 /Users/pcolazurdo/SINGU/src/gosdkv2-bug/pkg/database/database_test.go:55 s=0
   261: 0x112d535 M=1 compress/gzip.NewWriterLevel /usr/local/Cellar/go/1.18.1/libexec/src/compress/gzip/gzip.go:64 s=0
   262: 0x1136510 M=1 runtime/pprof.newProfileBuilder /usr/local/Cellar/go/1.18.1/libexec/src/runtime/pprof/proto.go:254 s=0
   263: 0x11330e6 M=1 runtime/pprof.profileWriter /usr/local/Cellar/go/1.18.1/libexec/src/runtime/pprof/pprof.go:797 s=0
   264: 0x1132efe M=1 runtime/pprof.StartCPUProfile /usr/local/Cellar/go/1.18.1/libexec/src/runtime/pprof/pprof.go:784 s=0
   265: 0x113d416 M=1 testing/internal/testdeps.TestDeps.StartCPUProfile /usr/local/Cellar/go/1.18.1/libexec/src/testing/internal/testdeps/deps.go:48 s=0
   266: 0x10e449d M=1 testing.(*M).before /usr/local/Cellar/go/1.18.1/libexec/src/testing/testing.go:1865 s=0
   267: 0x10e2c06 M=1 testing.(*M).Run /usr/local/Cellar/go/1.18.1/libexec/src/testing/testing.go:1710 s=0
   268: 0x1347389 M=1 main.main _testmain.go:49 s=0
   269: 0x10387d1 M=1 runtime.main /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:250 s=0
   270: 0x110a205 M=1 regexp/syntax.(*compiler).inst /usr/local/Cellar/go/1.18.1/libexec/src/regexp/syntax/compile.go:164 s=0
             regexp/syntax.(*compiler).rune /usr/local/Cellar/go/1.18.1/libexec/src/regexp/syntax/compile.go:273 s=0
   271: 0x110983e M=1 regexp/syntax.(*compiler).compile /usr/local/Cellar/go/1.18.1/libexec/src/regexp/syntax/compile.go:101 s=0
   272: 0x11095f1 M=1 regexp/syntax.(*compiler).compile /usr/local/Cellar/go/1.18.1/libexec/src/regexp/syntax/compile.go:147 s=0
   273: 0x110819b M=1 regexp/syntax.Compile /usr/local/Cellar/go/1.18.1/libexec/src/regexp/syntax/compile.go:74 s=0
   274: 0x111e51a M=1 regexp.compile /usr/local/Cellar/go/1.18.1/libexec/src/regexp/regexp.go:180 s=0
   275: 0x111f090 M=1 regexp.Compile /usr/local/Cellar/go/1.18.1/libexec/src/regexp/regexp.go:135 s=0
             regexp.MustCompile /usr/local/Cellar/go/1.18.1/libexec/src/regexp/regexp.go:315 s=0
   276: 0x12db4c7 M=1 github.com/aws/aws-sdk-go-v2/service/sts/internal/endpoints.init /Users/pcolazurdo/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/sts@v1.16.3/internal/endpoints/endpoints.go:98 s=0
   277: 0x1045685 M=1 runtime.doInit /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:6222 s=0
   278: 0x10455d0 M=1 runtime.doInit /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:6199 s=0
   279: 0x1038792 M=1 runtime.main /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:233 s=0
   280: 0x103bc3d M=1 runtime.allocm /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:1743 s=0
   281: 0x103c41a M=1 runtime.newm /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:2090 s=0
   282: 0x103ca8e M=1 runtime.startm /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:2305 s=0
   283: 0x103cfb9 M=1 runtime.wakep /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:2404 s=0
   284: 0x103e784 M=1 runtime.resetspinning /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:3036 s=0
   285: 0x103ed3d M=1 runtime.schedule /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:3194 s=0
   286: 0x103b44c M=1 runtime.mstart1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:1425 s=0
   287: 0x103b34b M=1 runtime.mstart0 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:1376 s=0
   288: 0x1066404 M=1 runtime.mstart /usr/local/Cellar/go/1.18.1/libexec/src/runtime/asm_amd64.s:367 s=0
   289: 0x1040b47 M=1 runtime.malg /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:4034 s=0
   290: 0x1040dee M=1 runtime.newproc1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:4082 s=0
   291: 0x1040d24 M=1 runtime.newproc.func1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:4056 s=0
   292: 0x1066508 M=1 runtime.systemstack /usr/local/Cellar/go/1.18.1/libexec/src/runtime/asm_amd64.s:469 s=0
Mappings
1: 0x0/0x0/0x0   [FN]

cpu.prof

PeriodType: cpu nanoseconds
Period: 10000000
Time: 2022-04-14 15:32:56.533468 +0100 IST
Duration: 4.08
Samples:
samples/count cpu/nanoseconds
          1   10000000: 1 2 3 4 5 6 7 
          1   10000000: 8 9 10 11 12 13 14 15 16 
          1   10000000: 17 18 19 20 21 22 21 23 5 6 7 
          1   10000000: 24 18 25 26 3 4 5 6 7 
          1   10000000: 27 28 29 30 31 32 33 34 35 36 37 
          2   20000000: 8 9 10 11 12 38 39 40 41 21 42 43 44 45 44 46 44 47 21 48 49 50 51 50 52 50 53 21 22 21 23 5 6 7 
          1   10000000: 54 55 56 57 58 59 60 61 62 63 
          1   10000000: 64 65 66 67 68 25 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 82 84 82 85 82 86 82 87 82 88 82 89 82 90 82 91 21 92 93 94 95 94 96 97 94 98 94 99 21 100 101 40 102 40 103 40 104 40 41 21 42 43 
          2   20000000: 105 106 107 108 109 66 110 
          1   10000000: 105 106 107 111 112 113 109 66 110 
          3   30000000: 105 106 114 36 
          1   10000000: 115 31 32 33 34 35 36 37 
          1   10000000: 116 117 118 119 120 121 94 95 94 96 97 94 98 94 99 21 100 101 40 102 40 103 40 104 40 41 21 42 43 44 45 44 46 44 47 21 48 49 50 51 50 52 50 53 21 22 21 23 5 6 7 
          1   10000000: 122 18 25 123 124 125 126 127 
          1   10000000: 128 129 130 131 132 66 110 
          1   10000000: 133 134 135 36 136 137 138 139 140 141 142 143 144 145 78 79 80 81 82 83 82 84 82 85 82 86 82 87 82 88 82 89 82 90 82 91 21 92 93 94 95 94 96 97 94 98 94 99 21 100 101 40 102 40 103 40 104 40 41 21 42 43 
          1   10000000: 146 147 148 149 150 151 152 153 154 155 156 157 158 159 125 126 127 
          1   10000000: 160 161 162 163 66 67 68 25 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 82 84 82 85 82 86 82 87 82 88 82 89 82 90 82 91 21 92 93 94 95 94 96 97 94 98 94 99 21 100 101 40 102 40 103 40 104 40 41 21 164 
          2   20000000: 165 166 167 168 169 
          1   10000000: 170 171 172 173 174 154 155 156 157 158 159 125 126 127 
          1   10000000: 8 9 10 11 12 175 176 177 62 63 
          1   10000000: 178 179 180 181 176 177 62 63 
          1   10000000: 182 183 184 185 186 187 188 189 190 
          1   10000000: 8 9 10 11 12 191 177 62 63 
          1   10000000: 192 193 180 181 176 177 62 63 
          1   10000000: 194 195 180 181 176 177 62 63 
          1   10000000: 196 25 197 198 199 62 63 
          1   10000000: 8 200 201 202 203 204 205 206 207 62 63 
          2   20000000: 208 18 209 210 211 212 213 214 215 60 61 62 63 
          1   10000000: 116 18 25 216 217 218 219 220 61 62 63 
          1   10000000: 54 221 222 57 58 59 60 61 62 63 
          1   10000000: 223 114 36 224 225 226 227 228 229 230 231 232 233 234 59 60 61 62 63 
          1   10000000: 235 236 
          5   50000000: 54 237 238 57 58 59 60 61 62 63 
          1   10000000: 105 106 107 239 240 241 242 243 162 31 32 33 34 35 36 
          1   10000000: 244 245 246 243 162 31 32 33 34 35 36 37 
          1   10000000: 146 247 248 249 250 36 
          3   30000000: 8 251 252 253 254 255 256 257 258 259 260 261 262 263 
          1   10000000: 264 265 266 232 233 234 59 60 61 62 63 
          1   10000000: 267 268 269 228 229 209 270 271 272 273 274 275 276 277 60 61 62 63 
          1   10000000: 54 221 278 57 58 59 60 61 62 63 
          2   20000000: 146 247 279 188 280 190 
          1   10000000: 281 54 282 283 284 57 58 59 60 61 62 63 
          1   10000000: 8 200 201 202 203 204 205 285 286 287 288 289 290 291 292 293 
          1   10000000: 182 8 294 295 296 297 253 254 255 256 257 213 214 298 60 61 62 63 
          1   10000000: 170 299 300 301 152 153 154 155 156 157 158 159 125 126 127 
          1   10000000: 8 251 252 253 254 255 256 257 213 214 215 60 61 62 63 
          1   10000000: 8 251 252 253 254 255 256 257 213 214 298 60 61 62 63 
          1   10000000: 302 236 
          1   10000000: 303 268 269 228 229 25 304 305 306 307 308 309 310 82 86 82 87 82 88 82 89 82 90 82 91 21 92 93 94 95 94 96 97 94 98 94 99 21 100 101 40 102 40 103 40 104 40 41 21 42 43 44 45 44 46 44 47 21 48 49 50 51 50 52 
          1   10000000: 311 312 313 314 315 277 60 61 62 63 
          1   10000000: 8 9 10 11 12 316 317 318 277 60 61 62 63 
          1   10000000: 319 320 321 322 323 324 325 36 
          2   20000000: 8 200 201 202 203 204 205 285 326 327 328 293 
          1   10000000: 128 329 330 331 332 188 280 190 
          1   10000000: 319 320 321 322 323 333 36 
          1   10000000: 146 247 334 188 189 190 
          1   10000000: 183 184 185 186 187 188 189 190 
          1   10000000: 183 184 185 186 187 188 280 190 
          1   10000000: 335 336 337 338 339 340 341 342 343 36 344 345 346 347 348 349 350 
          1   10000000: 146 247 351 352 36 
          1   10000000: 353 354 355 340 341 342 343 36 344 345 346 347 348 349 350 
Locations
     1: 0x10691f7 M=1 runtime.duffcopy /usr/local/Cellar/go/1.18.1/libexec/src/runtime/duff_amd64.s:389 s=0
     2: 0x132526e M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.addOpPutItemDiscoverEndpointMiddleware /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_op_PutItem.go:359 s=0
     3: 0x1324d69 M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*Client).addOperationPutItemMiddlewares /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_op_PutItem.go:332 s=0
     4: 0x1321f3c M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*Client).invokeOperation /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_client.go:206 s=0
     5: 0x1324806 M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*Client).PutItem /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_op_PutItem.go:80 s=0
     6: 0x134700f M=1 github.com/edited/sdkbug/pkg/database.putItem /Users/edited/SINGU/src/gosdkv2-bug/pkg/database/database_test.go:177 s=0
     7: 0x1345e18 M=1 github.com/edited/sdkbug/pkg/database.TestConnectionReuseForPutContentHash.func2 /Users/edited/SINGU/src/gosdkv2-bug/pkg/database/database_test.go:58 s=0
     8: 0x106505e M=1 syscall.syscall /usr/local/Cellar/go/1.18.1/libexec/src/runtime/sys_darwin.go:23 s=0
     9: 0x10916b9 M=1 internal/syscall/unix.GetEntropy /usr/local/Cellar/go/1.18.1/libexec/src/internal/syscall/unix/getentropy_darwin.go:21 s=0
    10: 0x1156b2d M=1 crypto/rand.getEntropy /usr/local/Cellar/go/1.18.1/libexec/src/crypto/rand/rand_getentropy.go:24 s=0
    11: 0x1156db2 M=1 crypto/rand.(*devReader).Read /usr/local/Cellar/go/1.18.1/libexec/src/crypto/rand/rand_unix.go:61 s=0
    12: 0x108e439 M=1 io.ReadAtLeast /usr/local/Cellar/go/1.18.1/libexec/src/io/io.go:331 s=0
    13: 0x11569fd M=1 io.ReadFull /usr/local/Cellar/go/1.18.1/libexec/src/io/io.go:350 s=0
             crypto/rand.Read /usr/local/Cellar/go/1.18.1/libexec/src/crypto/rand/rand.go:25 s=0
    14: 0x13469cb M=1 github.com/edited/sdkbug/pkg/database.pseudo_uuid /Users/edited/SINGU/src/gosdkv2-bug/pkg/database/database_test.go:130 s=0
    15: 0x13467af M=1 github.com/edited/sdkbug/pkg/database.fakeHash /Users/edited/SINGU/src/gosdkv2-bug/pkg/database/database_test.go:119 s=0
    16: 0x1345d97 M=1 github.com/edited/sdkbug/pkg/database.TestConnectionReuseForPutContentHash.func2 /Users/edited/SINGU/src/gosdkv2-bug/pkg/database/database_test.go:55 s=0
    17: 0x1069465 M=1 runtime.memclrNoHeapPointers /usr/local/Cellar/go/1.18.1/libexec/src/runtime/memclr_amd64.s:143 s=0
    18: 0x100e3c5 M=1 runtime.mallocgc /usr/local/Cellar/go/1.18.1/libexec/src/runtime/malloc.go:1089 s=0
    19: 0x100bdd3 M=1 runtime.convT /usr/local/Cellar/go/1.18.1/libexec/src/runtime/iface.go:331 s=0
    20: 0x12930ab M=1 github.com/aws/smithy-go/middleware.(*InitializeStep).HandleMiddleware /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_initialize.go:104 s=0
    21: 0x129384a M=1 github.com/aws/smithy-go/middleware.decoratedHandler.Handle /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/middleware.go:57 s=0
    22: 0x1290c9a M=1 github.com/aws/smithy-go/middleware.(*Stack).HandleMiddleware /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/stack.go:109 s=0
    23: 0x132210a M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*Client).invokeOperation /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_client.go:218 s=0
    24: 0x10693aa M=1 runtime.memclrNoHeapPointers /usr/local/Cellar/go/1.18.1/libexec/src/runtime/memclr_amd64.s:80 s=0
    25: 0x100e8a6 M=1 runtime.newobject /usr/local/Cellar/go/1.18.1/libexec/src/runtime/malloc.go:1259 s=0
    26: 0x1325233 M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.addOpPutItemDiscoverEndpointMiddleware /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/api_op_PutItem.go:359 s=0
    27: 0x105463c M=1 runtime.step /usr/local/Cellar/go/1.18.1/libexec/src/runtime/symtab.go:1118 s=0
    28: 0x1053968 M=1 runtime.pcvalue /usr/local/Cellar/go/1.18.1/libexec/src/runtime/symtab.go:915 s=0
    29: 0x1054273 M=1 runtime.funcspdelta /usr/local/Cellar/go/1.18.1/libexec/src/runtime/symtab.go:1038 s=0
    30: 0x105b8fc M=1 runtime.gentraceback /usr/local/Cellar/go/1.18.1/libexec/src/runtime/traceback.go:187 s=0
    31: 0x101ee79 M=1 runtime.scanstack /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgcmark.go:783 s=0
    32: 0x101dd84 M=1 runtime.markroot.func1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgcmark.go:241 s=0
    33: 0x101da04 M=1 runtime.markroot /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgcmark.go:214 s=0
    34: 0x101fa1e M=1 runtime.gcDrain /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgcmark.go:1047 s=0
    35: 0x101c8a4 M=1 runtime.gcBgMarkWorker.func2 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgc.go:1276 s=0
    36: 0x1066508 M=1 runtime.systemstack /usr/local/Cellar/go/1.18.1/libexec/src/runtime/asm_amd64.s:469 s=0
    37: 0x101c570 M=1 runtime.gcBgMarkWorker /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgc.go:1263 s=0
    38: 0x129e48f M=1 io.ReadFull /usr/local/Cellar/go/1.18.1/libexec/src/io/io.go:350 s=0
             github.com/aws/smithy-go/rand.(*UUID).GetUUID /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/rand/uuid.go:43 s=0
    39: 0x129eb1a M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.ClientRequestID.HandleBuild /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/middleware.go:34 s=0
    40: 0x1293c2a M=1 github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_build.go:200 s=0
    41: 0x129234e M=1 github.com/aws/smithy-go/middleware.(*BuildStep).HandleMiddleware /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_build.go:114 s=0
    42: 0x12943fe M=1 github.com/aws/smithy-go/middleware.serializeWrapHandler.HandleSerialize /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_serialize.go:192 s=0
    43: 0x1332cf6 M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*awsAwsjson10_serializeOpPutItem).HandleSerialize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/serializers.go:1887 s=0
    44: 0x12944e6 M=1 github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_serialize.go:208 s=0
    45: 0x131dfb9 M=1 github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery.(*DiscoverEndpoint).HandleSerialize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery@v1.7.3/middleware.go:70 s=0
    46: 0x13310ef M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*ResolveEndpoint).HandleSerialize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/endpoints.go:116 s=0
    47: 0x12935b1 M=1 github.com/aws/smithy-go/middleware.(*SerializeStep).HandleMiddleware /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_serialize.go:122 s=0
    48: 0x12941c2 M=1 github.com/aws/smithy-go/middleware.initializeWrapHandler.HandleInitialize /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_initialize.go:184 s=0
    49: 0x1335dea M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*validateOpPutItem).HandleInitialize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/validators.go:550 s=0
    50: 0x129428a M=1 github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_initialize.go:200 s=0
    51: 0x128f3b4 M=1 github.com/aws/smithy-go/middleware.(*setLogger).HandleInitialize /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/logging.go:45 s=0
    52: 0x129e9b0 M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.RegisterServiceMetadata.HandleInitialize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/metadata.go:40 s=0
    53: 0x12930ee M=1 github.com/aws/smithy-go/middleware.(*InitializeStep).HandleMiddleware /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_initialize.go:114 s=0
    54: 0x1065328 M=1 crypto/x509/internal/macos.syscall /usr/local/Cellar/go/1.18.1/libexec/src/runtime/sys_darwin.go:100 s=0
    55: 0x11edc15 M=1 crypto/x509/internal/macos.SecPolicyCreateSSL /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/internal/macos/security.go:150 s=0
    56: 0x11f6612 M=1 crypto/x509.(*Certificate).systemVerify /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/root_darwin.go:35 s=0
    57: 0x11fa9a6 M=1 crypto/x509.(*Certificate).Verify /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/verify.go:750 s=0
    58: 0x12128d7 M=1 crypto/tls.(*Conn).verifyServerCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:868 s=0
    59: 0x120fc4c M=1 crypto/tls.(*clientHandshakeState).doFullHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:513 s=0
    60: 0x120f644 M=1 crypto/tls.(*clientHandshakeState).handshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:431 s=0
    61: 0x120e349 M=1 crypto/tls.(*Conn).clientHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:230 s=0
    62: 0x120c670 M=1 crypto/tls.(*Conn).handshakeContext /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:1452 s=0
    63: 0x127f290 M=1 crypto/tls.(*Conn).HandshakeContext /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:1402 s=0
             net/http.(*persistConn).addTLS.func2 /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:1537 s=0
    64: 0x106996e M=1 runtime.memmove /usr/local/Cellar/go/1.18.1/libexec/src/runtime/memmove_amd64.s:372 s=0
    65: 0x104efb3 M=1 runtime.copystack /usr/local/Cellar/go/1.18.1/libexec/src/runtime/stack.go:911 s=0
    66: 0x104f656 M=1 runtime.newstack /usr/local/Cellar/go/1.18.1/libexec/src/runtime/stack.go:1110 s=0
    67: 0x1016f2b M=1 runtime.heapBitsSetType /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mbitmap.go:832 s=0
    68: 0x100e4f2 M=1 runtime.mallocgc /usr/local/Cellar/go/1.18.1/libexec/src/runtime/malloc.go:1117 s=0
    69: 0x1038f11 M=1 runtime.acquireSudog /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:400 s=0
    70: 0x10481fa M=1 runtime.selectgo /usr/local/Cellar/go/1.18.1/libexec/src/runtime/select.go:299 s=0
    71: 0x127d7c5 M=1 net/http.(*Transport).getConn /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:1375 s=0
    72: 0x12793ee M=1 net/http.(*Transport).roundTrip /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:581 s=0
    73: 0x1270118 M=1 net/http.(*Transport).RoundTrip /usr/local/Cellar/go/1.18.1/libexec/src/net/http/roundtrip.go:17 s=0
    74: 0x12a180d M=1 github.com/aws/aws-sdk-go-v2/aws/transport/http.suppressBadHTTPRedirectTransport.RoundTrip /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/transport/http/client.go:294 s=0
    75: 0x124ac77 M=1 net/http.send /usr/local/Cellar/go/1.18.1/libexec/src/net/http/client.go:252 s=0
    76: 0x124a51a M=1 net/http.(*Client).send /usr/local/Cellar/go/1.18.1/libexec/src/net/http/client.go:176 s=0
    77: 0x124c8b4 M=1 net/http.(*Client).do /usr/local/Cellar/go/1.18.1/libexec/src/net/http/client.go:725 s=0
    78: 0x12a0b04 M=1 net/http.(*Client).Do /usr/local/Cellar/go/1.18.1/libexec/src/net/http/client.go:593 s=0
             github.com/aws/aws-sdk-go-v2/aws/transport/http.(*BuildableClient).Do /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/transport/http/client.go:69 s=0
    79: 0x129ba06 M=1 github.com/aws/smithy-go/transport/http.ClientHandler.Handle /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/client.go:55 s=0
    80: 0x1293d82 M=1 github.com/aws/smithy-go/middleware.deserializeWrapHandler.HandleDeserialize /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_deserialize.go:190 s=0
    81: 0x129d28f M=1 github.com/aws/smithy-go/transport/http.(*RequestResponseLogger).HandleDeserialize /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/middleware_http_logging.go:58 s=0
    82: 0x1293e4a M=1 github.com/aws/smithy-go/middleware.decoratedDeserializeHandler.HandleDeserialize /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_deserialize.go:206 s=0
    83: 0x131bade M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb/internal/customizations.(*Checksum).HandleDeserialize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/internal/customizations/checksum.go:47 s=0
    84: 0x129edb9 M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.RecordResponseTiming.HandleDeserialize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/middleware.go:57 s=0
    85: 0x13267aa M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*awsAwsjson10_deserializeOpPutItem).HandleDeserialize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/deserializers.go:3977 s=0
    86: 0x129f527 M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.(*requestIDRetriever).HandleDeserialize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/request_id_retriever.go:28 s=0
    87: 0x12a1b3e M=1 github.com/aws/aws-sdk-go-v2/aws/transport/http.(*responseErrorWrapper).HandleDeserialize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/transport/http/response_error_middleware.go:29 s=0
    88: 0x129cb3e M=1 github.com/aws/smithy-go/transport/http.(*closeResponseBody).HandleDeserialize /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/middleware_close_response_body.go:60 s=0
    89: 0x129c93e M=1 github.com/aws/smithy-go/transport/http.(*errorCloseResponseBodyMiddleware).HandleDeserialize /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/middleware_close_response_body.go:29 s=0
    90: 0x129f385 M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.addRawResponse.HandleDeserialize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/middleware.go:154 s=0
    91: 0x12927ce M=1 github.com/aws/smithy-go/middleware.(*DeserializeStep).HandleMiddleware /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_deserialize.go:120 s=0
    92: 0x1293fa2 M=1 github.com/aws/smithy-go/middleware.finalizeWrapHandler.HandleFinalize /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_finalize.go:184 s=0
    93: 0x12b918d M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*SignHTTPRequestMiddleware).HandleFinalize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/middleware.go:315 s=0
    94: 0x129406a M=1 github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_finalize.go:200 s=0
    95: 0x12a6001 M=1 github.com/aws/aws-sdk-go-v2/aws/retry.MetricsHeader.HandleFinalize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/retry/middleware.go:284 s=0
    96: 0x12a5067 M=1 github.com/aws/aws-sdk-go-v2/aws/retry.(*Attempt).handleAttempt /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/retry/middleware.go:166 s=0
    97: 0x12a45d0 M=1 github.com/aws/aws-sdk-go-v2/aws/retry.(*Attempt).HandleFinalize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/retry/middleware.go:90 s=0
    98: 0x131c618 M=1 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding.(*DisableGzip).HandleFinalize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.9.1/accept_encoding_gzip.go:67 s=0
    99: 0x1292c6e M=1 github.com/aws/smithy-go/middleware.(*FinalizeStep).HandleMiddleware /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_finalize.go:114 s=0
   100: 0x1293b62 M=1 github.com/aws/smithy-go/middleware.buildWrapHandler.HandleBuild /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/middleware/step_build.go:184 s=0
   101: 0x12a03b7 M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.(*requestUserAgent).HandleBuild /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/user_agent.go:217 s=0
   102: 0x12b89cb M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*computePayloadSHA256).HandleBuild /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/middleware.go:200 s=0
   103: 0x129ce94 M=1 github.com/aws/smithy-go/transport/http.(*ComputeContentLength).HandleBuild /Users/edited/go/pkg/mod/github.com/aws/smithy-go@v1.11.2/transport/http/middleware_content_length.go:49 s=0
   104: 0x129ec4c M=1 github.com/aws/aws-sdk-go-v2/aws/middleware.ClientRequestID.HandleBuild /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/middleware/middleware.go:42 s=0
   105: 0x1054b0b M=1 runtime.madvise /usr/local/Cellar/go/1.18.1/libexec/src/runtime/sys_darwin.go:199 s=0
   106: 0x1027fb8 M=1 runtime.sysUsed /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mem_darwin.go:33 s=0
             runtime.(*mheap).allocSpan /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mheap.go:1279 s=0
   107: 0x102781e M=1 runtime.(*mheap).allocManual /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mheap.go:938 s=0
   108: 0x104e150 M=1 runtime.stackalloc /usr/local/Cellar/go/1.18.1/libexec/src/runtime/stack.go:414 s=0
   109: 0x104eec5 M=1 runtime.copystack /usr/local/Cellar/go/1.18.1/libexec/src/runtime/stack.go:871 s=0
   110: 0x10665ea M=1 runtime.morestack /usr/local/Cellar/go/1.18.1/libexec/src/runtime/asm_amd64.s:547 s=0
   111: 0x104d9c5 M=1 runtime.stackpoolalloc /usr/local/Cellar/go/1.18.1/libexec/src/runtime/stack.go:205 s=0
   112: 0x104dd04 M=1 runtime.stackcacherefill /usr/local/Cellar/go/1.18.1/libexec/src/runtime/stack.go:288 s=0
   113: 0x104e244 M=1 runtime.stackalloc /usr/local/Cellar/go/1.18.1/libexec/src/runtime/stack.go:390 s=0
   114: 0x10277e4 M=1 runtime.(*mheap).alloc.func1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mheap.go:912 s=0
   115: 0x105ba33 M=1 runtime.gentraceback /usr/local/Cellar/go/1.18.1/libexec/src/runtime/traceback.go:232 s=0
   116: 0x106945c M=1 runtime.memclrNoHeapPointers /usr/local/Cellar/go/1.18.1/libexec/src/runtime/memclr_amd64.s:139 s=0
   117: 0x104d688 M=1 runtime.growslice /usr/local/Cellar/go/1.18.1/libexec/src/runtime/slice.go:275 s=0
   118: 0x12bbcb6 M=1 strings.(*Builder).WriteString /usr/local/Cellar/go/1.18.1/libexec/src/strings/builder.go:123 s=0
             github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).buildCanonicalHeaders /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:450 s=0
   119: 0x12b991d M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*httpSigner).Build /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:174 s=0
   120: 0x12bad2e M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.Signer.SignHTTP /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/v4.go:288 s=0
   121: 0x12b8ffc M=1 github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*SignHTTPRequestMiddleware).HandleFinalize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2@v1.16.2/aws/signer/v4/middleware.go:304 s=0
   122: 0x10694b1 M=1 runtime.memclrNoHeapPointers /usr/local/Cellar/go/1.18.1/libexec/src/runtime/memclr_amd64.s:159 s=0
   123: 0x10d3c44 M=1 context.WithDeadline /usr/local/Cellar/go/1.18.1/libexec/src/context/context.go:442 s=0
   124: 0x1166290 M=1 net.(*Dialer).DialContext /usr/local/Cellar/go/1.18.1/libexec/src/net/dial.go:379 s=0
   125: 0x127ca99 M=1 net/http.(*Transport).dial /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:1169 s=0
   126: 0x127fbbe M=1 net/http.(*Transport).dialConn /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:1607 s=0
   127: 0x127e44f M=1 net/http.(*Transport).dialConnFor /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:1449 s=0
   128: 0x1054c8b M=1 runtime.usleep /usr/local/Cellar/go/1.18.1/libexec/src/runtime/sys_darwin.go:248 s=0
   129: 0x100c8ca M=1 runtime.osyield /usr/local/Cellar/go/1.18.1/libexec/src/runtime/os_darwin.go:348 s=0
             runtime.lock2 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/lock_sema.go:71 s=0
   130: 0x104de33 M=1 runtime.lockWithRank /usr/local/Cellar/go/1.18.1/libexec/src/runtime/lockrank_off.go:22 s=0
             runtime.lock /usr/local/Cellar/go/1.18.1/libexec/src/runtime/lock_sema.go:36 s=0
             runtime.stackcacherelease /usr/local/Cellar/go/1.18.1/libexec/src/runtime/stack.go:305 s=0
   131: 0x104e555 M=1 runtime.stackfree /usr/local/Cellar/go/1.18.1/libexec/src/runtime/stack.go:488 s=0
   132: 0x104f0e6 M=1 runtime.copystack /usr/local/Cellar/go/1.18.1/libexec/src/runtime/stack.go:936 s=0
   133: 0x1053620 M=1 runtime.findfunc /usr/local/Cellar/go/1.18.1/libexec/src/runtime/symtab.go:827 s=0
   134: 0x105ba58 M=1 runtime.gentraceback /usr/local/Cellar/go/1.18.1/libexec/src/runtime/traceback.go:235 s=0
   135: 0x105e4f1 M=1 runtime.callers.func1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/traceback.go:903 s=0
   136: 0x105e437 M=1 runtime.callers /usr/local/Cellar/go/1.18.1/libexec/src/runtime/traceback.go:902 s=0
   137: 0x102dc69 M=1 runtime.mProf_Malloc /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mprof.go:343 s=0
   138: 0x100ea24 M=1 runtime.profilealloc /usr/local/Cellar/go/1.18.1/libexec/src/runtime/malloc.go:1295 s=0
   139: 0x100e5b8 M=1 runtime.mallocgc /usr/local/Cellar/go/1.18.1/libexec/src/runtime/malloc.go:1171 s=0
   140: 0x100e931 M=1 runtime.newarray /usr/local/Cellar/go/1.18.1/libexec/src/runtime/malloc.go:1281 s=0
   141: 0x100f78d M=1 runtime.makeBucketArray /usr/local/Cellar/go/1.18.1/libexec/src/runtime/map.go:363 s=0
   142: 0x100f548 M=1 runtime.makemap /usr/local/Cellar/go/1.18.1/libexec/src/runtime/map.go:329 s=0
   143: 0x124e9e9 M=1 net/http.Header.Clone /usr/local/Cellar/go/1.18.1/libexec/src/net/http/header.go:104 s=0
             net/http.cloneOrMakeHeader /usr/local/Cellar/go/1.18.1/libexec/src/net/http/clone.go:69 s=0
   144: 0x124d224 M=1 net/http.(*Client).makeHeadersCopier /usr/local/Cellar/go/1.18.1/libexec/src/net/http/client.go:754 s=0
   145: 0x124c235 M=1 net/http.(*Client).do /usr/local/Cellar/go/1.18.1/libexec/src/net/http/client.go:614 s=0
   146: 0x1055193 M=1 runtime.kevent /usr/local/Cellar/go/1.18.1/libexec/src/runtime/sys_darwin.go:399 s=0
   147: 0x10327d6 M=1 runtime.netpollopen /usr/local/Cellar/go/1.18.1/libexec/src/runtime/netpoll_kqueue.go:67 s=0
   148: 0x1062873 M=1 internal/poll.runtime_pollOpen /usr/local/Cellar/go/1.18.1/libexec/src/runtime/netpoll.go:239 s=0
   149: 0x1091dac M=1 internal/poll.(*pollDesc).init /usr/local/Cellar/go/1.18.1/libexec/src/internal/poll/fd_poll_runtime.go:39 s=0
   150: 0x10924e5 M=1 internal/poll.(*FD).Init /usr/local/Cellar/go/1.18.1/libexec/src/internal/poll/fd_unix.go:63 s=0
   151: 0x116f9f7 M=1 net.(*netFD).connect /usr/local/Cellar/go/1.18.1/libexec/src/net/fd_unix.go:85 s=0
   152: 0x117d3f3 M=1 net.(*netFD).dial /usr/local/Cellar/go/1.18.1/libexec/src/net/sock_posix.go:149 s=0
   153: 0x117cf51 M=1 net.socket /usr/local/Cellar/go/1.18.1/libexec/src/net/sock_posix.go:70 s=0
   154: 0x11778b7 M=1 net.internetSocket /usr/local/Cellar/go/1.18.1/libexec/src/net/ipsock_posix.go:142 s=0
   155: 0x117f584 M=1 net.(*sysDialer).doDialTCP /usr/local/Cellar/go/1.18.1/libexec/src/net/tcpsock_posix.go:65 s=0
   156: 0x117f478 M=1 net.(*sysDialer).dialTCP /usr/local/Cellar/go/1.18.1/libexec/src/net/tcpsock_posix.go:61 s=0
   157: 0x116802a M=1 net.(*sysDialer).dialSingle /usr/local/Cellar/go/1.18.1/libexec/src/net/dial.go:583 s=0
   158: 0x1167971 M=1 net.(*sysDialer).dialSerial /usr/local/Cellar/go/1.18.1/libexec/src/net/dial.go:551 s=0
   159: 0x11667f5 M=1 net.(*Dialer).DialContext /usr/local/Cellar/go/1.18.1/libexec/src/net/dial.go:428 s=0
   160: 0x10503fe M=1 runtime.getStackMap /usr/local/Cellar/go/1.18.1/libexec/src/runtime/stack.go:1319 s=0
   161: 0x104e872 M=1 runtime.adjustframe /usr/local/Cellar/go/1.18.1/libexec/src/runtime/stack.go:665 s=0
   162: 0x105be8e M=1 runtime.gentraceback /usr/local/Cellar/go/1.18.1/libexec/src/runtime/traceback.go:330 s=0
   163: 0x104f0d4 M=1 runtime.copystack /usr/local/Cellar/go/1.18.1/libexec/src/runtime/stack.go:930 s=0
   164: 0x1293831 M=1 github.com/aws/smithy-go/middleware.(*decoratedHandler).Handle <autogenerated>:1 s=0
   165: 0x10048fb M=1 runtime.cgocall /usr/local/Cellar/go/1.18.1/libexec/src/runtime/cgocall.go:157 s=0
   166: 0x1181095 M=1 net._C2func_getaddrinfo _cgo_gotypes.go:100 s=0
   167: 0x1182b1e M=1 net.cgoLookupIPCNAME.func1 /usr/local/Cellar/go/1.18.1/libexec/src/net/cgo_unix.go:160 s=0
   168: 0x118238c M=1 net.cgoLookupIPCNAME /usr/local/Cellar/go/1.18.1/libexec/src/net/cgo_unix.go:160 s=0
   169: 0x1182bda M=1 net.cgoIPLookup /usr/local/Cellar/go/1.18.1/libexec/src/net/cgo_unix.go:217 s=0
   170: 0x1065299 M=1 syscall.rawSyscall /usr/local/Cellar/go/1.18.1/libexec/src/runtime/sys_darwin.go:77 s=0
   171: 0x107674f M=1 syscall.socket /usr/local/Cellar/go/1.18.1/libexec/src/syscall/zsyscall_darwin_amd64.go:101 s=0
   172: 0x1076426 M=1 syscall.Socket /usr/local/Cellar/go/1.18.1/libexec/src/syscall/syscall_unix.go:466 s=0
   173: 0x117ea87 M=1 net.sysSocket /usr/local/Cellar/go/1.18.1/libexec/src/net/sys_cloexec.go:23 s=0
   174: 0x117cd10 M=1 net.socket /usr/local/Cellar/go/1.18.1/libexec/src/net/sock_posix.go:19 s=0
   175: 0x122cf25 M=1 io.ReadFull /usr/local/Cellar/go/1.18.1/libexec/src/io/io.go:350 s=0
             crypto/tls.generateECDHEParameters /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/key_schedule.go:115 s=0
   176: 0x120d9d6 M=1 crypto/tls.(*Conn).makeClientHello /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:135 s=0
   177: 0x120dd95 M=1 crypto/tls.(*Conn).clientHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:154 s=0
   178: 0x11ff152 M=1 vendor/golang.org/x/crypto/curve25519/internal/field.mul51 /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/curve25519/internal/field/fe.go:328 s=0
             vendor/golang.org/x/crypto/curve25519/internal/field.(*Element).Mult32 /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/curve25519/internal/field/fe.go:311 s=0
   179: 0x11ffad5 M=1 vendor/golang.org/x/crypto/curve25519.ScalarMult /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/curve25519/curve25519.go:59 s=0
   180: 0x11fff0f M=1 vendor/golang.org/x/crypto/curve25519.ScalarBaseMult /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/curve25519/curve25519.go:80 s=0
             vendor/golang.org/x/crypto/curve25519.x25519 /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/curve25519/curve25519.go:135 s=0
   181: 0x122cf84 M=1 vendor/golang.org/x/crypto/curve25519.X25519 /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/curve25519/curve25519.go:121 s=0
             crypto/tls.generateECDHEParameters /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/key_schedule.go:118 s=0
   182: 0x1055546 M=1 runtime.libcCall /usr/local/Cellar/go/1.18.1/libexec/src/runtime/sys_libc.go:48 s=0
   183: 0x1055373 M=1 runtime.pthread_cond_wait /usr/local/Cellar/go/1.18.1/libexec/src/runtime/sys_darwin.go:448 s=0
   184: 0x1032e2c M=1 runtime.semasleep /usr/local/Cellar/go/1.18.1/libexec/src/runtime/os_darwin.go:66 s=0
   185: 0x100cbe4 M=1 runtime.notesleep /usr/local/Cellar/go/1.18.1/libexec/src/runtime/lock_sema.go:181 s=0
   186: 0x103c90c M=1 runtime.mPark /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:1449 s=0
             runtime.stopm /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:2228 s=0
   187: 0x103dde4 M=1 runtime.findrunnable /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:2804 s=0
   188: 0x103ed18 M=1 runtime.schedule /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:3187 s=0
   189: 0x103f26c M=1 runtime.park_m /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:3336 s=0
   190: 0x1066482 M=1 runtime.mcall /usr/local/Cellar/go/1.18.1/libexec/src/runtime/asm_amd64.s:425 s=0
   191: 0x120d6a4 M=1 io.ReadFull /usr/local/Cellar/go/1.18.1/libexec/src/io/io.go:350 s=0
             crypto/tls.(*Conn).makeClientHello /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:107 s=0
   192: 0x11ff3b3 M=1 vendor/golang.org/x/crypto/curve25519/internal/field.feMul /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/curve25519/internal/field/fe_amd64.s:96 s=0
   193: 0x11ff986 M=1 vendor/golang.org/x/crypto/curve25519/internal/field.(*Element).Multiply /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/curve25519/internal/field/fe.go:299 s=0
             vendor/golang.org/x/crypto/curve25519.ScalarMult /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/curve25519/curve25519.go:49 s=0
   194: 0x11ff445 M=1 vendor/golang.org/x/crypto/curve25519/internal/field.feMul /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/curve25519/internal/field/fe_amd64.s:158 s=0
   195: 0x11ffb35 M=1 vendor/golang.org/x/crypto/curve25519/internal/field.(*Element).Multiply /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/curve25519/internal/field/fe.go:299 s=0
             vendor/golang.org/x/crypto/curve25519.ScalarMult /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/curve25519/curve25519.go:62 s=0
   196: 0x100e0a2 M=1 runtime.mallocgc /usr/local/Cellar/go/1.18.1/libexec/src/runtime/malloc.go:1047 s=0
   197: 0x11c7665 M=1 vendor/golang.org/x/crypto/cryptobyte.(*Builder).addLengthPrefixed /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/cryptobyte/builder.go:187 s=0
   198: 0x1217144 M=1 vendor/golang.org/x/crypto/cryptobyte.(*Builder).AddUint24LengthPrefixed /usr/local/Cellar/go/1.18.1/libexec/src/vendor/golang.org/x/crypto/cryptobyte/builder.go:146 s=0
             crypto/tls.(*clientHelloMsg).marshal /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_messages.go:104 s=0
   199: 0x120decd M=1 crypto/tls.(*Conn).clientHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:175 s=0
   200: 0x10778c8 M=1 syscall.write /usr/local/Cellar/go/1.18.1/libexec/src/syscall/zsyscall_darwin_amd64.go:1653 s=0
   201: 0x10931ed M=1 syscall.Write /usr/local/Cellar/go/1.18.1/libexec/src/syscall/syscall_unix.go:216 s=0
             internal/poll.ignoringEINTRIO /usr/local/Cellar/go/1.18.1/libexec/src/internal/poll/fd_unix.go:794 s=0
             internal/poll.(*FD).Write /usr/local/Cellar/go/1.18.1/libexec/src/internal/poll/fd_unix.go:383 s=0
   202: 0x116f768 M=1 net.(*netFD).Write /usr/local/Cellar/go/1.18.1/libexec/src/net/fd_posix.go:96 s=0
   203: 0x117a724 M=1 net.(*conn).Write /usr/local/Cellar/go/1.18.1/libexec/src/net/net.go:195 s=0
   204: 0x1209307 M=1 crypto/tls.(*Conn).write /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:919 s=0
   205: 0x12097d0 M=1 crypto/tls.(*Conn).writeRecordLocked /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:987 s=0
   206: 0x1209cc4 M=1 crypto/tls.(*Conn).writeRecord /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:1009 s=0
   207: 0x120dee8 M=1 crypto/tls.(*Conn).clientHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:175 s=0
   208: 0x106939c M=1 runtime.memclrNoHeapPointers /usr/local/Cellar/go/1.18.1/libexec/src/runtime/memclr_amd64.s:77 s=0
   209: 0x104d131 M=1 runtime.makeslice /usr/local/Cellar/go/1.18.1/libexec/src/runtime/slice.go:103 s=0
   210: 0x10ce5c4 M=1 bytes.makeSlice /usr/local/Cellar/go/1.18.1/libexec/src/bytes/buffer.go:229 s=0
   211: 0x10cdf9e M=1 bytes.(*Buffer).grow /usr/local/Cellar/go/1.18.1/libexec/src/bytes/buffer.go:142 s=0
   212: 0x1208b76 M=1 bytes.(*Buffer).Grow /usr/local/Cellar/go/1.18.1/libexec/src/bytes/buffer.go:161 s=0
             crypto/tls.(*Conn).readFromUntil /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:805 s=0
   213: 0x1206bc4 M=1 crypto/tls.(*Conn).readRecordOrCCS /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:658 s=0
   214: 0x1209e2c M=1 crypto/tls.(*Conn).readRecord /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:581 s=0
             crypto/tls.(*Conn).readHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:1016 s=0
   215: 0x120f9c4 M=1 crypto/tls.(*clientHandshakeState).doFullHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:471 s=0
   216: 0x122dae4 M=1 crypto/tls.prf12 /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/prf.go:68 s=0
   217: 0x122dd44 M=1 crypto/tls.prfAndHashForVersion /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/prf.go:95 s=0
   218: 0x122e168 M=1 crypto/tls.prfForVersion /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/prf.go:102 s=0
             crypto/tls.keysFromMasterSecret /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/prf.go:128 s=0
   219: 0x1210df6 M=1 crypto/tls.(*clientHandshakeState).establishKeys /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:659 s=0
   220: 0x120f65a M=1 crypto/tls.(*clientHandshakeState).handshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:434 s=0
   221: 0x11edb24 M=1 crypto/x509/internal/macos.SecCertificateCreateWithData /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/internal/macos/security.go:136 s=0
   222: 0x11f6ded M=1 crypto/x509.(*Certificate).systemVerify /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/root_darwin.go:26 s=0
   223: 0x1027e2f M=1 runtime.(*mheap).allocSpan /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mheap.go:1227 s=0
   224: 0x1027724 M=1 runtime.(*mheap).alloc /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mheap.go:906 s=0
   225: 0x101897a M=1 runtime.(*mcentral).grow /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mcentral.go:244 s=0
   226: 0x10187ae M=1 runtime.(*mcentral).cacheSpan /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mcentral.go:164 s=0
   227: 0x1017e2e M=1 runtime.(*mcache).refill /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mcache.go:162 s=0
   228: 0x100dce4 M=1 runtime.(*mcache).nextFree /usr/local/Cellar/go/1.18.1/libexec/src/runtime/malloc.go:886 s=0
   229: 0x100e364 M=1 runtime.mallocgc /usr/local/Cellar/go/1.18.1/libexec/src/runtime/malloc.go:1085 s=0
   230: 0x104d6a9 M=1 runtime.growslice /usr/local/Cellar/go/1.18.1/libexec/src/runtime/slice.go:278 s=0
   231: 0x11e612c M=1 crypto/x509/pkix.(*Name).FillFromRDNSequence /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/pkix/pkix.go:151 s=0
   232: 0x11f566d M=1 crypto/x509.parseCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/parser.go:912 s=0
   233: 0x11f62a4 M=1 crypto/x509.ParseCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/parser.go:990 s=0
   234: 0x121232e M=1 crypto/tls.(*Conn).verifyServerCertificate /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:849 s=0
   235: 0x7ff81c3f31ac M=1  :0 s=0
   236: 0x1041a60 M=1 runtime._ExternalCode /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:4433 s=0
   237: 0x11edde7 M=1 crypto/x509/internal/macos.SecTrustEvaluateWithError /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/internal/macos/security.go:195 s=0
   238: 0x11f6759 M=1 crypto/x509.(*Certificate).systemVerify /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/root_darwin.go:57 s=0
   239: 0x1026752 M=1 runtime.getempty.func1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgcwork.go:375 s=0
   240: 0x10265eb M=1 runtime.getempty /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgcwork.go:374 s=0
   241: 0x10240b8 M=1 runtime.(*stackScanState).addObject /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgcstack.go:288 s=0
   242: 0x101f63d M=1 runtime.scanframeworker /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgcmark.go:978 s=0
   243: 0x101f404 M=1 runtime.scanstack.func1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgcmark.go:780 s=0
   244: 0x101552d M=1 runtime.findObject /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mbitmap.go:402 s=0
   245: 0x101fd77 M=1 runtime.scanblock /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgcmark.go:1215 s=0
   246: 0x101f4fc M=1 runtime.scanframeworker /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgcmark.go:950 s=0
   247: 0x1032a0b M=1 runtime.netpoll /usr/local/Cellar/go/1.18.1/libexec/src/runtime/netpoll_kqueue.go:127 s=0
   248: 0x103dfb4 M=1 runtime.pollWork /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:2821 s=0
   249: 0x101fa31 M=1 runtime.gcDrain /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgcmark.go:1048 s=0
   250: 0x101c86d M=1 runtime.gcBgMarkWorker.func2 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mgc.go:1295 s=0
   251: 0x10776e8 M=1 syscall.read /usr/local/Cellar/go/1.18.1/libexec/src/syscall/zsyscall_darwin_amd64.go:1189 s=0
   252: 0x1092b24 M=1 syscall.Read /usr/local/Cellar/go/1.18.1/libexec/src/syscall/syscall_unix.go:188 s=0
             internal/poll.ignoringEINTRIO /usr/local/Cellar/go/1.18.1/libexec/src/internal/poll/fd_unix.go:794 s=0
             internal/poll.(*FD).Read /usr/local/Cellar/go/1.18.1/libexec/src/internal/poll/fd_unix.go:163 s=0
   253: 0x116f668 M=1 net.(*netFD).Read /usr/local/Cellar/go/1.18.1/libexec/src/net/fd_posix.go:55 s=0
   254: 0x117a544 M=1 net.(*conn).Read /usr/local/Cellar/go/1.18.1/libexec/src/net/net.go:183 s=0
   255: 0x12089fc M=1 crypto/tls.(*atLeastReader).Read /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:784 s=0
   256: 0x10ce3f7 M=1 bytes.(*Buffer).ReadFrom /usr/local/Cellar/go/1.18.1/libexec/src/bytes/buffer.go:204 s=0
   257: 0x1208be4 M=1 crypto/tls.(*Conn).readFromUntil /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:806 s=0
   258: 0x12061f5 M=1 crypto/tls.(*Conn).readRecordOrCCS /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:613 s=0
   259: 0x120bc4e M=1 crypto/tls.(*Conn).readRecord /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:581 s=0
             crypto/tls.(*Conn).Read /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:1284 s=0
   260: 0x1281f8d M=1 net/http.(*persistConn).Read /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:1929 s=0
   261: 0x10e6802 M=1 bufio.(*Reader).fill /usr/local/Cellar/go/1.18.1/libexec/src/bufio/bufio.go:106 s=0
   262: 0x10e695c M=1 bufio.(*Reader).Peek /usr/local/Cellar/go/1.18.1/libexec/src/bufio/bufio.go:144 s=0
   263: 0x1282dab M=1 net/http.(*persistConn).readLoop /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:2093 s=0
   264: 0x1069696 M=1 runtime.memmove /usr/local/Cellar/go/1.18.1/libexec/src/runtime/memmove_amd64.s:190 s=0
   265: 0x1014e24 M=1 runtime.typedmemmove /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mbarrier.go:171 s=0
   266: 0x11e61e7 M=1 crypto/x509/pkix.(*Name).FillFromRDNSequence /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/pkix/pkix.go:151 s=0
   267: 0x10307c3 M=1 runtime.(*spanSet).push /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mspanset.go:129 s=0
   268: 0x10188ac M=1 runtime.(*mcentral).uncacheSpan /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mcentral.go:234 s=0
   269: 0x1017e11 M=1 runtime.(*mcache).refill /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mcache.go:158 s=0
   270: 0x114cffb M=1 math/big.nat.make /usr/local/Cellar/go/1.18.1/libexec/src/math/big/nat.go:69 s=0
             math/big.nat.mul /usr/local/Cellar/go/1.18.1/libexec/src/math/big/nat.go:429 s=0
   271: 0x114f854 M=1 math/big.nat.expNN /usr/local/Cellar/go/1.18.1/libexec/src/math/big/nat.go:962 s=0
   272: 0x1147c38 M=1 math/big.(*Int).Exp /usr/local/Cellar/go/1.18.1/libexec/src/math/big/int.go:509 s=0
   273: 0x11d0504 M=1 crypto/rsa.encrypt /usr/local/Cellar/go/1.18.1/libexec/src/crypto/rsa/rsa.go:389 s=0
   274: 0x11cf618 M=1 crypto/rsa.VerifyPKCS1v15 /usr/local/Cellar/go/1.18.1/libexec/src/crypto/rsa/pkcs1v15.go:286 s=0
   275: 0x120084b M=1 crypto/tls.verifyHandshakeSignature /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/auth.go:45 s=0
   276: 0x122b7e4 M=1 crypto/tls.(*ecdheKeyAgreement).processServerKeyExchange /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/key_agreement.go:345 s=0
   277: 0x120fe39 M=1 crypto/tls.(*clientHandshakeState).doFullHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:534 s=0
   278: 0x11f64a4 M=1 crypto/x509.(*Certificate).systemVerify /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/root_darwin.go:15 s=0
   279: 0x103d704 M=1 runtime.findrunnable /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:2576 s=0
   280: 0x103fcf6 M=1 runtime.goexit0 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:3513 s=0
   281: 0x1055570 M=1 runtime.libcCall /usr/local/Cellar/go/1.18.1/libexec/src/runtime/sys_libc.go:52 s=0
   282: 0x11ed6d3 M=1 crypto/x509/internal/macos.CFRelease /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/internal/macos/corefoundation.go:152 s=0
   283: 0x11ed90b M=1 crypto/x509/internal/macos.ReleaseCFArray /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/internal/macos/corefoundation.go:204 s=0
   284: 0x11f6cbe M=1 crypto/x509.(*Certificate).systemVerify /usr/local/Cellar/go/1.18.1/libexec/src/crypto/x509/root_darwin.go:99 s=0
   285: 0x120aad3 M=1 crypto/tls.(*Conn).Write /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/conn.go:1158 s=0
   286: 0x108e883 M=1 io.copyBuffer /usr/local/Cellar/go/1.18.1/libexec/src/io/io.go:428 s=0
   287: 0x1281b5a M=1 io.Copy /usr/local/Cellar/go/1.18.1/libexec/src/io/io.go:385 s=0
             net/http.persistConnWriter.ReadFrom /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:1775 s=0
   288: 0x10e89ea M=1 bufio.(*Writer).ReadFrom /usr/local/Cellar/go/1.18.1/libexec/src/bufio/bufio.go:768 s=0
   289: 0x108e7ca M=1 io.copyBuffer /usr/local/Cellar/go/1.18.1/libexec/src/io/io.go:412 s=0
   290: 0x127526c M=1 io.Copy /usr/local/Cellar/go/1.18.1/libexec/src/io/io.go:385 s=0
             net/http.(*transferWriter).doBodyCopy /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transfer.go:411 s=0
   291: 0x1274db4 M=1 net/http.(*transferWriter).writeBody /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transfer.go:369 s=0
   292: 0x126d8c5 M=1 net/http.(*Request).write /usr/local/Cellar/go/1.18.1/libexec/src/net/http/request.go:698 s=0
   293: 0x1284b24 M=1 net/http.(*persistConn).writeLoop /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:2395 s=0
   294: 0x1076fee M=1 syscall.Close /usr/local/Cellar/go/1.18.1/libexec/src/syscall/zsyscall_darwin_amd64.go:509 s=0
   295: 0x10925b0 M=1 internal/poll.(*FD).destroy /usr/local/Cellar/go/1.18.1/libexec/src/internal/poll/fd_unix.go:84 s=0
   296: 0x1091cd2 M=1 internal/poll.(*FD).readUnlock /usr/local/Cellar/go/1.18.1/libexec/src/internal/poll/fd_mutex.go:232 s=0
   297: 0x1092bbd M=1 internal/poll.(*FD).Read /usr/local/Cellar/go/1.18.1/libexec/src/internal/poll/fd_unix.go:173 s=0
   298: 0x120fa4a M=1 crypto/tls.(*clientHandshakeState).doFullHandshake /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/handshake_client.go:482 s=0
   299: 0x1076a3e M=1 syscall.getpeername /usr/local/Cellar/go/1.18.1/libexec/src/syscall/zsyscall_darwin_amd64.go:144 s=0
   300: 0x1076309 M=1 syscall.Getpeername /usr/local/Cellar/go/1.18.1/libexec/src/syscall/syscall_unix.go:275 s=0
   301: 0x1170075 M=1 net.(*netFD).connect /usr/local/Cellar/go/1.18.1/libexec/src/net/fd_unix.go:161 s=0
   302: 0x7ff81c3a3ad5 M=1  :0 s=0
   303: 0x10307f4 M=1 runtime.(*spanSet).push /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mspanset.go:77 s=0
   304: 0x100f404 M=1 runtime.makemap_small /usr/local/Cellar/go/1.18.1/libexec/src/runtime/map.go:294 s=0
   305: 0x10f9c64 M=1 encoding/json.(*decodeState).objectInterface /usr/local/Cellar/go/1.18.1/libexec/src/encoding/json/decode.go:1084 s=0
   306: 0x10f6205 M=1 encoding/json.(*decodeState).object /usr/local/Cellar/go/1.18.1/libexec/src/encoding/json/decode.go:624 s=0
   307: 0x10f4d04 M=1 encoding/json.(*decodeState).value /usr/local/Cellar/go/1.18.1/libexec/src/encoding/json/decode.go:373 s=0
   308: 0x10f46dd M=1 encoding/json.(*decodeState).unmarshal /usr/local/Cellar/go/1.18.1/libexec/src/encoding/json/decode.go:180 s=0
   309: 0x1105f96 M=1 encoding/json.(*Decoder).Decode /usr/local/Cellar/go/1.18.1/libexec/src/encoding/json/stream.go:73 s=0
   310: 0x1326a04 M=1 github.com/aws/aws-sdk-go-v2/service/dynamodb.(*awsAwsjson10_deserializeOpPutItem).HandleDeserialize /Users/edited/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.15.3/deserializers.go:4000 s=0
   311: 0x11b34a9 M=1 crypto/elliptic.p256PointDoubleAsm /usr/local/Cellar/go/1.18.1/libexec/src/crypto/elliptic/p256_asm_amd64.s:2312 s=0
   312: 0x11af024 M=1 crypto/elliptic.(*p256Point).p256ScalarMult /usr/local/Cellar/go/1.18.1/libexec/src/crypto/elliptic/p256_asm.go:510 s=0
   313: 0x11ad04b M=1 crypto/elliptic.p256Curve.ScalarMult /usr/local/Cellar/go/1.18.1/libexec/src/crypto/elliptic/p256_asm.go:292 s=0
   314: 0x122d3e4 M=1 crypto/tls.(*nistParameters).SharedKey /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/key_schedule.go:175 s=0
   315: 0x122b342 M=1 crypto/tls.(*ecdheKeyAgreement).processServerKeyExchange /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/key_agreement.go:301 s=0
   316: 0x11a9ed8 M=1 io.ReadFull /usr/local/Cellar/go/1.18.1/libexec/src/io/io.go:350 s=0
             crypto/elliptic.GenerateKey /usr/local/Cellar/go/1.18.1/libexec/src/crypto/elliptic/elliptic.go:339 s=0
   317: 0x122d0c4 M=1 crypto/tls.generateECDHEParameters /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/key_schedule.go:132 s=0
   318: 0x122b2e4 M=1 crypto/tls.(*ecdheKeyAgreement).processServerKeyExchange /usr/local/Cellar/go/1.18.1/libexec/src/crypto/tls/key_agreement.go:295 s=0
   319: 0x1055433 M=1 runtime.pthread_cond_signal /usr/local/Cellar/go/1.18.1/libexec/src/runtime/sys_darwin.go:469 s=0
   320: 0x1032fe4 M=1 runtime.semawakeup /usr/local/Cellar/go/1.18.1/libexec/src/runtime/os_darwin.go:76 s=0
   321: 0x100cb2d M=1 runtime.notewakeup /usr/local/Cellar/go/1.18.1/libexec/src/runtime/lock_sema.go:161 s=0
   322: 0x103cba8 M=1 runtime.startm /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:2324 s=0
   323: 0x103cfb9 M=1 runtime.wakep /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:2404 s=0
   324: 0x1039f34 M=1 runtime.ready /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:859 s=0
   325: 0x1038ca5 M=1 runtime.goready.func1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:372 s=0
   326: 0x1281aae M=1 net/http.persistConnWriter.Write /usr/local/Cellar/go/1.18.1/libexec/src/net/http/transport.go:1766 s=0
   327: 0x10e80e1 M=1 bufio.(*Writer).Flush /usr/local/Cellar/go/1.18.1/libexec/src/bufio/bufio.go:628 s=0
   328: 0x126d85e M=1 net/http.(*Request).write /usr/local/Cellar/go/1.18.1/libexec/src/net/http/request.go:691 s=0
   329: 0x1045344 M=1 runtime.runqgrab /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:5879 s=0
   330: 0x104549c M=1 runtime.runqsteal /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:5914 s=0
   331: 0x103e291 M=1 runtime.stealWork /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:2891 s=0
   332: 0x103d78b M=1 runtime.findrunnable /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:2599 s=0
   333: 0x1040d52 M=1 runtime.newproc.func1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:4062 s=0
   334: 0x103db12 M=1 runtime.findrunnable /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:2767 s=0
   335: 0x105391e M=1 runtime.pcvalue /usr/local/Cellar/go/1.18.1/libexec/src/runtime/symtab.go:915 s=0
   336: 0x1054136 M=1 runtime.funcline1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/symtab.go:1024 s=0
   337: 0x105d9ab M=1 runtime.funcline /usr/local/Cellar/go/1.18.1/libexec/src/runtime/symtab.go:1034 s=0
             runtime.printcreatedby1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/traceback.go:773 s=0
   338: 0x105d8dd M=1 runtime.printcreatedby /usr/local/Cellar/go/1.18.1/libexec/src/runtime/traceback.go:763 s=0
   339: 0x105dde7 M=1 runtime.traceback1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/traceback.go:842 s=0
   340: 0x105ec84 M=1 runtime.traceback /usr/local/Cellar/go/1.18.1/libexec/src/runtime/traceback.go:782 s=0
             runtime.tracebackothers.func1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/traceback.go:1051 s=0
   341: 0x10396cc M=1 runtime.forEachGRace /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:590 s=0
   342: 0x105eb7a M=1 runtime.tracebackothers /usr/local/Cellar/go/1.18.1/libexec/src/runtime/traceback.go:1037 s=0
   343: 0x102f365 M=1 runtime.Stack.func1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mprof.go:867 s=0
   344: 0x102f22e M=1 runtime.Stack /usr/local/Cellar/go/1.18.1/libexec/src/runtime/mprof.go:857 s=0
   345: 0x13442ef M=1 go.uber.org/goleak/internal/stack.getStackBuffer /Users/edited/go/pkg/mod/go.uber.org/goleak@v1.1.12/internal/stack/stacks.go:124 s=0
   346: 0x1343c2d M=1 go.uber.org/goleak/internal/stack.getStacks /Users/edited/go/pkg/mod/go.uber.org/goleak@v1.1.12/internal/stack/stacks.go:73 s=0
   347: 0x1344d24 M=1 go.uber.org/goleak/internal/stack.All /Users/edited/go/pkg/mod/go.uber.org/goleak@v1.1.12/internal/stack/stacks.go:113 s=0
             go.uber.org/goleak.Find /Users/edited/go/pkg/mod/go.uber.org/goleak@v1.1.12/leaks.go:61 s=0
   348: 0x1344e14 M=1 go.uber.org/goleak.VerifyNone /Users/edited/go/pkg/mod/go.uber.org/goleak@v1.1.12/leaks.go:77 s=0
   349: 0x1345c4c M=1 github.com/edited/sdkbug/pkg/database.TestConnectionReuseForPutContentHash /Users/edited/SINGU/src/gosdkv2-bug/pkg/database/database_test.go:71 s=0
   350: 0x10e1641 M=1 testing.tRunner /usr/local/Cellar/go/1.18.1/libexec/src/testing/testing.go:1439 s=0
   351: 0x103b144 M=1 runtime.startTheWorldWithSema /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:1257 s=0
   352: 0x103acba M=1 runtime.startTheWorld.func1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:1104 s=0
   353: 0x1037982 M=1 runtime.printunlock /usr/local/Cellar/go/1.18.1/libexec/src/runtime/print.go:78 s=0
   354: 0x105c935 M=1 runtime.gentraceback /usr/local/Cellar/go/1.18.1/libexec/src/runtime/traceback.go:446 s=0
   355: 0x105dd70 M=1 runtime.traceback1 /usr/local/Cellar/go/1.18.1/libexec/src/runtime/traceback.go:835 s=0
Mappings
1: 0x0/0x0/0x0

@vudh1 vudh1 self-assigned this Apr 15, 2022
@pcolazurdo pcolazurdo mentioned this issue Apr 19, 2022
Closed
3 tasks
@skmcgrail
Copy link
Member

skmcgrail commented Apr 27, 2022 •
edited
Loading

I believe you may be confusing the TCP socket keepalive with the HTTP keepalive. Your code snippet that sets KeepAlive to -1 is setting it on the network dialer. You should be setting DisableKeepAlives on the HTTP transport to disable HTTP keep-alive. If HTTP keep-alive is on (which it is by default) then you will see the Go routines remain active that are spawned by the Go HTTP standard library.

See similar thread here #1434 about this topic around using goleak.

@skmcgrail
Copy link
Member

skmcgrail commented Apr 27, 2022 •
edited
Loading

For your first issue can you enable logging of service responses and retries? See the logging developer guide entry, and aws.ClientLogMode? The error message you got about retry quotas being exceeded means that you are getting some sort of retryable error (like a throttle) from the service, and you are exceeding all attempts. It would be helpful to capture what errors you are get exactly.

@skmcgrail skmcgrail removed the needs-triage This issue or PR still needs to be triaged. label Apr 27, 2022
@pcolazurdo
Copy link
Author

pcolazurdo commented May 5, 2022 •
edited
Loading

I've done a few additional tests with some interesting results:

  • Running this against a local dynamodb doesn't panic at all. Goroutines are kept under 200 and it always finishes cleanly

Now, running against DynamoDB in eu-west-1

  • Running with aws.LogRetries shows 0 messages.
  • Running with aws.LogRetries|aws.LogResponseEventMessage|aws.LogResponse shows the same number of requests as HTTP/1.1 200 OK
❯ grep "SDK 2022/05/04" /tmp/log.txt| wc -l
   11382
❯ grep "HTTP/1.1 200 OK" /tmp/log.txt| wc -l
   11382

About DisableKeepAlives, you were right, this seems to do the trick. I played before with DisableKeepAlives and didn't see much difference, but trying again, this setting seems to solve the problem at hand.

Now, this creates the question about why the client isn't better at managing the pool of existing connections. Disabling keep-alives help but it makes the performance much worse (30% less TPS even on local DynamoDB) as there is a need to re connect (including TLS handshake) for each new item. In other AWS SDKs, the recommendation is to enable keep-alives

@pcolazurdo
Copy link
Author

pcolazurdo commented May 13, 2022 •
edited
Loading

Ok, additional tests show that forcing the SDK client to use the HTTP endpoint avoids the crashes I kept seeing. This allows me to enable KeepAlives again and regain performance, but with the obvious risk of not having encrypted traffic between my client and the server.
The combination of keep-alives and tls enabled produce panics when trying to call DynamoDB with many items (500 concurrent and 10000 items total) - not sure where the problem lies, yet

@jarreds
Copy link

jarreds commented Aug 2, 2022

I'm seeing this same issue. This is fully reproducible in a unit test I have against a local DynamoDB.

DynamoDB: Query, failed to get rate limit token, retry quota exceeded, 3 available, 5 requested
  1. The error does not occur on Linux -- macOS only
  2. If I turn on the Golang race detector the error does not occur (which points to a tricky bug in the SDK or one of it's dependencies)

@vudh1 vudh1 removed their assignment Aug 25, 2022
@jhebert-coursera jhebert-coursera mentioned this issue Sep 21, 2022
Merged
@RanVaknin RanVaknin added p3 This is a minor priority issue l Effort estimation: large xl Effort estimation: very large and removed l Effort estimation: large labels Nov 14, 2022
@monkeybunnies
Copy link

Is there a workaround for this problem? With v1 go sdk package, can this same problem be reproduced?

@buddhike
Copy link

I was experiencing a similar issue with Kinesis client. Based on my investigation, this error is triggered by throttling implemented in retry module.

In my case, I was using the same client instance across a very large number of Go routines. As a result when retries are attempted, all Go routines use the same token bucket and run out of tokens.
Good news is, token bucket size is configurable. I was able to fix this by adjusting the number of tokens when constructing the config as shown below.

cfg, err := config.LoadDefaultConfig(ctx, config.WithRetryer(func() aws.Retryer {
	return retry.NewStandard(func(so *retry.StandardOptions) {
		so.RateLimiter = ratelimit.NewTokenRateLimit(1000000)
	})
}))

@yuseferi
Copy link

yuseferi commented Jun 21, 2023 •
edited
Loading

I have teh same problem, I'm going to test @buddhike solution + maybe after that disable ( or make it less time that the current )
keep-alive :)

@anton-cn
Copy link

anton-cn commented Jun 28, 2023 •
edited
Loading

Hey guys! I had somewhat similar problem with AWS SDK v1 and running PutItem in multiple goroutines. And the solution was using custom HTTPClient with a timeout. Something like (SDK v1)

awsCfg := aws.NewConfig().WithRegion(region).WithHTTPClient(&http.Client{Timeout: time.Second * 5})
ddbClient := dynamodb.New(sess, awsCfg)

You can read some details here https://medium.com/@nate510/don-t-use-go-s-default-http-client-4804cb19f779
I hope that help to fix your issue

@yuseferi
Copy link

yuseferi commented Jul 3, 2023

@buddhike I tried it, it resolved the problem of throttling but I think this number is too much, and every time a retry is required it creates a new limiter, do you think it cause the performance issue?

is there any way to say if the error type is Token Throtteling increase token limiter? not for all the retries ?

@arielshaqed arielshaqed mentioned this issue Sep 19, 2023
Closed
@N-o-Z N-o-Z mentioned this issue Sep 28, 2023
Open
@RobbertDM
Copy link

RobbertDM commented Dec 6, 2023 •
edited
Loading

I am experiencing the same with Athena Workgroups using the aws terraform provider.

│ Error: reading Athena WorkGroup (...): operation error Athena: GetWorkGroup, failed to get rate limit token, retry quota exceeded, 0 available, 5 requested

But also

│ Error: listing tags for Athena WorkGroup (arn:aws:athena:...): operation error Athena: ListTagsForResource, failed to get rate limit token, retry quota exceeded, 0 available, 5 requested

@RobbertDM RobbertDM mentioned this issue Dec 6, 2023
Closed
@RobbertDM
Copy link

@pcolazurdo
Copy link
Author

I've done some more investigations but haven't been able to publish them as they are hard to repro, and I haven't had enough time on my pockte so far. I'll see if I can publish something clearer during the holidays, but as far as I've been able to determine, the issue is with underlying Go and the crypto/tls package. I was able to repro the same issue even without using the AWS SDK. So far all my analysis point to that. I'll try to write a bug report on the Go SDK, once I get some time

@lucix-aws
Copy link
Contributor

failed to get rate limit token, retry quota exceeded, 0 available, 5 requested

This behavior is expected, but I'd say we've clearly done a poor job of communicating how this works in our public documentation.

This error surfaces from a client-side rate-limiting mechanism introduced in SDK v2 (it's part of our newer cross-SDK specification for retry behavior that was meant to standardize how this works across SDKs). @buddhike's comment above is accurate at a high level.

Fortunately, this is configurable and @buddhike's solution is going to be functionally adequate for most in the absence of an explicit "off" switch. @yuseferi mentioned performance concerns, but there shouldn't be any - each operation still has the configured maximum number of attempts that it will obey.

I'm going to address two things here as followup:

  • document this in our retries and timeouts dev guide entry
  • add an explicit "no-op" rate limiting implementation that you can pull in when configuring a retryer

@github-actions GitHub Actions
Copy link

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

@lucix-aws lucix-aws mentioned this issue Mar 18, 2024
Merged
@lucix-aws lucix-aws removed bug This issue is a bug. p3 This is a minor priority issue xl Effort estimation: very large labels Mar 18, 2024
@lucix-aws lucix-aws mentioned this issue Mar 19, 2024
Merged
@stanhu stanhu mentioned this issue Oct 9, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests