panic: runtime error: invalid memory address or nil pointer dereference

[hikariatama]

I am not a Go expert, but this seems like the issue with S3 API. I am using Backblaze B2 bucket with Lifecycle configured to "Keep last version" - maybe it's the problem after all ❓

Log:

museum-1  | INFO[0100]delete.go:74 deleteEmbedding Deleting from all datacenters [ru-1]          item=10008021 queue_id=15954
museum-1  | panic: runtime error: invalid memory address or nil pointer dereference
museum-1  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8b1eb4]
museum-1  | 
museum-1  | goroutine 124 [running]:
museum-1  | github.com/aws/aws-sdk-go/aws/client.(*Client).NewRequest(...)
museum-1  |     /go/pkg/mod/github.com/aws/aws-sdk-go@v1.34.13/aws/client/client.go:85
museum-1  | github.com/aws/aws-sdk-go/service/s3.(*S3).newRequest(0x4000097720?, 0x50?, {0x127f2e0?, 0x4000097720?}, {0x127af60?, 0x4001222b00?})
museum-1  |     /go/pkg/mod/github.com/aws/aws-sdk-go@v1.34.13/service/s3/service.go:95 +0x34
museum-1  | github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsV2Request(0x48?, 0x4000097720)
museum-1  |     /go/pkg/mod/github.com/aws/aws-sdk-go@v1.34.13/service/s3/api.go:5983 +0x1d0
museum-1  | github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsV2(0x10ea0c0?, 0x4001487f80?)
museum-1  |     /go/pkg/mod/github.com/aws/aws-sdk-go@v1.34.13/service/s3/api.go:6031 +0x1c
museum-1  | github.com/ente-io/museum/pkg/controller.(*ObjectCleanupController).DeleteAllObjectsWithPrefix(0x4001123b30, {0x40010b8bd0, 0x22}, {0x400131f898, 0x4})museum-1  |     /etc/ente/pkg/controller/object_cleanup.go:258 +0x150
museum-1  | github.com/ente-io/museum/pkg/controller/embedding.(*Controller).deleteEmbedding(0x400112cb80, {0x130a021?, {0x400131ea78?, 0x1?}})
museum-1  |     /etc/ente/pkg/controller/embedding/delete.go:77 +0x5c8
museum-1  | github.com/ente-io/museum/pkg/controller/embedding.(*Controller).CleanupDeletedEmbeddings(0x400112cb80)
museum-1  |     /etc/ente/pkg/controller/embedding/delete.go:42 +0x194
museum-1  | main.setupAndStartCrons.func9()
museum-1  |     /etc/ente/cmd/museum/main.go:872 +0x20
museum-1  | github.com/robfig/cron/v3.FuncJob.Run(0x0?)
museum-1  |     /go/pkg/mod/github.com/robfig/cron/v3@v3.0.1/cron.go:136 +0x24
museum-1  | github.com/robfig/cron/v3.(*Cron).startJob.func1()
museum-1  |     /go/pkg/mod/github.com/robfig/cron/v3@v3.0.1/cron.go:312 +0x60
museum-1  | created by github.com/robfig/cron/v3.(*Cron).startJob in goroutine 76
museum-1  |     /go/pkg/mod/github.com/robfig/cron/v3@v3.0.1/cron.go:310 +0xa8

After manually deleting item from db the problem solves itself for a while, but then another broken embedding appears. I am on the last version of ente.

[mnvr]

I think the issue might be in the configuration of the "ru-1" datacenter. From the backtrace it seems that the AWS SDK gets angry when we try to create a new instance of the client, so the code is likely passing some malformed data to it when creating it here - https://github.com/ente-io/ente/blob/main/server/pkg/utils/s3config/s3config.go#L121

So double check your s3 config block in the yaml.

[hikariatama]

Most probably the bug is in my shitty migration between data centers. Probably the guide would be pretty neat. I just copied the files between two buckets and changed the yaml configs. As I see, I need to change something in Postgres as well