[Bug]: S3 storage doesn't work with index node (curlCode: 3, URL using bad/illegal format or missing URL)

[mkotsalainen]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: v2.2.10 - Helm chart 4.0.25
- Deployment mode(standalone or cluster): cluster

Current Behavior

After switching to S3 storage from Minio I can’t index my data any more. I see that data gets into s3 the bucket and it shows up in milvus-attu but when I try to load the collection nothing happens. When I check the logs I see this:

indexnode [2023/07/19 07:21:06.702 +00:00] [WARN] [indexcgowrapper/helper.go:76] ["failed to create index, C Runtime Exception: [UnexpectedError] Error:GetObjectS
ize:  curlCode: 3, URL using bad/illegal format or missing URL\n"]

These are my Heml chart settings (I use IRSA roles for s3 access)

milvus:
  externalS3:
    enabled: true
    bucketName: xxx-milvus
    host: s3.eu-west-1.amazonaws.com
    useIAM: true
  minio:
    enabled: false

Because of this error the indexnode keeps crashing. I wonder if this is related to: #25725 - they mention S3 also but they have a different stack trace.

Expected Behavior

when I index my data it should work

Steps To Reproduce

Use the 4.0.25 Helm chart and try to store data to S3.

Milvus Log

No response

Anything else?

No response

[yanliang567]

/assign @jiaoew1991
/unassign

[zhagnlu]

Could you provide all logs ?

[zhagnlu]

Could you check S3 is using virtual hosting bucket or path bucket mode ? For now, We just support path bucket mode, next release will support virtual hosting bucket mode

[jiaoew1991]

/assign @zhagnlu
/unassign

[mkotsalainen]

Sorry for late reply!

Could you check S3 is using virtual hosting bucket or path bucket mode ? For now, We just support path bucket mode, next release will support virtual hosting bucket mode

It looks like it is virtual hosted mode. This is the url I get when I copy url of a file https://xxx-milvus.s3.eu-west-1.amazonaws.com/file.txt Can I change this setting on the bucket somehow? I've just created this bucket with Terraform and created IRSA role so pod service account can access the bucket.

I've included complete logs of the indexnode.
logs.txt

What I find strange is that data nodes don't seem to have this problem?

[zhagnlu]

Sorry for late reply!

Could you check S3 is using virtual hosting bucket or path bucket mode ? For now, We just support path bucket mode, next release will support virtual hosting bucket mode

It looks like it is virtual hosted mode. This is the url I get when I copy url of a file https://xxx-milvus.s3.eu-west-1.amazonaws.com/file.txt Can I change this setting on the bucket somehow? I've just created this bucket with Terraform and created IRSA role so pod service account can access the bucket.

I've included complete logs of the indexnode. logs.txt

What I find strange is that data nodes don't seem to have this problem?

Sorry for late reply too, as you say, need virtual host mode to support. 2.2.12 have supported it, you can try it. To use this mode, need to change milvus.yaml configs. as show below :

[mkotsalainen]

Aha but how do I add the useVirtualHost parameter via helm chart - it doesn't seem to be exposed as a parameter?

https://github.com/milvus-io/milvus-helm/blob/bd9a12da01363b69004356091d81f122a498b77e/charts/milvus/values.yaml#L429

Something else that I don't understand - should I configure minio if I have enabled externalS3? Aren't they mutuall exclusive?

https://github.com/milvus-io/milvus-helm/blob/bd9a12da01363b69004356091d81f122a498b77e/charts/milvus/values.yaml#L859

[mkotsalainen]

To anwer my own question last question:
https://milvus.io/docs/configure-helm.md#Modify-the-configuration-file

I can't get virtual host mode to work however. I'm running v2.2.13

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.