ls against protected buckets does not work

[jhiemstrawisc]

When the origin is configured to point to private buckets (those with a secret/access key), the ls functionality triggered by issuing a PROPFIND request to the origin fails. I suspect the source of this bug is being touched and potentially fixed in #44 where the S3 listing URL is created, but we should confirm. My hunch is that the 403 being fixed by that PR is the same reason ls is not working.

[jhiemstrawisc]

Making some noise for this issue because it appears the fix for issue #44 did not fix this particular 403 error.

[rw2]

Testing with the below methodology results in success.

[root@dc2be604757a testdata]# cat prop_query
<d:propfind xmlns:d='DAV:'>
<d:prop>
<d:displayname/>
<d:resourcetype/>
<d:getcontentlength/>
<d:getcontenttype/>
<d:getetag/>
<d:getlastmodified/>
</d:prop>

curl -k -H 'Depth: 1' -X PROPFIND http://localhost:8080/my-magic-path/10mb.dat -d prop_query

<D:multistatus xmlns:D="DAV:" xmlns:ns1="http://apache.org/dav/props/" xmlns:ns0="DAV:">
<D:response xmlns:lp1="DAV:" xmlns:lp2="http://apache.org/dav/props/" xmlns:lp3="LCGDM:">
<D:href>/my-magic-path/10mb.dat</D:href>
<D:propstat>
<D:prop>
lp1:getcontentlength10485760</lp1:getcontentlength>
lp1:getlastmodifiedThu, 01 Jan 1970 00:00:00 GMT</lp1:getlastmodified>
lp1:iscollection0</lp1:iscollection>
lp1:executableF</lp1:executable>
</D:prop>
<D:status>HTTP/1.1 200 OK</D:status>
</D:propstat>
</D:response>
</D:multistatus>

[jhiemstrawisc]

Here's the pertinent config I'm using to generate the error:

s3.begin
s3.path_name         /my-prefix
s3.bucket_name     foo
s3.access_key_file /workspaces/pelican_xrootd_s3/tests/access.key
s3.secret_key_file  /workspaces/pelican_xrootd_s3/tests/secret.key
s3.service_name    s3
s3.region                us-east-1
s3.url_style            path
s3.service_url        https://s3dev.chtc.wisc.edu
s3.end

This should point my namespace /my-prefix at my protected bucket called foo, which has two test objects in it. First, I confirm I can download one of these test objects:

# curl -v http://localhost:8443/my-prefix/test.txt
> GET /my-prefix/test.txt HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/7.76.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Server: XrootD/v5.7.1
< Content-Length: 14
< 
Hello, world!

All looks good there.

Next, I try to send a PROPFIND to the configured prefix, which should map to the root of the bucket and list the two test objects:

# curl -v -X PROPFIND http://localhost:8443/my-prefix/ -H "Depth: 1"

> PROPFIND /my-prefix/ HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/7.76.1
> Accept: */*
> Depth: 1
> 
< HTTP/1.1 403 Forbidden
< Connection: Close
< Server: XrootD/v5.7.1
< Content-Length: 54
< 
Unable to locate /my-prefix/; operation not permitted

I believe the following is all of the relevant logging to come out of xrootd:

241104 21:01:38 28522 http_Protocol:  rc:31 got hdr line: PROPFIND /my-prefix/ HTTP/1.1

241104 21:01:38 28522 http_Protocol:  Parsing first line: PROPFIND /my-prefix/ HTTP/1.1

241104 21:01:38 28522 http_Protocol:  rc:22 got hdr line: Host: localhost:8443

241104 21:01:38 28522 http_Protocol:  rc:25 got hdr line: User-Agent: curl/7.76.1

241104 21:01:38 28522 http_Protocol:  rc:13 got hdr line: Accept: */*

241104 21:01:38 28522 http_Protocol:  rc:10 got hdr line: Depth: 1

241104 21:01:38 28522 http_Protocol:  rc:2 got hdr line: 

241104 21:01:38 28522 http_Protocol:  rc:2 detected header end.
241104 21:01:38 28522 XrootdBridge: unknown.1:41@localhost login as nobody
241104 21:01:38 28522 unknown.1:41@localhost http_Protocol:  Process. lp:0xffff80008fc8 reqstate: 0
241104 21:01:38 28522 unknown.1:41@localhost http_Protocol: Setting monitor info curl/7.76.1
241104 21:01:38 28522 http_Req:  XrdHttpReq::Data! final=False
241104 21:01:38 28522 unknown.1:41@localhost http_Req: PostProcessHTTPReq req: 8 reqstate: 0 final_:True
241104 21:01:38 28522 unknown.1:41@localhost http_Protocol:  Process. lp:(nil) reqstate: 0
241104 21:01:38 28522 unknown.1:41@localhost http_Protocol: Process is exiting rc:0
241104 21:01:38 28522 unknown.1:41@localhost ofs_stat:  fn=/my-prefix/
241104 21:01:38 28522 s3_Stat: Stat'ing path /my-prefix/
241104 21:01:38 28527 s3_SetupHandle: Sending HTTP request https://s3dev.chtc.wisc.edu/foo?delimiter=%2F&list-type=2&max-keys=1&prefix=
241104 21:01:38 28527 s3_CurlWorker: Curl worker thread 28518 is running 1operations
== Info:   Trying 128.104.100.181:443...
== Info: Connected to s3dev.chtc.wisc.edu (128.104.100.181) port 443 (#0)
== Info: ALPN, offering h2
== Info: ALPN, offering http/1.1
== Info:  CAfile: /etc/pki/tls/certs/ca-bundle.crt
== Info: TLSv1.0 (OUT), TLS header, Certificate Status (22):
== Info: TLSv1.3 (OUT), TLS handshake, Client hello (1):
== Info: TLSv1.2 (IN), TLS header, Certificate Status (22):
== Info: TLSv1.3 (IN), TLS handshake, Server hello (2):
== Info: TLSv1.2 (IN), TLS header, Finished (20):
== Info: TLSv1.2 (IN), TLS header, Unknown (23):
== Info: TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
== Info: TLSv1.2 (IN), TLS header, Unknown (23):
== Info: TLSv1.3 (IN), TLS handshake, Certificate (11):
== Info: TLSv1.2 (IN), TLS header, Unknown (23):
== Info: TLSv1.3 (IN), TLS handshake, CERT verify (15):
== Info: TLSv1.2 (IN), TLS header, Unknown (23):
== Info: TLSv1.3 (IN), TLS handshake, Finished (20):
== Info: TLSv1.2 (OUT), TLS header, Finished (20):
== Info: TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
== Info: TLSv1.2 (OUT), TLS header, Unknown (23):
== Info: TLSv1.3 (OUT), TLS handshake, Finished (20):
== Info: SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
== Info: ALPN, server accepted to use http/1.1
== Info: Server certificate:
== Info:  subject: CN=s3dev.chtc.wisc.edu
== Info:  start date: Sep 19 01:53:50 2024 GMT
== Info:  expire date: Dec 18 01:53:49 2024 GMT
== Info:  subjectAltName: host "s3dev.chtc.wisc.edu" matched cert's "s3dev.chtc.wisc.edu"
== Info:  issuer: C=US; O=Let's Encrypt; CN=E5
== Info:  SSL certificate verify ok.
== Info: TLSv1.2 (OUT), TLS header, Unknown (23):
=> Send header, 0000000563 bytes (0x00000233)
GET /foo?delimiter=%2F&list-type=2&max-keys=1&prefix= HTTP/1.1
Host: s3dev.chtc.wisc.edu
User-Agent: xrootd-s3/devel
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=36JC76CW36WWSFQ2SX5R/20241104/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=cc608dbae80848db77366b825a4a336128b23670dadbd272c9225dd6d400ce94
Content-Length: 0
Content-Type: binary/octet-stream
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241104T210138Z


== Info: TLSv1.2 (IN), TLS header, Unknown (23):
== Info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
== Info: TLSv1.2 (IN), TLS header, Unknown (23):
== Info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
== Info: old SSL session ID is stale, removing
== Info: TLSv1.2 (IN), TLS header, Unknown (23):
== Info: Mark bundle as not supporting multiuse
== Info: Connection #0 to host s3dev.chtc.wisc.edu left intact
241104 21:01:38 28522 s3_Stat: Failed to stat path /my-prefix/; response code 403
241104 21:01:38 28522 ofs_stat: unknown.1:41@localhost Unable to locate /my-prefix/; operation not permitted

At first I wondered whether there might be an issue with the empty prefix being constructed in the HTTPs request (241104 21:01:38 28527 s3_SetupHandle: Sending HTTP request https://s3dev.chtc.wisc.edu/foo?delimiter=%2F&list-type=2&max-keys=1&prefix=). However, playing around with this had no effect on the resulting 403.

[jhiemstrawisc]

As an update, Rich and I went over our two approaches together and we found this issue is specifically tied to path-style setups. It appears to work with virtual buckets.

[rw2]

More data while I'm thinking of it. Path style urls are deprecated and will go away someday. Currently, you cannot have a bucket that is listable unless you specifically add the ListObjects grant to public. This definitely isn't the whole problem, but was masking some things, so I'm recording it here.

e.g.

	{
		"Effect": "Allow",
		"Principal": "*",
		"Action": "s3:ListBucket",
		"Resource": "arn:aws:s3:::hubzero-private-rich"
	},

[bbockelm]

@jhiemstrawisc - can you produce a unit test that fails with minio?

[jhiemstrawisc]

I'll look into it in the next week.

[bbockelm]

Did some online debugging in Slack and I've been convinced that the original issue has been fixed. However, Justin's test bucket certainly revealed a separate bug which I've filed as #63.