Fetching logs does not work when using Quobyte s3 object storage

[bashofmann]

What steps did you take and what happened:
When creating a backup with using OpenStack s3 object storage, backup and restore work just fine, but fetching the logs fails:

$ ark backup logs backup-test
An error occurred: request failed: <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The resource you requested does not exist</Message><Resource>/metakube-cluster-backup-fhgbvx65xg-ark/backup-test/backup-test-logs.gz</Resource><RequestId>2925491272881701579</RequestId></Error>

$ ark restore logs backup-test-20180830163119
An error occurred: request failed: <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The resource you requested does not exist</Message><Resource>/metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz</Resource><RequestId>1135366011835801288</RequestId></Error>

The files exist though and also have the correct valid content:

$ s3cmd la --recursive
2018-08-30 14:48         0   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/
2018-08-30 14:13      2446   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/backup-test-logs.gz
2018-08-30 14:13      5907   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/backup-test.tar.gz
2018-08-30 14:31       647   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz
2018-08-30 14:31        82   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-results.gz

$ s3cmd get s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz
download: 's3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz' -> './restore-backup-test-20180830163119-logs.gz'  [1 of 1]
 647 of 647   100% in    0s   110.17 kB/s  done

$ gunzip restore-backup-test-20180830163119-logs.gz

$ cat restore-backup-test-20180830163119-logs
time="2018-08-30T14:31:21Z" level=info msg="Not including resource" groupResource=events logSource="pkg/restore/restore.go:124"
...

What did you expect to happen:

Logs are displayed correctly

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/ark -n heptio-ark
  No errors.
• ark backup describe <backupname> or kubectl get backup/<backupname> -n heptio-ark -o yaml
  No errors.
• ark backup logs <backupname>
  Doesn't work, see above. No errors when manually downloading and viewing them.
• ark restore describe <restorename> or kubectl get restore/<restorename> -n heptio-ark -o yaml
  No errors.
• ark restore logs <restorename>
  Doesn't work, see above. No errors when manually downloading and viewing them.

Environment:

• Ark version (use ark version): 0.9.3
• Kubernetes version (use kubectl version): 1.11.2 on server and client
• Cloud provider or hardware configuration: OpenStack
• OS (e.g. from /etc/os-release): Ubuntu Xenial

[ncdc]

I wonder if the leading / in e.g. /metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz is causing the problem.

@bashofmann can you run aws s3 presign s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz and share the returned url?

[bashofmann]

The result is:

$ aws --endpoint-url https://s3.cbk.cloud.syseleven.net s3 presign s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz
https://s3.cbk.cloud.syseleven.net/metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz?AWSAccessKeyId=xxxx&Signature=xxxx%3D&Expires=1535647777

With the correct AWSAccessKeyId and Signature the URL also works and the file can be downloaded.

[bashofmann]

For reference our ark config is:

apiVersion: ark.heptio.com/v1
backupStorageProvider:
  bucket: metakube-cluster-backup-fhgbvx65xg-ark
  config:
    region: cbk
    s3ForcePathStyle: "true"
    s3Url: https://s3.cbk.cloud.syseleven.net
  name: aws
  resticLocation: metakube-cluster-backup-fhgbvx65xg-restic
backupSyncPeriod: 1m
gcSyncPeriod: 1m
kind: Config

[ncdc]

Hmm, I'm not seeing anything in the code that would put a leading / at a quick glance. What OpenStack version are you running?

[bashofmann]

I debugged this and the path of the download URL the ark server returns to the CLI is correct:

https://s3.cbk.cloud.syseleven.net/metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz

But the query parameters are completely different to what aws s3 presign returns:

?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=xxx%2F20180830%2Fcbk%2Fs3%2Faws4_request&X-Amz-Date=20180830T162545Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=yyy

If I replace these query parameters with AWSAccessKeyId, Signature and Expires, the URL works.

[bashofmann]

The same problem is there when trying to download a backup with the cli, by the way.

[ncdc]

What OpenStack version are you running?

[bashofmann]

We are running OpenStack Mitaka and use Quobyte 2.5 for S3 Storage

[ncdc]

So s3 is directly provided by Quobyte? OpenStack isn't really relevant/involved?

[bashofmann]

Sorry for the late reply, I checked with our OpenStack team and yes S3 is directly provided by Quobyte. It seems that Quobyte only supports the V2 Signer API, but ark uses V4.
Would it be possible to make the version configurable? For example like this:

func (o *objectStore) CreateSignedURL(bucket, key string, ttl time.Duration) (string, error) {
	req, _ := o.s3.GetObjectRequest(&s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	})

	if o.signatureVersion == "2" {
		req.Handlers.Sign.Remove(v4.SignRequestHandler)
		req.Handlers.Sign.PushBackNamed(v2.SignRequestHandler)
	}

	return req.Presign(ttl)
}

[ncdc]

@bashofmann we can look into this. I checked the aws sdk, and the v2 SignRequestHandler is in a private package. We can import it and use it, but Amazon's guidance is that anything in there is for the SDK's internal use and is subject to breaking changes.

We would need to define a new key for our aws ObjectStore's config to specify signature version. Would you be interested in trying to put together a PR for this?

[bashofmann]

I looked into it a bit further and according to the AWS Python CLI, S3 is either using V4 of the Signer or V1, which seems to not even implemented in the go library.
I could work on porting the V1 algorithm to go and make this configurable, but would this be even in the scope of being included in ark directly or better handled in a separate object_storage plugin?

Relevant Python implementation: https://github.com/oNestLab/botocore/blob/d6c1be296e8cfe0706cb0c8bbcad9c095d0f4d09/botocore/auth.py#L860-L862

[ncdc]

@bashofmann the aws go sdk does have a v2 signer, if that's what you need (with the caveat from the repo that it's a private api and may change/go away/etc).

You also could do a separate quobyte plugin if you wanted to. We have #193 as a possible way to make it easier to have common AWS behavior with differences here and there (e.g. quobyte signatures), but we haven't implemented it yet.

[bashofmann]

The v2 signer unfortunately does not work, I already tried it. And from the python code there are some differences between v1 and v2.

[ncdc]

Ah, that's unfortunate. I'll talk with the team about whether or not we want to own a v1 signing algorithm in the Ark code base and get back to you.

[bashofmann]

FYI: I with the V1 algorithm downloading logs and backups works correctly with Quobyte S3

[ncdc]

cc @skriss

[bashofmann]

@ncdc @skriss Did you already have time to discuss if we should contribute this as a PR to ark or write a plugin for this?

[ncdc]

We haven't had a chance to chat, but I'd be ok with a PR in the short term, and #193 long term. WDYT @skriss?