Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Entry content-type and aws s3 user metadata #2557

Merged
merged 9 commits into from
Oct 17, 2021
Merged

Conversation

nopcoder
Copy link
Contributor

@nopcoder nopcoder commented Oct 14, 2021

Fix #2296
Fix #2429

  1. ContentType added to entry metadata:
    S3 gateway multipart upload and put object can pass content-type header and it will be returned as part of the entry metadata and get object content-type. Object head request will report the content type as S3 does.
    OpenAPI's link object, stage, and upload accept content-type and store them as part of the object metadata.
    Link physical object API call was updated to return the object stats in order to identify the content-type the server set in case we didn't pass any. We kept the same status code in order not to break backward compatibility.
    lakectl upload was added a '--content-type' flag to support setting value while uploading the file.
  2. AWS S3 user-metadata passed as x-amz-meta-*.
    S3 gateway will pass the headers as the entry metadata fields and return all metadata with x-amz-meta- prefix as headers on getting object request
    No changes were introduced to our OpenAPI - didn't add new functionality to get/set the object metadata in this PR.
  3. Graveler identity will consider object content-type as part of the object identity.

Future work will include testing this feature.

@nopcoder nopcoder self-assigned this Oct 14, 2021
@nopcoder nopcoder requested a review from arielshaqed October 14, 2021 10:50
@nopcoder nopcoder marked this pull request as ready for review October 14, 2021 10:51
Copy link
Contributor

@arielshaqed arielshaqed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat, thanks!

This is really good stuff; the only required changes are to the DDL update filenames, which AFAICT need to say "35" not "34" :-/

api/swagger.yml Show resolved Hide resolved
cmd/lakectl/cmd/fs.go Outdated Show resolved Hide resolved
cmd/lakectl/cmd/fs.go Outdated Show resolved Hide resolved
docs/reference/commands.md Show resolved Hide resolved
pkg/api/controller.go Outdated Show resolved Hide resolved
pkg/ddl/000034_gateway_multipart_content_type.down.sql Outdated Show resolved Hide resolved
pkg/ddl/000034_gateway_multipart_content_type.up.sql Outdated Show resolved Hide resolved
pkg/gateway/multiparts/tracker.go Show resolved Hide resolved
pkg/gateway/operations/headobject.go Show resolved Hide resolved
pkg/gateway/operations/operation_utils.go Show resolved Hide resolved
@nopcoder
Copy link
Contributor Author

Neat, thanks!

This is really good stuff; the only required changes are to the DDL update filenames, which AFAICT need to say "35" not "34" :-/

was late to the party, your PR took it ;)

nopcoder and others added 4 commits October 15, 2021 11:28
Co-authored-by: arielshaqed <ariels@treeverse.io>
Co-authored-by: arielshaqed <ariels@treeverse.io>
@nopcoder nopcoder requested a review from arielshaqed October 15, 2021 15:09
Copy link
Contributor

@arielshaqed arielshaqed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks!

@nopcoder nopcoder merged commit b4e8654 into master Oct 17, 2021
@nopcoder nopcoder deleted the feature/s3-meta branch October 17, 2021 06:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Trino AVRO format access via S3 gateway Support Content-Type
2 participants