Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(aws-s3-assets): Object keys don't preserve the full extension name #12699

Closed
zhangfand opened this issue Jan 25, 2021 · 5 comments · Fixed by #20874
Closed

(aws-s3-assets): Object keys don't preserve the full extension name #12699

zhangfand opened this issue Jan 25, 2021 · 5 comments · Fixed by #20874
Labels
@aws-cdk/aws-s3-assets bug This issue is a bug. effort/small Small work item – less than a day of effort in-progress This issue is being actively worked on. p1

Comments

@zhangfand
Copy link

zhangfand commented Jan 25, 2021

I'm using aws-s3-assets to upload data artifacts I used for a Sagemaker model. According to the documentation of Sagemaker, the S3 object needs to end with tar.gz extension. However, the uploaded asset is renamed to <random-id>.gz.

Reproduction Steps

import * as cdk from "@aws-cdk/core";
import * as s3assets from "@aws-cdk/aws-s3-assets";

export class ModelStack extends cdk.Stack {
    constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);
        const modelArtifactAsset = new s3assets.Asset(this, 'ModelArtifact', {
            path: "artifact.tar.gz"
        });
        new cdk.CfnOutput(this, "modelArtifactS3Url", {
            description: "S3 url to the model artifact.",
            exportName: "modelArtifactS3Url",
            value: modelArtifactAsset.s3ObjectUrl,
        });
    }
}

What did you expect to happen?

I expect the s3 object key is ended with tar.gz

What actually happened?

The generated object is named as <id>.gz. Here is an example output from one run:
s3://<bucket-name>/assets/f43ce33c42b10b3e1fe78d87a46cd0dff8611c37637edcf1857a36c33f302ee5.gz

Environment

  • CDK CLI Version :: 1.85.0
  • Framework Version:: 1.86.0
  • Node.js Version:: v15.2.0
  • OS : Mac OS X
  • Language (Version): Typescript 3.9.7

Other

A possible fix is to add an optional parameter for customize the extension name of the asset.


This is 🐛 Bug Report

@zhangfand zhangfand added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jan 25, 2021
@iliapolo
Copy link
Contributor

iliapolo commented Feb 2, 2021

Happens because we use path.extname when calculating the target asset name:

const stagedPath = this.stagingDisabled
? this.sourcePath
: path.resolve(this.assetOutdir, renderAssetFilename(assetHash, path.extname(this.sourcePath)));
.

Which means this would happen for any multi dot extension. Maybe worth applying some heuristics here or possibly exposing some configuration for the user to specify the extension of an asset.

@iliapolo iliapolo added effort/small Small work item – less than a day of effort p1 and removed needs-triage This issue or PR still needs to be triaged. labels Feb 2, 2021
@javiercani
Copy link

Are there any news about this issue?

@github-actions
Copy link

This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

@github-actions github-actions bot added the closing-soon This issue will automatically close in 4 days unless further comments are made. label Jun 28, 2022
@peterwoodworth
Copy link
Contributor

in progress

@peterwoodworth peterwoodworth added in-progress This issue is being actively worked on. and removed closing-soon This issue will automatically close in 4 days unless further comments are made. labels Jun 28, 2022
@mergify mergify bot closed this as completed in #20874 Jul 14, 2022
mergify bot pushed a commit that referenced this issue Jul 14, 2022
…r.gz (#20874)

using aws-s3-assets to upload data artifacts of extension `tar.gz` returns an uploaded asset renamed to `<random Id>.gz`. 

This PR proposes that the AssetStaging Object should be able to check if the uploaded artifact is a `tar.gz` or any other archive tar file with a compression extension and return the appropriate extension name as stagedPath.

closes #12699

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

guessi pushed a commit to guessi/aws-cdk that referenced this issue Jul 17, 2022
…r.gz (aws#20874)

using aws-s3-assets to upload data artifacts of extension `tar.gz` returns an uploaded asset renamed to `<random Id>.gz`. 

This PR proposes that the AssetStaging Object should be able to check if the uploaded artifact is a `tar.gz` or any other archive tar file with a compression extension and return the appropriate extension name as stagedPath.

closes aws#12699

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-s3-assets bug This issue is a bug. effort/small Small work item – less than a day of effort in-progress This issue is being actively worked on. p1
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants