Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(core): S3 copy object with foreign characters #17151

Merged
merged 6 commits into from
Dec 10, 2024
Merged

Conversation

HjorturJ
Copy link
Member

@HjorturJ HjorturJ commented Dec 5, 2024

Encode URI to allow s3 to upload unicode characters

What

Error is thrown when uploading a file with foreign characters, this is a known issue with the copy feature of s3 in the aws sdk v3

Why

So it works

Screenshots / Gifs

Attach Screenshots / Gifs to help reviewers understand the scope of the pull request

Checklist:

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • Formatting passes locally with my changes
  • I have rebased against main before asking for a review

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of special characters in source paths during copy operations in S3.

Copy link
Contributor

coderabbitai bot commented Dec 5, 2024

Walkthrough

The changes involve a modification to the copyObject method in the S3Service class located in libs/nest/aws/src/lib/s3.service.ts. The CopySource property is now encoded using encodeURIComponent(copySource) to ensure proper handling of special characters in the source path before executing the S3 copy operation. This change does not affect other methods or functionalities within the class.

Changes

File Path Change Summary
libs/nest/aws/src/lib/s3.service.ts Modified copyObject method to encode CopySource using encodeURIComponent(copySource).

Possibly related PRs

Suggested labels

automerge, high priority

Suggested reviewers

  • robertaandersen
  • norda-gunni
  • baering

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@HjorturJ HjorturJ added the deploy-feature Deploys features to dev label Dec 5, 2024
@datadog-island-is
Copy link

datadog-island-is bot commented Dec 5, 2024

Datadog Report

All test runs 0d4323b 🔗

34 Total Test Services: 0 Failed, 32 Passed
⬆️ Test Sessions change in coverage: 2 increased, 149 no change

Test Services
This report shows up to 10 services
Service Name Failed Known Flaky New Flaky Passed Skipped Total Time Code Coverage Change Test Service View
air-discount-scheme-web 0 0 0 2 0 8.49s 1 no change Link
api 0 0 0 4 0 3.01s 1 no change Link
api-domains-education 0 0 0 8 0 22.3s 1 no change Link
application-api-files 0 0 0 2 0 4.48s 1 no change Link
application-core 0 0 0 97 0 14.97s 1 no change Link
application-system-api 0 0 0 46 0 2m 22.6s 1 no change Link
application-template-api-modules 0 0 0 118 0 2m 37.73s 1 no change Link
application-templates-accident-notification 0 0 0 148 0 20.9s 1 no change Link
application-templates-criminal-record 0 0 0 2 0 11.94s 1 no change Link
application-templates-driving-license 0 0 0 13 0 16.81s 1 no change Link

Copy link

codecov bot commented Dec 5, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 35.73%. Comparing base (4a6defe) to head (694e7ba).
Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main   #17151   +/-   ##
=======================================
  Coverage   35.73%   35.73%           
=======================================
  Files        6928     6928           
  Lines      147830   147768   -62     
  Branches    42130    42096   -34     
=======================================
- Hits        52827    52810   -17     
+ Misses      95003    94958   -45     
Flag Coverage Δ
air-discount-scheme-backend 48.17% <ø> (ø)
air-discount-scheme-web 0.00% <ø> (ø)
api 3.33% <ø> (ø)
api-domains-air-discount-scheme 37.90% <ø> (ø)
api-domains-assets 26.71% <ø> (ø)
api-domains-auth-admin 48.49% <ø> (ø)
api-domains-communications 39.42% <ø> (ø)
api-domains-criminal-record 47.92% <ø> (ø)
api-domains-driving-license 44.81% <ø> (ø)
api-domains-education 31.22% <ø> (ø)
api-domains-health-insurance 35.33% <ø> (ø)
api-domains-mortgage-certificate 35.16% <ø> (ø)
api-domains-payment-schedule 42.04% <ø> (ø)
application-api-files 61.73% <ø> (ø)
application-core 76.01% <ø> (ø)
application-system-api 38.76% <ø> (+0.01%) ⬆️
application-template-api-modules 27.79% <ø> (-0.02%) ⬇️
application-templates-accident-notification 28.93% <ø> (ø)
application-templates-car-recycling 3.12% <ø> (ø)
application-templates-criminal-record 25.78% <ø> (ø)
application-templates-driving-license 18.12% <ø> (ø)
application-templates-estate 13.79% <ø> (ø)
application-templates-example-payment 24.71% <ø> (ø)
application-templates-financial-aid 14.42% <ø> (ø)
application-templates-general-petition 23.13% <ø> (ø)
application-templates-inheritance-report 6.54% <ø> (ø)
application-templates-marriage-conditions 15.11% <ø> (ø)
application-templates-mortgage-certificate 43.56% <ø> (ø)
application-templates-parental-leave 29.95% <ø> (ø)
application-types 6.57% <ø> (ø)
application-ui-components 1.24% <ø> (ø)
application-ui-shell 22.43% <ø> (ø)
clients-charge-fjs-v2 28.35% <ø> (ø)
clients-driving-license 40.79% <ø> (ø)
clients-driving-license-book 43.83% <ø> (ø)
clients-financial-statements-inao 49.59% <ø> (ø)
clients-license-client 1.26% <ø> (ø)
clients-regulations 42.87% <ø> (ø)
clients-rsk-company-registry 31.04% <ø> (ø)
clients-rsk-personal-tax-return 38.32% <ø> (ø)
clients-smartsolutions 12.77% <ø> (ø)
clients-syslumenn 49.41% <ø> (ø)
clients-zendesk 50.46% <ø> (ø)
cms 0.40% <ø> (ø)
cms-translations 38.75% <ø> (ø)
dokobit-signing 62.10% <ø> (ø)
email-service 60.07% <ø> (ø)
feature-flags 90.76% <ø> (ø)
file-storage 45.86% <ø> (ø)
financial-aid-backend 51.40% <ø> (+0.01%) ⬆️
judicial-system-api 19.97% <ø> (ø)
judicial-system-audit-trail 68.94% <ø> (ø)
judicial-system-backend 55.74% <ø> (ø)
judicial-system-message 66.35% <ø> (ø)
judicial-system-message-handler 48.06% <ø> (ø)
judicial-system-scheduler 70.73% <ø> (ø)
license-api 42.99% <ø> (+0.09%) ⬆️
nest-aws 51.93% <ø> (ø)
nest-config 76.38% <ø> (ø)
nest-feature-flags 51.00% <ø> (ø)
nova-sms 61.42% <ø> (ø)
portals-admin-regulations-admin 1.80% <ø> (ø)
portals-core 19.76% <ø> (ø)
services-auth-admin-api 52.55% <ø> (ø)
services-auth-delegation-api 58.43% <ø> (-0.02%) ⬇️
services-auth-ids-api 52.54% <ø> (ø)
services-auth-public-api 49.40% <ø> (ø)
services-sessions 65.35% <ø> (ø)
services-university-gateway 49.51% <ø> (+0.11%) ⬆️
services-user-notification 46.58% <ø> (ø)
services-user-profile 56.98% <ø> (+0.06%) ⬆️
web 2.43% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
libs/nest/aws/src/lib/s3.service.ts 61.70% <ø> (ø)

... and 13 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a6defe...694e7ba. Read the comment docs.

@HjorturJ HjorturJ marked this pull request as ready for review December 9, 2024 13:26
@HjorturJ HjorturJ requested a review from a team as a code owner December 9, 2024 13:26
@HjorturJ HjorturJ added the automerge Merge this PR as soon as all checks pass label Dec 9, 2024
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (2)
libs/nest/aws/src/lib/s3.service.ts (2)

66-66: LGTM! Proper encoding for S3 CopySource

The addition of encodeURIComponent is the correct solution for handling foreign characters in S3 copy operations. This follows AWS best practices for handling special characters in S3 object keys.

Consider adding a code comment explaining why the encoding is necessary, for future maintainers:

-      CopySource: encodeURIComponent(copySource),
+      // URI encode to handle foreign/special characters in S3 object keys
+      CopySource: encodeURIComponent(copySource),

Line range hint 1-275: Implementation follows best practices

The S3Service implementation adheres to the coding guidelines:

  • ✓ Properly typed with TypeScript
  • ✓ Reusable across different NextJS apps
  • ✓ Follows effective tree-shaking practices with named imports

Consider extracting the S3 URI encoding logic into a separate utility function if similar encoding needs are identified in other operations. This would promote code reuse and consistency across the service.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between e87eb22 and 5a1ea26.

📒 Files selected for processing (1)
  • libs/nest/aws/src/lib/s3.service.ts (1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
libs/nest/aws/src/lib/s3.service.ts (1)

Pattern libs/**/*: "Confirm that the code adheres to the following:

  • Reusability of components and hooks across different NextJS apps.
  • TypeScript usage for defining props and exporting types.
  • Effective tree-shaking and bundling practices."
🔇 Additional comments (1)
libs/nest/aws/src/lib/s3.service.ts (1)

Line range hint 63-75: Verify similar S3 operations

While the fix for copyObject is correct, let's verify if other S3 operations in the service might need similar encoding for consistency.

✅ Verification successful

No encoding needed for other S3 operations

The encodeURIComponent is only necessary for the CopySource parameter in the CopyObjectCommand as it represents a URL path that may contain special characters. Other S3 operations found in the service (GetObjectCommand, HeadObjectCommand, DeleteObjectCommand) only use Bucket and Key parameters which don't require URL encoding as they are simple identifiers.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for other S3 operations that might need encoding
ast-grep --pattern 'new $_(Command|Upload) ({ 
  $$$
  Key: $_,
  $$$
})'

Length of output: 75


Script:

#!/bin/bash
# Let's try a different pattern to search for S3 operations
rg "new \w+(Command|Upload)" -A 5 libs/nest/aws/src/lib/s3.service.ts

Length of output: 853

@kodiakhq kodiakhq bot merged commit b3e0878 into main Dec 10, 2024
415 checks passed
@kodiakhq kodiakhq bot deleted the fix/s3-copy-metadata branch December 10, 2024 13:13
thorhildurt pushed a commit that referenced this pull request Dec 11, 2024
* Testing overwriting the metadata

* chore: nx format:write update dirty files

* Found a cleaner work around

---------

Co-authored-by: andes-it <builders@andes.is>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automerge Merge this PR as soon as all checks pass deploy-feature Deploys features to dev
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants