fix(aws-lambda,netlify): base64 encode binary responses #1274

harlan-zw · 2023-05-31T15:28:18Z

🔗 Linked issue

See https://nuxt-og-image-playground-netlify.netlify.app/api/resvg.png

Code

import { defineEventHandler, setHeader } from 'h3'
import { Resvg } from '@resvg/resvg-js'

export default defineEventHandler(async (e) => {
    const resvgJS = new Resvg('<svg xmlns="http://www.w3.org/2000/svg" version="1.1" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:svgjs="http://svgjs.dev/svgjs" viewBox="0 0 700 700" width="700" height="700"><defs><linearGradient gradientTransform="rotate(150, 0.5, 0.5)" x1="50%" y1="0%" x2="50%" y2="100%" id="ffflux-gradient"><stop stop-color="hsl(315, 100%, 72%)" stop-opacity="1" offset="0%"></stop><stop stop-color="hsl(227, 100%, 50%)" stop-opacity="1" offset="100%"></stop></linearGradient><filter id="ffflux-filter" x="-20%" y="-20%" width="140%" height="140%" filterUnits="objectBoundingBox" primitiveUnits="userSpaceOnUse" color-interpolation-filters="sRGB">\n' +
        '  <feTurbulence type="fractalNoise" baseFrequency="0.005 0.003" numOctaves="2" seed="2" stitchTiles="stitch" x="0%" y="0%" width="100%" height="100%" result="turbulence"></feTurbulence>\n' +
        '  <feGaussianBlur stdDeviation="20 0" x="0%" y="0%" width="100%" height="100%" in="turbulence" edgeMode="duplicate" result="blur"></feGaussianBlur>\n' +
        '  <feBlend mode="color-dodge" x="0%" y="0%" width="100%" height="100%" in="SourceGraphic" in2="blur" result="blend"></feBlend>\n' +
        '  \n' +
        '</filter></defs><rect width="700" height="700" fill="url(#ffflux-gradient)" filter="url(#ffflux-filter)"></rect></svg>', {
        logLevel: 'trace',
        fitTo: {
            mode: 'width',
            value: 1200,
        },
    })
    setHeader(e, 'Content-Type', 'image/png')
    return resvgJS.render().asPng()
})

❓ Type of change

📖 Documentation (updates to the documentation or readme)
🐞 Bug fix (a non-breaking change that fixes an issue)
👌 Enhancement (improving an existing functionality like performance)
✨ New feature (a non-breaking change that adds functionality)
⚠️ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description

Netlify uses AWS Lambda. AWS Lambda requires sending buffers as strings (they are re-encoded via a proxy afaik).

Nitro currently converts all buffers to default toString(). This breaks images, as they need to be sent with base64 encoding (see https://aws.amazon.com/blogs/compute/handling-binary-data-using-amazon-api-gateway-http-apis/), with a special flag isBase64Encoded.

The fix is to check if we're working with a buffer and the response is an image, then use base 64 encoding. This may affect other response types though.

📝 Checklist

I have linked an issue or discussion.
I have updated the documentation accordingly.

See nitrojs/nitro#1274

codecov · 2023-05-31T15:43:41Z

Codecov Report

Merging #1274 (a5547c9) into main (421d625) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1274   +/-   ##
=======================================
  Coverage   76.21%   76.22%           
=======================================
  Files          73       73           
  Lines        7433     7435    +2     
  Branches      728      728           
=======================================
+ Hits         5665     5667    +2     
  Misses       1767     1767           
  Partials        1        1

see 1 file with indirect coverage changes

src/runtime/entries/netlify-lambda.ts

harlan-zw · 2023-05-31T15:47:22Z

src/runtime/entries/netlify-lambda.ts

@@ -30,9 +31,20 @@ export async function lambda(
    body: event.body, // TODO: handle event.isBase64Encoded
  });

+  const headers = normalizeOutgoingHeaders(r.headers);
+  // image buffers must be base64 encoded
+  if (Buffer.isBuffer(r.body) && headers["content-type"].startsWith("image/")) {


this header check is hacky, looking for a better alternative

maybe it could just be omitted?

If we trust mime.getType to return the correct type, this should be fine for images, as all images types should start with image

However we should do this for all Binary media for lambda based presets.

As always with AWS, the list of supported media types isn't documented anywhere, altough, chatGPT gives me this:

image/png image/jpeg image/gif application/octet-stream application/pdf application/x-zip-compressed audio/mpeg audio/mp4 video/mp4 application/zip audio/webm video/webm

Which sorts of make sense... But I think a cleaner way to handle all possible use cases without relying on the header would be to use a package such as https://www.npmjs.com/package/isbinaryfile, because I'm not sure Buffer works in all scenarios.

We might also need to rely on the header anyways, as what AWS does with Binary files which aren't media (whatever they mean) is currently a mistery.

This reverts commit 1db2102.

pi0 · 2023-07-19T09:16:12Z

Indeed it should be also generic to support any blob/binrary response regardless of mime type.

…etlify-images

…lify-images # Conflicts: # src/runtime/utils.lambda.ts

Hebilicious

LGTM

src/runtime/utils.lambda.ts

…lify-images # Conflicts: # src/runtime/utils.lambda.ts

fix(netlify): support images responses

06a24c7

harlan-zw requested a review from pi0 May 31, 2023 15:28

harlan-zw added a commit to nuxt-modules/og-image that referenced this pull request May 31, 2023

fix: netlify images not working

314043d

See nitrojs/nitro#1274

chore: prefer native

d62e95f

vercel bot deployed to Preview May 31, 2023 15:37 View deployment

harlan-zw mentioned this pull request May 31, 2023

fix: netlify images not working nuxt-modules/og-image#53

Merged

harlan-zw commented May 31, 2023

View reviewed changes

src/runtime/entries/netlify-lambda.ts Outdated Show resolved Hide resolved

harlan-zw commented May 31, 2023

View reviewed changes

harlan-zw changed the title ~~fix(netlify): support images responses~~ fix(netlify): support image responses May 31, 2023

fix: aws-lambda base

b299394

harlan-zw changed the title ~~fix(netlify): support image responses~~ fix(aws-lambda,netlify): support image responses May 31, 2023

[autofix.ci] apply automated fixes

60630ac

vercel bot deployed to Preview May 31, 2023 16:24 View deployment

chore: remove Buffer import

1db2102

vercel bot deployed to Preview June 2, 2023 09:59 View deployment

harlan-zw added 2 commits June 2, 2023 18:00

Revert "chore: remove Buffer import"

0f35897

This reverts commit 1db2102.

chore: remove Buffer import #2

a126411

vercel bot deployed to Preview June 2, 2023 10:01 View deployment

vercel bot deployed to Preview June 2, 2023 10:08 View deployment

Hebilicious self-assigned this Jun 30, 2023

Hebilicious self-requested a review June 30, 2023 23:12

Hebilicious added preset:netlify preset:aws-lambda enhancement New feature or request labels Jun 30, 2023 — with Volta.net

Hebilicious mentioned this pull request Jun 30, 2023

AWS Serverless-Integration corrupts assets #852

Closed

harlan-zw and others added 4 commits July 2, 2023 03:05

fix(netlify): support images responses

e88681a

chore: prefer native

d70b78b

fix: aws-lambda base

62854ed

[autofix.ci] apply automated fixes

dabe204

harlan-zw and others added 4 commits July 20, 2023 11:41

Merge branch 'fix/netlify-images' of github.com:unjs/nitro into fix/n…

2f5fef3

…etlify-images

Merge branch 'main' of github.com:unjs/nitro into fix/netlify-images

73e4697

fix: more generic lambda body normalisation

11ddd60

[autofix.ci] apply automated fixes

e783d67

vercel bot deployed to Preview July 20, 2023 09:01 View deployment

harlan-zw added 2 commits July 20, 2023 13:04

fix: avoid converting json and tests

5d7bd79

Merge remote-tracking branch 'origin/fix/netlify-images' into fix/net…

cb0de90

…lify-images # Conflicts: # src/runtime/utils.lambda.ts

Hebilicious reviewed Jul 20, 2023

View reviewed changes

vercel bot deployed to Preview July 20, 2023 10:08 View deployment

pi0 changed the title ~~fix(aws-lambda,netlify): support image responses~~ fix(aws-lambda,netlify): base64 encode binary responses Jul 20, 2023

pi0 reviewed Jul 20, 2023

View reviewed changes

src/runtime/utils.lambda.ts Outdated Show resolved Hide resolved

harlan-zw and others added 2 commits July 20, 2023 13:37

fix: conditionally use isUtf8

a9152fe

[autofix.ci] apply automated fixes

ffb4a86

vercel bot deployed to Preview July 20, 2023 10:42 View deployment

harlan-zw and others added 4 commits July 20, 2023 13:45

fix: add hacky fallback

0469c29

Merge remote-tracking branch 'origin/fix/netlify-images' into fix/net…

f902b4b

…lify-images # Conflicts: # src/runtime/utils.lambda.ts

[autofix.ci] apply automated fixes

26bbd7a

test: simplify and run against netlify as well

7f82680

vercel bot deployed to Preview July 20, 2023 10:52 View deployment

add check for text content types

2902edf

vercel bot deployed to Preview July 21, 2023 09:33 View deployment

pi0 added 2 commits July 21, 2023 11:33

improve util

8a01526

remove buffer import

a5547c9

vercel bot deployed to Preview July 21, 2023 09:35 View deployment

pi0 merged commit 7b8475c into main Jul 21, 2023
6 checks passed

pi0 deleted the fix/netlify-images branch July 21, 2023 10:59

pi0 mentioned this pull request Aug 4, 2023

[draft] v2.6.0 #1526

Closed

NtchPlayer mentioned this pull request Aug 26, 2023

Error on image compression in public directory with AWS-Preset nuxt/nuxt#22832

Closed

Azcray mentioned this pull request Aug 26, 2023

fix(aws-lambda, netlify): add isBase64Encoded response field #1645

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(aws-lambda,netlify): base64 encode binary responses #1274

fix(aws-lambda,netlify): base64 encode binary responses #1274

harlan-zw commented May 31, 2023 •

edited

Loading

codecov bot commented May 31, 2023 •

edited

Loading

harlan-zw May 31, 2023 •

edited

Loading

Hebilicious Jul 1, 2023 •

edited

Loading

pi0 commented Jul 19, 2023

Hebilicious left a comment

fix(aws-lambda,netlify): base64 encode binary responses #1274

fix(aws-lambda,netlify): base64 encode binary responses #1274

Conversation

harlan-zw commented May 31, 2023 • edited Loading

🔗 Linked issue

❓ Type of change

📚 Description

📝 Checklist

codecov bot commented May 31, 2023 • edited Loading

Codecov Report

harlan-zw May 31, 2023 • edited Loading

Choose a reason for hiding this comment

Hebilicious Jul 1, 2023 • edited Loading

Choose a reason for hiding this comment

pi0 commented Jul 19, 2023

Hebilicious left a comment

Choose a reason for hiding this comment

harlan-zw commented May 31, 2023 •

edited

Loading

codecov bot commented May 31, 2023 •

edited

Loading

harlan-zw May 31, 2023 •

edited

Loading

Hebilicious Jul 1, 2023 •

edited

Loading