Support serving images under s3 subdirectories, Fix to make /fit-in/ work; Fix for VipsJpeg: Invalid SOS error plus several other critical fixes. #130

rpong · 2019-07-24T18:15:16Z

This PR includes fixes for the following v4 issues:

S3 subdirectories not working.
/fit-in/ url directive not working.
Exif orientation not honoured/followed.
VipsJpeg: Invalid SOS parameters for sequential JPEG errors. This is observable with images taken with Samsung cameras, causing the api/cloudfront endpoint to output a blank json {} instead of the image.
Wrong resizing if the path/filename contains an 'x' with surrounding digits (eg: /image8x8.jpg)

Sample paths in which this was tested on aside from the unit tests:

/100x100/directory1/directory2/image.jpg
/fit-in/100x100/directory1/directory2/image.jpg
/fit-in/100x100/filters:someFilters/directory1/directory2/image.jpg
/filters:someFilters/directory1/directory2/image.jpg
/100x100/filters:someFilters/directory1/directory2/image_8x8x8x12.jpg

You may start using this PR/Fork with this Exported Function zip: https://github.com/innovationlove/serverless-image-handler/releases/tag/4.0.1-ilove

Just replace the code that your Serverless Image Handler Function is using with the package above. USE AT YOUR OWN RISK.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

beyerz · 2019-07-25T11:30:24Z

I have manually added this solution to our lambda-function and can confirm that it performs as defined.
@rpong You are a life saver!

denys1yermakov · 2019-07-25T21:43:16Z

@rpong, images without resizing still doesn't work, example:
/directory1/directory2/image.jpg
If you can fix it in that pull request will be super helpfull, thanks!

codyhawke · 2019-07-26T21:29:42Z

Been trying to get this going - everything seems to work fine, except it's not resizing the image when in a subdirectory - example url:
https://xxxxxxx.cloudfront.net/400x0/products/img/product01.jpg

…follow exif-specified rotation by default

…my bad

rpong · 2019-07-27T07:28:31Z

@rpong, images without resizing still doesn't work, example:
/directory1/directory2/image.jpg
If you can fix it in that pull request will be super helpfull, thanks!

sorry, my bad. this should be fixed now.

rpong · 2019-07-27T07:31:46Z

Been trying to get this going - everything seems to work fine, except it's not resizing the image when in a subdirectory - example url:
https://xxxxxxx.cloudfront.net/400x0/products/img/product01.jpg

can you attach the actual photo and actual filename? we discovered an anomaly with filenames containing an 'x' and digits around it causing improper resizing, we will be applying the fix for the PR too in a bit, just want to make sure about your use case and perhaps cover it with our fix later today. thanks man..

gendigbadig · 2019-07-27T08:15:25Z

Been trying to get this going - everything seems to work fine, except it's not resizing the image when in a subdirectory - example url:
https://xxxxxxx.cloudfront.net/400x0/products/img/product01.jpg

can you attach the actual photo and actual filename? we discovered an anomaly with filenames containing an 'x' and digits around it causing improper resizing, we will be applying the fix for the PR too in a bit, just want to make sure about your use case and perhaps cover it with our fix later today. thanks man..

in my use case, some of key path contains 'x' characters, and i discovered dimension path checking on thumbor-mapping.js is using only String.prototype.includes method, so i change those line to using regex match

Line 39

           else if (edit.match(/^\d+x\d+$/)) { // change this line
                this.edits.resize = {};
                const dims = edit.split('x');
                this.edits.resize.width = Number(dims[0]);
                this.edits.resize.height = Number(dims[1]);
            }

Please add it to your PR, thank you :)

denys1yermakov · 2019-07-27T08:32:54Z

Been trying to get this going - everything seems to work fine, except it's not resizing the image when in a subdirectory - example url:
https://xxxxxxx.cloudfront.net/400x0/products/img/product01.jpg

can you attach the actual photo and actual filename? we discovered an anomaly with filenames containing an 'x' and digits around it causing improper resizing, we will be applying the fix for the PR too in a bit, just want to make sure about your use case and perhaps cover it with our fix later today. thanks man..

in my use case, some of key path contains 'x' characters, and i discovered dimension path checking on thumbor-mapping.js is using only String.prototype.includes method, so i change those line to using regex match

Line 39
           else if (edit.match(/^\d+x\d+$/)) { // change this line
                this.edits.resize = {};
                const dims = edit.split('x');
                this.edits.resize.width = Number(dims[0]);
                this.edits.resize.height = Number(dims[1]);
            } 
Please add it to your PR, thank you :)

that work, thanks !

gendigbadig · 2019-07-27T08:34:02Z

also, in previous version (thumbor), i have some use case that unfortunately unsupported on current version:

using key path without file extension
Example: https://xxxx.cloudfront.net/64x64/images/svara-radio-logo/download/58e5ecc7cab82dc1700c0043
My workaround is remove path extension checking on parseRequestType method on image-request.js

parseRequestType(event) {
    const path = event["path"];
    // ----
    const matchDefault = new RegExp(
      /^(\/?)([0-9a-zA-Z+\/]{4})*(([0-9a-zA-Z+\/]{2}==)|([0-9a-zA-Z+\/]{3}=))?$/
    );
    const matchThumbor = new RegExp(
      /^(\/?)((fit-in)?|(filters:.+\(.?\))?|(unsafe)?).*$/  // Remove extension on this regex
    ); 
    const matchCustom = new RegExp(/(\/?)(.*)(jpg|png|webp|tiff|jpeg)/);
    const definedEnvironmentVariables =
      process.env.REWRITE_MATCH_PATTERN !== "" &&
      process.env.REWRITE_SUBSTITUTION !== "" &&
      process.env.REWRITE_MATCH_PATTERN !== undefined &&
      process.env.REWRITE_SUBSTITUTION !== undefined;
    // ----
    if (matchDefault.test(path)) {
      // use sharp
      return "Default";
    } else if (matchCustom.test(path) && definedEnvironmentVariables) {
      // use rewrite function then thumbor mappings
      return "Custom";
    } else if (matchThumbor.test(path)) {
      // use thumbor mappings
      return "Thumbor";
    } else {
      throw {
        status: 400,
        code: "RequestTypeError",
        message:
          "The type of request you are making could not be processed. Please ensure that your original image is of a supported file type (jpg, png, tiff, webp) and that your image request is provided in the correct syntax. Refer to the documentation for additional guidance on forming image requests."
      };
    }
  }

using smart path
For now, my workaround is only omit the smart keyword on parseImageKey method at image-request.js

parseImageKey(event, requestType) {
    if (requestType === "Default") {
      // Decode the image request and return the image key
      const decoded = this.decodeRequest(event);
      return decoded.key;
    } else if (requestType === "Thumbor" || requestType === "Custom") {
      // Parse the key from the end of the path
      const pathParts = event["path"].split("/");

      for (let i = 0; i < pathParts.length; i++) {
        if (
          pathParts[i] !== "" &&
          pathParts[i] !== "fit-in" &&
          pathParts[i].match(/^\d+x\d+$/) == null &&
          pathParts[i].match(/^filters:/) == null &&
          pathParts[i].match(/^smart$/) == null  // add this line
        ) {
          const parts = pathParts.slice(i);
          return parts.join("/");
        }
      }
    }

    // Return an error for all other conditions
    throw {
      status: 400,
      code: "ImageEdits::CannotFindImage",
      message:
        "The image you specified could not be found. Please check your request syntax as well as the bucket you specified to ensure it exists."
    };
  }

using filters fill with same usage as thumbor, ex:filters:fill(white,1)
Currently my workaround on this problem is adding some hardcoded line to convert filter value to color object that accepted by sharpjs, ex: 'white,1' to {r: 255, g:255, b: 255, alpha: 0}
I added it on method mapFilter() at thumbor-mapping.js

maybe you have more clean workaround for this problem
thank you

…h digits surrounding it eg: 8x8

rpong · 2019-07-27T15:48:48Z

Been trying to get this going - everything seems to work fine, except it's not resizing the image when in a subdirectory - example url:
https://xxxxxxx.cloudfront.net/400x0/products/img/product01.jpg

can you attach the actual photo and actual filename? we discovered an anomaly with filenames containing an 'x' and digits around it causing improper resizing, we will be applying the fix for the PR too in a bit, just want to make sure about your use case and perhaps cover it with our fix later today. thanks man..

in my use case, some of key path contains 'x' characters, and i discovered dimension path checking on thumbor-mapping.js is using only String.prototype.includes method, so i change those line to using regex match

Line 39
           else if (edit.match(/^\d+x\d+$/)) { // change this line
                this.edits.resize = {};
                const dims = edit.split('x');
                this.edits.resize.width = Number(dims[0]);
                this.edits.resize.height = Number(dims[1]);
            } 
Please add it to your PR, thank you :)

Nice! thank you for this, I added this to the PR but moved it outside the foreach loop so it is processed/looked up only once. Just updated the PR as well so the tests pass and several other critical things that required us to fix before we launched our beta app this morning.

…h digits surrounding it eg: 8x8

codyhawke · 2019-07-28T03:40:13Z

Still not working for some of my URLS such as the following:
https://xxxxxxxxx.cloudfront.net/400x400/products/img/ricky-powell-beastie-boys-18x13-1xrun-01.jpg

rpong · 2019-07-28T08:34:09Z

Still not working for some of my URLS such as the following:
https://xxxxxxxxx.cloudfront.net/400x400/products/img/ricky-powell-beastie-boys-18x13-1xrun-01.jpg

https://i.ibb.co/v4RRc5h/Screen-Shot-2019-07-28-at-4-35-55-PM.png

thanks for that use case, just pushed a change to the PR which should take care of it, strange that the unit-tests missed this.

adam91holt · 2019-07-28T21:57:57Z

Hi @rpong

Thanks for all your hard work. Another issue I just spotted is when you form a URL like so...

https://d1h4lz25tmeef3.cloudfront.net/fit-in/400x/clients/demo/images/flip/ichts176-navaksha-original-imaeceu7sqmcekt8.jpeg

Specifying "400x" with the old Thumbor used to give you the best fit for 400 and figure it out itself. However, doing that now will give you "NoSuchKey". If you use 400x400 for example then it works fine.

https://d1h4lz25tmeef3.cloudfront.net/fit-in/400x400/clients/demo/images/flip/ichts176-navaksha-original-imaeceu7sqmcekt8.jpeg

rpong · 2019-07-29T04:35:06Z

https://d1h4lz25tmeef3.cloudfront.net/fit-in/400x400/clients/demo/images/flip/ichts176-navaksha-original-imaeceu7sqmcekt8.jpeg

Hi @adam91holt , noted on this, for now, if this is an actual production use case for you, what we do to circumvent around this is to specify zero and a fit-in.

https://d1h4lz25tmeef3.cloudfront.net/fit-in/400x0/clients/demo/images/flip/ichts176-navaksha-original-imaeceu7sqmcekt8.jpeg

I did not realize they support 400x before, such dimension convention also wasn't in the unit tests so we did not cover that.

beyerz · 2019-09-01T08:15:22Z

This PR resolves some major BC issues for me. Is there an expected time that we can see this merged back into the main repo?

carlmathisen · 2019-09-10T12:01:09Z

Any news on this PR making it in?

andyfurniss4 · 2019-09-17T13:35:10Z

I've tried implementing this manually by exporting the existing Lambda function code, making the changes to the files and then re-uploading a newly zipped version of the function but I'm getting the following error:

module initialization error: Error
at Object.hasVendoredLibvips (/var/task/node_modules/sharp/lib/libvips.js:61:13)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/constructor.js:9:22)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/index.js:3:15)

Any ideas?

rpong · 2019-09-18T02:48:09Z

I've tried implementing this manually by exporting the existing Lambda function code, making the changes to the files and then re-uploading a newly zipped version of the function but I'm getting the following error:

module initialization error: Error
at Object.hasVendoredLibvips (/var/task/node_modules/sharp/lib/libvips.js:61:13)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/constructor.js:9:22)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/index.js:3:15)

Any ideas?

easier to just use this pre-built one: #130 (comment) goodluck and let us know.

GeorgeBellTMH · 2019-09-23T05:54:11Z

Any updates?

komandar · 2019-11-07T12:33:53Z

I've tried implementing this manually by exporting the existing Lambda function code, making the changes to the files and then re-uploading a newly zipped version of the function but I'm getting the following error:
module initialization error: Error
at Object.hasVendoredLibvips (/var/task/node_modules/sharp/lib/libvips.js:61:13)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/constructor.js:9:22)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/index.js:3:15)
Any ideas?
easier to just use this pre-built one: #130 (comment) goodluck and let us know.

Worked very well for me in combination with #157

padre · 2019-12-03T20:43:31Z

@rpong Thanks for your contribution.

But there is a problem: if the filename contains square brackets, the regular expression does not work.

Example: [test].jpg does not work (throw error 400)

rpong · 2019-12-18T11:03:02Z

I've tried implementing this manually by exporting the existing Lambda function code, making the changes to the files and then re-uploading a newly zipped version of the function but I'm getting the following error:
module initialization error: Error
at Object.hasVendoredLibvips (/var/task/node_modules/sharp/lib/libvips.js:61:13)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/constructor.js:9:22)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/var/task/node_modules/sharp/lib/index.js:3:15)
Any ideas?
easier to just use this pre-built one: #130 (comment) goodluck and let us know.
Worked very well for me in combination with #157

Glad to help, btw just released the same package we use with Node 12.x -> https://github.com/innovationlove/serverless-image-handler/releases/tag/4.0.3

Goodluck.

rpong · 2019-12-18T11:03:32Z

@rpong Thanks for your contribution.

But there is a problem: if the filename contains square brackets, the regular expression does not work.

Example: [test].jpg does not work (throw error 400)

Will look into this within the week.

padre · 2019-12-19T15:52:32Z

@rpong Thank you very much, again! But I don't think it's working properly. How do you crop an image to exactly a specified size?

Before upgrading, if I didn't use the fit-in argument, the image auto-cropped and auto-resized to be EXACTLY the specified size. Now domain.com/fit-in/50x50/image.jpg and domain.com/50x50/image.jpg produces the same result :-(

Vita1ik · 2019-12-26T12:06:23Z

Why in this case https://cloudfront-fake-endpoint.net/fit-in/400x0/image_id.jpg image is not covered, I receive an empty object(

It doesn't work with lambda which runs on node.js 12.x, but I have the older Serverless Image Handler that runs the lambda on node.js 8.10 and this one works properly

OBrown92 · 2020-01-09T10:48:30Z

Got also some problems with the newest version 4.1. Is it still possible to deploy version 4 and then upgrade the node version with the release from innovationlove?

Vita1ik · 2020-01-15T14:54:26Z

Why in this case https://cloudfront-fake-endpoint.net/fit-in/400x0/image_id.jpg image is not covered, I receive an empty object(

It doesn't work with lambda which runs on node.js 12.x, but I have the older Serverless Image Handler that runs the lambda on node.js 8.10 and this one works properly

filters:upscale() solved my issue =)

beomseoklee · 2020-02-06T21:35:35Z

Thanks for your effort here, @rpong
We have updated our solution, and your pull request has been merged into our master branch. However, as we've merged many pull requests at this time, the source code might not exactly same as what you've done. In addition, due to our internal process, we've merged source code manually, so please understand how your effort has been merged.

You can refer to the recent changes here

ijhar8 · 2020-03-09T13:09:18Z

@rpong this solution is not working for image size more thane 4MB .
getting error .
{"message": "Internal server error"}

OBrown92 · 2020-03-09T13:13:35Z

I think this is more a Aws Lambda problem because the request and response size is limited:
Request and response body payload size are maximized to 6 MB. (aws)
Maybe some overhead or something consuming the 2mb left.

giovannipds · 2020-03-16T21:55:02Z

Is there a way to use the old fit-in behavior url without having to overwrite the code?

cxrod · 2020-04-06T22:19:05Z

@giovannipds the old fit-it is working on the last release. Just update your stack with the latest template.

giovannipds · 2020-04-14T12:04:35Z

@cxrod seriously? I probably have edited the code before putting the stack up the last time I've updated here (29 days ago).

giovannipds · 2020-04-14T12:05:54Z

@cxrod I won't re-do this right now since it's already working as I need on my stack 😆 but thanks for letting me know.

cxrod · 2020-04-15T20:57:12Z

@giovannipds sorry for the late reply! I was observing for a long time the PR with the fix, until it was released on February

giovannipds · 2020-04-15T21:35:52Z

@cxrod no worries! o/

Arpee Ong added 2 commits July 25, 2019 02:10

Support serving images under s3 subdirectories

98c94a4

Handle all thumbor URLs when trying to get s3 key

fe1bf84

rpong mentioned this pull request Jul 25, 2019

images in subfolders doesn't work #112

Closed

Regex adjustments to pass the Tests

3fc4fd0

Make /fit-in/ directive work again

03f3a1a

rpong mentioned this pull request Jul 25, 2019

/fit-in not working with 4.0.0 #114

Closed

rpong changed the title ~~Support serving images under s3 subdirectories~~ Support serving images under s3 subdirectories and fix to make /fit-in/ work Jul 25, 2019

Arpee Ong added 2 commits July 27, 2019 15:00

Fix VipsJpeg: Invalid SOS parameters for sequential JPEG errors, and …

7c117fa

…follow exif-specified rotation by default

Fix issue wherein image is not served if no dimension is set, sorry, …

ec97cac

…my bad

rpong changed the title ~~Support serving images under s3 subdirectories and fix to make /fit-in/ work~~ Support serving images under s3 subdirectories, Fix to make /fit-in/ work and Fix for VipsJpeg: Invalid SOS error Jul 27, 2019

Fix for proper resizing if the filename or path contains letter x wit…

7dbf592

…h digits surrounding it eg: 8x8

rpong changed the title ~~Support serving images under s3 subdirectories, Fix to make /fit-in/ work and Fix for VipsJpeg: Invalid SOS error~~ Support serving images under s3 subdirectories, Fix to make /fit-in/ work and Fix for VipsJpeg: Invalid SOS error plus several other fixes. Jul 27, 2019

rpong mentioned this pull request Jul 27, 2019

Getting a "The specified key does not exist" error even though the object is present in S3 #123

Closed

Fix for proper resizing if the filename or path contains letter x wit…

2f1bf56

…h digits surrounding it eg: 8x8

Improve getting the s3 key from various filename conventions

4993496

rpong mentioned this pull request Jul 28, 2019

Feature Request: Default Image Fallback #137

Closed

rpong force-pushed the master branch from 5c0c83b to aae497d Compare December 18, 2019 09:33

Arpee Ong added 3 commits December 18, 2019 17:49

Update Runtime to Node12.x, Update Sharp to 0.23.1

63a4f64

Handle incomplete dimensions

bf10401

Reverting sharp to 0.22.1 due to fit-in/inside behaviour in 0.23.x

36381ed

tapughose mentioned this pull request Dec 24, 2019

Image handler V4.1 cannot access objects inside S3 folder #175

Closed

OBrown92 mentioned this pull request Feb 7, 2020

400x0 is resulting in a empty response #183

Closed

georgebearden closed this Feb 10, 2020

beomseoklee mentioned this pull request Feb 13, 2020

Semantic File URLs #184

Open

Support serving images under s3 subdirectories, Fix to make /fit-in/ work; Fix for VipsJpeg: Invalid SOS error plus several other critical fixes. #130

Support serving images under s3 subdirectories, Fix to make /fit-in/ work; Fix for VipsJpeg: Invalid SOS error plus several other critical fixes. #130

Conversation

rpong commented Jul 24, 2019 • edited Loading

beyerz commented Jul 25, 2019

denys1yermakov commented Jul 25, 2019 • edited Loading

codyhawke commented Jul 26, 2019

rpong commented Jul 27, 2019

rpong commented Jul 27, 2019 • edited Loading

gendigbadig commented Jul 27, 2019 • edited Loading

denys1yermakov commented Jul 27, 2019

gendigbadig commented Jul 27, 2019 • edited Loading

rpong commented Jul 27, 2019

codyhawke commented Jul 28, 2019

rpong commented Jul 28, 2019 • edited Loading

adam91holt commented Jul 28, 2019

rpong commented Jul 29, 2019

beyerz commented Sep 1, 2019

carlmathisen commented Sep 10, 2019

andyfurniss4 commented Sep 17, 2019

rpong commented Sep 18, 2019

GeorgeBellTMH commented Sep 23, 2019

komandar commented Nov 7, 2019

padre commented Dec 3, 2019

rpong commented Dec 18, 2019

rpong commented Dec 18, 2019

padre commented Dec 19, 2019

Vita1ik commented Dec 26, 2019

OBrown92 commented Jan 9, 2020

Vita1ik commented Jan 15, 2020

beomseoklee commented Feb 6, 2020

ijhar8 commented Mar 9, 2020

OBrown92 commented Mar 9, 2020 • edited Loading

giovannipds commented Mar 16, 2020

cxrod commented Apr 6, 2020

giovannipds commented Apr 14, 2020

giovannipds commented Apr 14, 2020

cxrod commented Apr 15, 2020

giovannipds commented Apr 15, 2020

rpong commented Jul 24, 2019 •

edited

Loading

denys1yermakov commented Jul 25, 2019 •

edited

Loading

rpong commented Jul 27, 2019 •

edited

Loading

gendigbadig commented Jul 27, 2019 •

edited

Loading

gendigbadig commented Jul 27, 2019 •

edited

Loading

rpong commented Jul 28, 2019 •

edited

Loading

OBrown92 commented Mar 9, 2020 •

edited

Loading