[SMARTSCAN] [$500] LOW: Android - Scan-When adding big image, thumbnail display is not clear, image distorted

[izarutskaya]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Version Number: 1.4.9-0
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause-Internal Team
Slack conversation: @

Action Performed:

1. Launch app
2. Tap fab---Request money--Scan
3. Upload below given image
4. Select a user
5. Tap request
6. Tap request created
7. Note the thumbnail receipt

Expected Result:

When adding big image, thumbnail display must be clear.

Actual Result:

When adding big image as scan receipt, thumbnail display is not clear, image distorted.

Workaround:

Unknown

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android: Native
• Android: mWeb Chrome
• iOS: Native
• iOS: mWeb Safari
• MacOS: Chrome / Safari
• MacOS: Desktop

Screenshots/Videos

View all open jobs on GitHub

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~0122f62b0e85d3a0e6
• Upwork Job ID: 1732723141179482112
• Last Price Increase: 2023-12-07

Issue Owner

Current Issue Owner: @Julesssss

[melvin-bot]

Triggered auto assignment to @abekkala (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~0122f62b0e85d3a0e6

[melvin-bot]

Bug0 Triage Checklist (Main S/O)

• This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @Ollyws (External)

[ZhenjaHorbach]

Proposal

Please re-state the problem that we are trying to solve in this issue.

Scan-When adding big image, thumbnail display is not clear, image distorted

What is the root cause of that problem?

This is actually expected behavior
In addition to the original images
We use thumbnail versions
The height of which is (1024)
And if we have such a long image
We get such funny values (37x1024)

What changes do you think we should make in order to solve the problem?

The most important thing is we need to have access to better images
Which are stored on the backend
And use them if we have problems with image quality (We can evaluate quality using Image.getSize )

App/src/libs/ReceiptUtils.ts

Line 51 in 10727a7

return {thumbnail: `${path}.1024.jpg`, image: path};

---

But from the front side we can use the original image in case, for example, thumbnail photo width is less than 200 pixels (Or we can calculate the aspect ratio If our image is too long, then there is no point in showing thumbnail)

For this, we can add new conditions and use Image

const [isBlurryThumbnail,setBlurryThumbnail] = useState(false)

	Image.getSize(thumbnailSource, (width, height) => {
		if(width < 200) {
		    setBlurryThumbnail(true)
		}
	});
	
	

App/src/components/ReportActionItem/ReportActionItemImage.js

Lines 49 to 73 in 10727a7

	function ReportActionItemImage({thumbnail, image, enablePreviewModal, transaction, isLocalFile}) {
	const styles = useThemeStyles();
	const {translate} = useLocalize();
	const imageSource = tryResolveUrlFromApiRoot(image || '');
	const thumbnailSource = tryResolveUrlFromApiRoot(thumbnail || '');
	const isEReceipt = !_.isEmpty(transaction) && TransactionUtils.hasEReceipt(transaction);
	let receiptImageComponent;
	if (isEReceipt) {
	receiptImageComponent = (
	<View style={[styles.w100, styles.h100]}>
	<EReceiptThumbnail transactionID={transaction.transactionID} />
	</View>
	);
	} else if (thumbnail && !isLocalFile) {
	receiptImageComponent = (
	<ThumbnailImage
	previewSourceURL={thumbnailSource}
	style={[styles.w100, styles.h100]}
	isAuthTokenRequired
	shouldDynamicallyResize={false}
	/>
	);
	} else {

And then if isBlurryThumbnail is true we will use imageSource instead thumbnailSource

App/src/components/ReportActionItem/ReportActionItemImage.js

Line 67 in 10727a7

previewSourceURL={thumbnailSource}

What alternative solutions did you explore? (Optional)

NA

[Ollyws]

@ZhenjaHorbach Thanks for the proposal but it seems like a bit of a workaround.
I think a better solution would be to refine the thumbnail cropping on the backend.

@abekkala Could we have an internal engineer take a look? Thanks!

[melvin-bot]

Current assignee @Ollyws is eligible for the Internal assigner, not assigning anyone new.

[melvin-bot]

Triggered auto assignment to @yuwenmemon (Engineering), see https://stackoverflow.com/c/expensify/questions/4319 for more details.

[abekkala]

@yuwenmemon just wanted to get your opinion on this
#32649 (comment)

[yuwenmemon]

Yeah I'd rather go with cropping the image for a thumbnail on the backend than the proposed solution. Let's make this internal.

[melvin-bot]

Current assignee @Ollyws is eligible for the Internal assigner, not assigning anyone new.

[abekkala]

tagged Greg in #wave6-collect-submitters to get confirmation to add this to that project

Edit: this may actually be wave 5

[Julesssss]

We can no longer download the large image attached above... Github shows the image as not available when I try to open it or download it 😕

I found two alternative images:

• With both images the quality is reduced within the 'Receipt scan in progress' View
• With image 1 below I'm not seeing any image quality reductions, but this is a much shorter image
• With image 2 the app is crashing, though that is a different bug from the original report

Testing image 1

screen-20240110-153736.mp4

Image 1

Testing image 2

screen-20240110-154834.mp4

image 2

[Julesssss]

Here's the Android Exception: we previously resolved a similar issue by switching image libraires and I'll probably handle this separately.

FATAL EXCEPTION: main
    Process: com.expensify.chat, PID: 28882
    java.lang.RuntimeException: Canvas: trying to draw too large(149730560bytes) bitmap.
    	at android.graphics.RecordingCanvas.throwIfCannotDraw(RecordingCanvas.java:266)
    	at android.graphics.BaseRecordingCanvas.drawBitmap(BaseRecordingCanvas.java:94)
    	at android.graphics.drawable.BitmapDrawable.draw(BitmapDrawable.java:549)
    	at android.widget.ImageView.onDraw(ImageView.java:1446)
    	at expo.modules.image.ExpoImageView.onDraw(ExpoImageView.kt:204)
    	at android.view.View.draw(View.java:23900)
    	at expo.modules.image.ExpoImageView.draw(ExpoImageView.kt:200)
    	at android.view.View.updateDisplayListIfDirty(View.java:22767)
    	at android.view.View.draw(View.java:23631)
    	at android.view.ViewGroup.drawChild(ViewGroup.java:4559)
    	at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4320)
    	at android.view.View.updateDisplayListIfDirty(View.java:22758)
    	at android.view.View.draw(View.java:23631)
    	at android.view.ViewGroup.drawChild(ViewGroup.java:4559)
    	at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4320)

[Julesssss]

I created a separate issue for resolving the exception here, which will become my top focus.

[Julesssss]

Beneath a height/width ratio of about 1:3, the thumbnail becomes indistinguishable:

Image1: 1904/19660 > 99/1024 ❌
Image2: 730/4283 > 175/1024 ❌
Image3: 730/2603 > 287/1024 ❌
Image4: 730/1796 > 416/1024 ✅

We currently lock thumbnail resolution to 1024px max. To resolve the issue we'll need to apply further logic to images with a w/h ratio greater than 3:1 in either width or height. I see a few options:

1. Crop the image at the center -- this keeps thumbnails small, but the preview doesn't show the true height/width ✅
2. Increase the max width from 1024 -- this isn't ideal as very tall or wide images will have the same problem with low-resolution thumbnails ❌
3. Prevent users from uploading images above a certain resolution -- bad as users receipts will fail and need to be edited manually ❌
4. Discard the thumbnail and display the full-resolution image in NewDot -- workaround ❌

NewDot thumbnails are displayed much larger than OldDot, so we should probably limit these changes to NewDot to avoid breaking OldDot

note: our thumbnail suffix for avatars is _128.jpeg

[Julesssss]

When is thumbnail created?

A) OldDot Receipt: Scaled to fit into rectangular receipt
B) OldDot Attachment: Displayed unedited
C) NewDot Receipt: cropped & blurry
D) NewDot Receipt preview: cropped
E) NewDot Attachment: Displayed unedited

/	Image
A
B
C
D
E

[Julesssss]

Based on the above, the best course of action is to crop image previews, providing they are NewDot receipts. We still want to keep the scale for tall/wide image attachments.

In both OldDot and NewDot, receipts AND image attachments pass through the same Web-PDF code. The next step is to figure out how to crop with Imagemagick CLI tools and how to pass a shouldCrop flag.

Current Resize logic:

/usr/bin/convert -background white -flatten -limit memory 128mb -limit map 128mb -scale '{$geometry}x{$geometry}> [inputfile] [outputfile]

[Julesssss]

After trialing a few different ImageMagick features (resize, thumbnail, crop) today I figured out the arguments that allows me to perform both cropping and resizing in a single command. Also the ordering of these arguments appear to affect query speed.

Crop and resize, but ONLY if the image exceeds 512 in width or height

// base
convert large-image-1.jpeg -resize 512x512\^ -gravity center -extent 512x512 test.png

// full commad
convert large-image-1.jpeg -resize 512x512\^ -gravity center -extent 512x512 -background white -flatten -limit memory 128mb -limit map 128mb test.png

For images with a width AND height beneath our desired thumbnail size, we run this query to ensure the thumbnail is still created:

// base 
convert small.png -resize 512x512\> test.png

// full command
convert small.png -resize 512x512\> -background white -flatten -limit memory 128mb -limit map 128mb test.png

I have a WIP PR here, the next step is to pass a shouldCrop param and test against all possible usages of this function

[Julesssss]

I made more progress on the PR today:

• Logic is in place for handling images both larger and smaller than the desired width/height
• I debugged all the NewDot/OldDot attachment/Request flows in both Web-E & Web-PDFs
• A shouldCropReceipt param from WEB is being passed down to the Web-PDF createReceipt function

Next Steps:

• Set the createReceipt param in Web-E dynamically
• Check whether ImageMagic scale function can continue to be used for non-cropped images
• Thoroughly test varients of image types and pdfs
• Open up PRs

[Julesssss]

Implementation is complete:

• The Web-E PR that introduces the request param is currently in review
• The main Web-PDFs PR is on hold, but the code and tests are completed

There is a new CI issue within Web-PDFs that I need to fix.

[Julesssss]

I'm ill today, but updated the Web-E PR based on feedback. It should be merged today 🤞

[Julesssss]

I've been sick. I'm back 50% now and trying to get the 2nd half of this task merged.

[Julesssss]

I resolved the CI issue and the full PR is now in review 🎉

[Julesssss]

The PR received a small change request which I have made, it's now back in review.