[Payment 2025-02-05] [$250] Start chat - User's name containing accent not shown in results if no accent in search query

[vincdargento]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Version Number: 9.0.72.0
Reproducible in staging?: Yes
Reproducible in production?: Yes
If this was caught on HybridApp, is this reproducible on New Expensify Standalone?: Yes, reproducible on both
If this was caught during regression testing, add the test name, ID and link from TestRail: N/A
Email or phone of affected tester (no customers): N/A
Issue reported by: Applause Internal Team

Action Performed:

precondition: there is a user with a name containing an accent mark e.g. Álex, Timón, D'artgnan

1. Go to https://staging.new.expensify.com/ and log in
2. Click on FAB > Start a chat
3. Enter a user's name without an accent mark, e.g. Álex, Timón

Expected Result:

User is displayed in the search results

Actual Result:

User is not displayed in the the search results unless the accent mark is entered. However, when setting an address in Profile, countries that have diacritics are shown when searching without them.

Workaround:

Unknown

Platforms:

• Android: Standalone
• Android: HybridApp
• Android: mWeb Chrome
• iOS: Standalone
• iOS: HybridApp
• iOS: mWeb Safari
• MacOS: Chrome / Safari
• MacOS: Desktop

Screenshots/Videos

bug.mp4

View all open jobs on GitHub

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~021869454978810524666
• Upwork Job ID: 1869454978810524666
• Last Price Increase: 2024-12-18
• Automatic offers:
• nyomanjyotisa | Contributor | 105550541

Issue Owner

Current Issue Owner: @strepanier03

[melvin-bot]

Triggered auto assignment to @strepanier03 (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details. Please add this bug to a GH project, as outlined in the SO.

[Shahidullah-Muffakir]

Proposal

Please re-state the problem that we are trying to solve in this issue.

The problem is that when users search for a term like Alex, it does not return results like Álex, even though Álex exists in the DB. This happens because the search is treating accented and non-accented characters as distinct

What is the root cause of that problem?

The root cause is that we are not accounting for accented characters. When a user types a query like Alex, it does not match with Álex or other similar variants with accents, since the comparison is done exactly as typed, without normalizing for accent differences.

What changes do you think we should make in order to solve the problem?

Frontend Changes: We should deaccent the input query before sending it to the backend. This can be done using the normalize() method in JavaScript, which will convert accented characters (e.g., Á) into their unaccented versions (e.g., A).

App/src/components/Search/SearchRouter/SearchRouterList.tsx

Lines 391 to 393 in 840c227

	useEffect(() => {
	ReportUserActions.searchInServer(autocompleteQueryValue.trim());
	}, [autocompleteQueryValue]);

as:

   useEffect(() => {
        // Normalize (deaccent) the debounced input value before sending to the server
        const normalizedQuery = debouncedInputValue.trim().normalize("NFD").replace(/[\u0300-\u036f]/g, "");
    
        ReportUserActions.searchInServer(normalizedQuery);
    }, [debouncedInputValue]);

Backend Changes: The backend should also be updated to handle accent-insensitive searches

[nyomanjyotisa]

Edited by proposal-police: This proposal was edited at 2024-12-06 05:19:16 UTC.

Proposal

Please re-state the problem that we are trying to solve in this issue.

Start chat - User's name containing accent not shown in results if no accent in search query

What is the root cause of that problem?

We don't sanitize the search term to removes diacritical marks like we did in the country selector and we don't sanitize the data during the filter

App/src/pages/settings/Profile/PersonalDetails/CountrySelectionPage.tsx

Line 35 in 840c227

searchValue: StringUtils.sanitizeString(`${countryISO}${countryName}`),

App/src/libs/OptionsListUtils.ts

Lines 1630 to 1665 in 840c227

	const matchResults = searchTerms.reduceRight((items, term) => {
	const recentReports = filterArrayByMatch(items.recentReports, term, (item) => {
	const values: string[] = [];
	if (item.text) {
	values.push(item.text);
	}
	if (item.login) {
	values.push(item.login);
	values.push(item.login.replace(CONST.EMAIL_SEARCH_REGEX, ''));
	}
	if (item.isThread) {
	if (item.alternateText) {
	values.push(item.alternateText);
	}
	} else if (!!item.isChatRoom || !!item.isPolicyExpenseChat) {
	if (item.subtitle) {
	values.push(item.subtitle);
	}
	}
	return uniqFast(values);
	});
	const personalDetails = filterArrayByMatch(items.personalDetails, term, (item) => uniqFast(getPersonalDetailSearchTerms(item)));
	const currentUserOptionSearchText = items.currentUserOption ? uniqFast(getCurrentUserSearchTerms(items.currentUserOption)).join(' ') : '';
	const currentUserOption = isSearchStringMatch(term, currentUserOptionSearchText) ? items.currentUserOption : null;
	return {
	recentReports: recentReports ?? [],
	personalDetails: personalDetails ?? [],
	userToInvite: null,
	currentUserOption,
	};
	}, options);

What changes do you think we should make in order to solve the problem?

Sanitize the search term and the data while filtering

Change this code to the following

See updated code

    const matchResults = searchTerms.reduceRight((items, term) => {
        const sanitizedTerm = StringUtils.sanitizeString(term);
    
        const recentReports = filterArrayByMatch(items.recentReports, sanitizedTerm, (item) => {
            const values: string[] = [];
            if (item.text) {
                values.push(StringUtils.sanitizeString(item.text));
            }
    
            if (item.login) {
                values.push(StringUtils.sanitizeString(item.login));
                values.push(StringUtils.sanitizeString(item.login.replace(CONST.EMAIL_SEARCH_REGEX, '')));
            }
    
            if (item.isThread) {
                if (item.alternateText) {
                    values.push(StringUtils.sanitizeString(item.alternateText));
                }
            } else if (!!item.isChatRoom || !!item.isPolicyExpenseChat) {
                if (item.subtitle) {
                    values.push(StringUtils.sanitizeString(item.subtitle));
                }
            }
    
            return uniqFast(values);
        });
    
        const personalDetails = filterArrayByMatch(items.personalDetails, sanitizedTerm, (item) =>
            uniqFast(getPersonalDetailSearchTerms(item).map(StringUtils.sanitizeString))
        );
    
        const currentUserOptionSearchText = items.currentUserOption
            ? uniqFast(getCurrentUserSearchTerms(items.currentUserOption).map(StringUtils.sanitizeString)).join(' ')
            : '';
    
        const currentUserOption = isSearchStringMatch(sanitizedTerm, currentUserOptionSearchText)
            ? items.currentUserOption
            : null;
    
        return {
            recentReports: recentReports ?? [],
            personalDetails: personalDetails ?? [],
            userToInvite: null,
            currentUserOption,
        };
    }, options);

Also we need to update the BE accordingly as well, and sanitize the searchInput param if needed

POC

New-Expensify.36.mp4

What alternative solutions did you explore? (Optional)

Same solution as the main solution, but use StringUtils.normalizeAccents instead of StringUtils.sanitizeString

[melvin-bot]

@strepanier03 Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

[strepanier03]

I am having a hard time confirming the expected behavior as well on Mac/Chrome.

I created an account with the name Álex Timón D'artgnan.

• When I search any portion of Álex Timón D'artgnan, or the whole name, I get no results.
• It's also true I get no results when searching Alex Timon Dartgnan either.

[melvin-bot]

@strepanier03 Eep! 4 days overdue now. Issues have feelings too...

[melvin-bot]

@strepanier03 Still overdue 6 days?! Let's take care of this!

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~021869454978810524666

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @c3024 (External)

[melvin-bot]

@strepanier03 @c3024 this issue was created 2 weeks ago. Are we close to approving a proposal? If not, what's blocking us from getting this issue assigned? Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

[c3024]

There seems to be a problem with logging in now. I will check the proposals later and update.

[c3024]

We need to sanitize the text in the returned results to ensure they appear correctly in the search results for the search text.

@nyomanjyotisa 's RCA and proposal LGTM for the frontend!

Backend needs to normalize accents and send relevant results too!

👀 🎀 👀 C+ reviewed.

[c3024]

🎀 👀 🎀

[melvin-bot]

Triggered auto assignment to @MonilBhavsar, see https://stackoverflow.com/c/expensify/questions/7972 for more details.

[melvin-bot]

@strepanier03, @MonilBhavsar, @c3024 Whoops! This issue is 2 days overdue. Let's get this updated quick!

[melvin-bot]

📣 @nyomanjyotisa 🎉 An offer has been automatically sent to your Upwork account for the Contributor role 🎉 Thanks for contributing to the Expensify app!

Offer link
Upwork job
Please accept the offer and leave a comment on the Github issue letting us know when we can expect a PR to be ready for review 🧑‍💻
Keep in mind: Code of Conduct | Contributing 📖

[MonilBhavsar]

@nyomanjyotisa it would be great to add automated tests for this use case

[yuwenmemon]

This went out in the deploy but I do want to highlight the fact that it wasn't working for our QA team: #55698 (comment)

So there might be some revisiting necessary in this issue.

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.0.90-6 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• User's name containing accent not shown in results if no accent in search query #55698

If no regressions arise, payment will be issued on 2025-02-05. 🎊

For reference, here are some details about the assignees on this issue:

• @nyomanjyotisa requires payment automatic offer (Contributor)
• @c3024 requires payment through NewDot Manual Requests

[melvin-bot]

@c3024 @strepanier03 @c3024 The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed. Please copy/paste the BugZero Checklist from here into a new comment on this GH and complete it. If you have the K2 extension, you can simply click: [this button]

[strepanier03]

@nyomanjyotisa and @c3024 - Can one of you check on Yuwen's comment here and confirm if there's anything more that needs to happen here?

If not, feel free to complete the checklist and I'll handle the reg test if you suggest one. Then I'll handle payment.

[c3024]

@yuwenmemon

When the contact is not available on the client, it is searched for on the server. A backend fix is required for that, as specified here.

When the contact is available on the client, the filtering works fine, as seen in the initial part of the video shared. So, I think there is nothing left to be handled on the frontend here.

cc: @strepanier03

[c3024]

BugZero Checklist:

• [Contributor] Classify the bug:

Bug classification

Source of bug:

• 1a. Result of the original design (eg. a case wasn't considered)
• 1b. Mistake during implementation
• 1c. Backend bug
• 1z. Other:

Where bug was reported:

• 2a. Reported on production (eg. bug slipped through the normal regression and PR testing process on staging)
• 2b. Reported on staging (eg. found during regression or PR testing)
• 2d. Reported on a PR
• 2z. Other:

Who reported the bug:

• 3a. Expensify user
• 3b. Expensify employee
• 3c. Contributor
• 3d. QA
• 3z. Other:

• [Contributor] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake.

  Link to comment: This is a specific case of handling the accented letters. No PR can be held responsible for this bug.

• [Contributor] If the regression was CRITICAL (e.g. interrupts a core flow) A discussion in #expensify-open-source has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner.

  Link to discussion: The regression is not critical enough to warrant a discussion.

• [Contributor] If it was decided to create a regression test for the bug, please propose the regression test steps using the template below to ensure the same bug will not reach production again.

• [BugZero Assignee] Create a GH issue for creating/updating the regression test once above steps have been agreed upon.

  Link to issue:

Regression Test Proposal

Precondition:

Test:

1. Click on FAB > Start a chat
2. Enter a user's name without an accent mark, e.g. Alex or Timon for Álex or Timón
3. Verify that the result with name with the accent mark (Álex or Timón) is displayed in the search results for the search term (Alex or Timon).

Do we agree 👍 or 👎

[strepanier03]

Payment Summary

• $250 -- @nyomanjyotisa paid via automatic offer (Contributor)
• $250 -- @c3024 requires payment through NewDot Manual Requests

[strepanier03]

Payment summary above @c3024 - feel free to request.