[HOLD for payment 2023-08-30] [$1000] Web - Markdown pasted from selection-copy is different from clipboard-copy

[kbecciv]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Action Performed:

1. Open any report
2. Paste this markdown (it has a heading, then a code snippet and a quote)

test heading

Code snippet

link

3. Now use the copy to clipboard option, and copy the sent message
4. Paste the message in the composer and send it again (notice that there are no difference in both messages)
5. Now select the sent message text by cursor and press cmd+c
6. Paste the text in the composer and send the message.
7. Notice that the markdown message has an extra line in the code snippet part.

Expected Result:

Message copied from selection should be the same that that copied from context menu

Actual Result:

Message copied from selection has an extra line

Workaround:

Unknown

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android / native
• Android / Chrome
• iOS / native
• iOS / Safari
• MacOS / Chrome / Safari
• MacOS / Desktop

Version Number: 1.3.45-8
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation

Screen.Recording.2023-07-26.at.2.34.29.PM.mp4

Recording.3922.mp4

Expensify/Expensify Issue URL:
Issue reported by: @huzaifa-99
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1690364432300409

View all open jobs on GitHub

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~019c94943755b5fa6d
• Upwork Job ID: 1684694244929720320
• Last Price Increase: 2023-08-03
• Automatic offers:
• mollfpr | Reviewer | 26012873
• huzaifa-99 | Reporter | 26012875

[melvin-bot]

Triggered auto assignment to @puneetlath (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[melvin-bot]

Bug0 Triage Checklist (Main S/O)

• This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~019c94943755b5fa6d

[melvin-bot]

Current assignee @puneetlath is eligible for the External assigner, not assigning anyone new.

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @mollfpr (External)

[akamefi202]

Proposal

Please re-state the problem that we are trying to solve in this issue.

An extra line is added to the markdown text when the user copy & paste the message which contains code snippet using selection & keyboard shortcut.

What is the root cause of that problem?

```
code
```
text

If we add above message and copy it using clipboard button and keyboard shortcut each other, different html contents are saved in clipboard.

<pre>code<br></pre>text \\ copied by clipboard button in context menu

<div><div><comment><pre><div><span>code</span><br></div></pre>text</comment></div></div> \\ copied by selection & Ctrl + C

App/src/components/Composer/index.js

Lines 298 to 301 in 34d1d95

	handlePastedHTML(html) {
	const parser = new ExpensiMark();
	this.paste(parser.htmlToMarkdown(html));
	}

If we paste it in the composer, above html texts will be parsed by handlePastedHTML function of ExpensiMark class and parsed text will be shown in the composer.

https://github.com/Expensify/expensify-common/blob/9940dd127c2d44809c98ee628a8057f08c93bfc9/lib/ExpensiMark.js#L322-L324
For parsing of code snippet, handlePastedHTML function adds head & tail line breaks.
And it removes the new line character at the end of <pre> tag if it exists, for preventing addition of new line characters.

<pre><div><span>code</span><br></div></pre>

But through rendering of HTML content, <div> tag is appended inside <pre> and also <br> tag is inside of <div> tag.
So the end new line character won't be removed and we will see an extra line in markdown text.

What changes do you think we should make in order to solve the problem?

App/src/hooks/useCopySelectionHelper.js

Lines 8 to 9 in 975146f

	function copySelectionToClipboard() {
	const selection = SelectionScraper.getCurrentSelection();

We should parse selection in advance, before saving it to clipboard & parsing it using htmlToMarkdown function.
We should remove the child <div> tag inside <pre> tag and the problem will be fixed.

function parseSelectionHtml(html) {
    return html.replace(/<pre><div>(.*?)<\/div><\/pre>/g, (match, g1) => `<pre>${g1}</pre>`);
}

We might update this parser function for further updates related to selection, or we can add it to ExpensiMark class.

What alternative solutions did you explore? (Optional)

We can add new rule to htmlToMarkdownRules of ExpensifyMark.js directly.
So we can remove the nested tag inside <pre> tag while html selection is being parsed using htmlToMarkdown function.

{
    name: 'nested',
    regex: /<(pre)(?:"[^"]*"|'[^']*'|[^'">])*><(div|code)(?:"[^"]*"|'[^']*'|[^'">])*>([\s\S]*?)<\/\2><\/pre>/gi,
    replacement: '<pre>$3</pre>'
},

[puneetlath]

@mollfpr please review when you get a chance, thanks!

[mollfpr]

Reviewing

[mollfpr]

Need more time for this, I felt unwell and tried again the next day 🙏

[mollfpr]

@akamefi202 Sorry for the delay 🙏

We should parse selection in advance, before saving it to clipboard & parsing it using htmlToMarkdown function.

Why should we parse the selection in advance? The selection listener is only called if we make a text selection in the App right? How do we handle the case where we make text selection outside of the App, for example from this OP issue?

return html.replace(/

(.*?)</div></pre>/g, (match, g1) => <pre>${g1}</pre>);

It seems you didn't have the <br> tag in the expression for the regex rules.

In this case, can we correctly handle the parser on pasting the text?

[akamefi202]

@mollfpr

The selection listener is only called if we make a text selection in the App right? How do we handle the case where we make text selection outside of the App, for example from this OP issue?

Yes, the listener is only called if we copy the selection using keyboard shortcut(Ctrl + C) inside the app.
And if this listener is called, selection will be saved to clipboard in both html & text format.

It seems you didn't have the
tag in the expression for the regex rules. In this case, can we correctly handle the parser on pasting the text?

App/src/components/Composer/index.js

Lines 298 to 301 in 34d1d95

	handlePastedHTML(html) {
	const parser = new ExpensiMark();
	this.paste(parser.htmlToMarkdown(html));
	}

It will be parsed using htmlToMarkdown function of ExpensiMark class when we paste the text.

[mollfpr]

@akamefi202 Could you attach a working result video?

[akamefi202]

@mollfpr Here you go. Please check from 0:55 of video.
Screencast from 2023年08月03日 22时34分26秒.webm

[melvin-bot]

📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸

[mollfpr]

Thanks @akamefi202

We should parse selection in advance, before saving it to clipboard & parsing it using htmlToMarkdown function.
We should remove the child

tag inside

 tag and the problem will be fixed.

Are the changes we need to parse the selection before set to the clipboard? You said that the break line tag is removed from htmlToMarkdown, then why does the issue still occur on copying the text from another source e.g. this issue OP?

Screen.Recording.2023-08-03.at.23.39.17.mov

[akamefi202]

@mollfpr
I only thought about selection inside the app.
Should we fix this problem too? The subject of this issue is that markdown pasted from selection-copy is different from clipboard-copy.
Copying from Github can't avoid making a difference because of completely different html structure.

[akamefi202]

@mollfpr
I think we need to update htmlToMarkdownRules of expensify-common to fix above issue.
Currently, htmlToMarkdownRules contains rules only for parsing html selection from Expensify & Slack.
We need to add rules for parsing html selection from Github too.
https://github.com/Expensify/expensify-common/blob/1076899202ad338232a6cfeb4b242234aa932cb0/lib/ExpensiMark.js#L241-L247

This is example of html selection from Github. It contains unnecessary attributes and tags.

<h3 dir="auto" style="box-sizing: border-box; margin-top: 24px; margin-bottom: 16px; font-size: 1.25em; font-weight: var(--base-text-weight-semibold, 600); line-height: 1.25; color: rgb(31, 35, 40); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;Noto Sans&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">test heading</h3><div class="snippet-clipboard-content notranslate position-relative overflow-auto" style="box-sizing: border-box; position: relative !important; overflow: auto !important; color: rgb(31, 35, 40); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;Noto Sans&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><pre class="notranslate" style="box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, &quot;SF Mono&quot;, Menlo, Consolas, &quot;Liberation Mono&quot;, monospace; font-size: 11.9px; margin-top: 0px; margin-bottom: 16px; overflow-wrap: normal; padding: 16px; overflow: auto; line-height: 1.45; color: var(--fgColor-default, var(--color-fg-default)); background-color: var(--bgColor-muted, var(--color-canvas-subtle)); border-radius: 6px;"><code class="notranslate" style="box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, &quot;SF Mono&quot;, Menlo, Consolas, &quot;Liberation Mono&quot;, monospace; font-size: 11.9px; padding: 0px; margin: 0px; white-space: pre; background: transparent; border-radius: 6px; word-break: normal; border: 0px; display: inline; overflow: visible; line-height: inherit; overflow-wrap: normal;">Code snippet
</code></pre></div><blockquote style="box-sizing: border-box; margin: 0px 0px 16px; padding: 0px 1em; color: var(--fgColor-muted, var(--color-fg-muted)); border-left: .25em solid var(--borderColor-default, var(--color-border-default)); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;Noto Sans&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><p dir="auto" style="box-sizing: border-box; margin-top: 0px; margin-bottom: 0px;"><a href="https://github.com/Expensify/App/issues/www.example.com" style="box-sizing: border-box; background-color: transparent; color: var(--fgColor-accent, var(--color-accent-fg)); text-decoration: none;">link</a></p></blockquote>

I suggest to add below two rules.
First rule is for deleting unnecessary attributes of tags.
Second rule is for deleting unnecessary child tag inside <pre> tag.

{
  name: 'attribute',
  regex: /( *(style|dir|class)=(\"|\')([\s\S]*?)\3)/gi,
  replacement: ''
},
{
  name: 'nested',
  regex: /<pre><(div|code)>([\s\S]*?)<\/\1><\/pre>/gi,
  replacement: '<pre>$2</pre>'
},

Working result:
Screencast from 2023年08月04日 03时45分15秒.webm

[mollfpr]

I only thought about selection inside the app.

We can fix it in both if it's the same root cause.

First rule is for deleting unnecessary attributes of tags.

@akamefi202 Why do we need this? Because of this, there's a failure in the jest.

Although the solution is working. I want to know why div or code causes the newline to appear inside the block code.

[akamefi202]

@mollfpr
Thank you for your opinion. I agree with you. First rule isn't required. We can fix both issues by adding one rule.

Although the solution is working. I want to know why div or code causes the newline to appear inside the block code.

#23659 (comment)
I explained the reason in my proposal.
Currently, handlePastedHTML function adds head & tail line breaks in default. And it removes the new line character at the end of <pre> tag if it exists, for preventing addition of new line characters.
But through rendering of HTML content, <div> or <code> tag is appended inside <pre>. So <br> tag is also inside of <div> tag. As a result, the end new line character won't be removed and we will see an extra line in markdown text.
You can check the regex in below code and it will help understanding.

https://github.com/Expensify/expensify-common/blob/1076899202ad338232a6cfeb4b242234aa932cb0/lib/ExpensiMark.js#L321-L325

[akamefi202]

@mollfpr
For example, if we copy and paste message using clipboard button in context menu. handlePastedHTML function parses html code through below steps.

<pre>code<br></pre> -> <pre>code<pre> -> ```\ncode\n```

But if we copy and pasting using selection and keyboard shortcut.

<pre><div>code<br></div></pre> -> <pre><div>code<br></div></pre> -> ```\ncode\n\n```

[mollfpr]

Thanks @akamefi202, that make sense to me.

I have tested the proposal, and we only need to add 1 rule to remove the <div> or <code> tag to make the test work. Let's go with @akamefi202 proposal then.

🎀 👀 🎀 C+ reviewed!

[melvin-bot]

Current assignee @puneetlath is eligible for the choreEngineerContributorManagement assigner, not assigning anyone new.

[puneetlath]

Sorry, just catching up. I think I like this solution better:

Second rule is for deleting unnecessary child tag inside <pre> tag.

That way we handle it on paste, rather than on copy. And that way no matter how many divs exist inside the pre tags, they'd get removed. What do y'all think?

[akamefi202]

Yeah, I suggested it and @mollfpr agreed with that solution too.

[puneetlath]

Ah ok. The link seemed to be to the earlier proposal, so I wasn't sure. Let's do it then!

[melvin-bot]

📣 @mollfpr 🎉 An offer has been automatically sent to your Upwork account for the Reviewer role 🎉 Thanks for contributing to the Expensify app!

Offer link
Upwork job

[melvin-bot]

📣 @akamefi202 You have been assigned to this job!
Please apply to the Upwork job and leave a comment on the Github issue letting us know when we can expect a PR to be ready for review 🧑‍💻
Once you apply to this job, your Upwork ID will be stored and you will be automatically hired for future jobs!
Keep in mind: Code of Conduct | Contributing 📖

[melvin-bot]

📣 @huzaifa-99 🎉 An offer has been automatically sent to your Upwork account for the Reporter role 🎉 Thanks for contributing to the Expensify app!

Offer link
Upwork job

[akamefi202]

I applied to the Upwork job and I believe that the PR will be ready by tomorrow.

[akamefi202]

I updated proposal.
I added the discussed solution as alternative solution.

[akamefi202]

@mollfpr @puneetlath
PR is ready.

[melvin-bot]

@puneetlath @mollfpr @akamefi202 this issue was created 2 weeks ago. Are we close to approving a proposal? If not, what's blocking us from getting this issue assigned? Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

[melvin-bot]

Based on my calculations, the pull request did not get merged within 3 working days of assignment. Please, check out my computations here:

• when @akamefi202 got assigned: 2023-08-08 16:29:40 Z
• when the PR got merged: 2023-08-17 18:48:00 UTC
• days elapsed: 7

On to the next one 🚀

[akamefi202]

@puneetlath @mollfpr It's Aug 30th. Could you please help me with payment? And can we close this issue now?

[puneetlath]

All paid. Thanks everyone!