Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support search in iframe based webview #96307

Closed
mjbvz opened this issue Apr 27, 2020 · 30 comments · Fixed by #125856
Closed

Support search in iframe based webview #96307

mjbvz opened this issue Apr 27, 2020 · 30 comments · Fixed by #125856
Assignees
Labels
electron-12-update Issues and items related to electron 12 update feature-request Request for new features or functionality insiders-released Patch has been released in VS Code Insiders on-release-notes Issue/pull request mentioned in release notes on-testplan sandbox Running VSCode in a node-free environment web Issues related to running VSCode in the web webview Webview issues
Milestone

Comments

@mjbvz
Copy link
Collaborator

mjbvz commented Apr 27, 2020

For #83188

Problem

On desktop, VS Code's webviews using the electron <webview> tag. This gives a nice api for searching inside a webview

However on web we use normal iframes for webviews. These currently don't support the search functionality because there is no equivalent API for searching inside a iframe.

This is one blocker for using normal iframes for the webviews in desktop VS Code: #83188

Request

Implement search for iframe based webviews. This should attempt to match the behavior of the existing search functionality as much as possible

/cc @deepak1556

@mjbvz mjbvz added feature-request Request for new features or functionality webview Webview issues web Issues related to running VSCode in the web labels Apr 27, 2020
@mjbvz mjbvz added this to the May 2020 milestone Apr 27, 2020
@mjbvz mjbvz self-assigned this Apr 27, 2020
@mjbvz
Copy link
Collaborator Author

mjbvz commented Apr 27, 2020

Solutions investigated:

  • Window.find — Non-standard. Doesn't provide enough control

  • JavaScript library (such as mark.js) — Requires JS on page. Modifies page content

  • Find text and other proposals — Never standardized and no longer active :(

@mjbvz
Copy link
Collaborator Author

mjbvz commented Feb 27, 2021

I've explored various ways to implement this. There are two parts to this:

  1. Performing the search in the webview document itself.

    Here we need to get a list of ranges. This is more complicated than it may sound since a search result may span multiple html nodes

  2. Highlight the results

    The only way I found that might let us do this somewhat reliably is execCommand('backColor'). This API is terrible to work with

@deepak1556 deepak1556 modified the milestones: Backlog, March 2021 Feb 27, 2021
@deepak1556
Copy link
Collaborator

deepak1556 commented Feb 27, 2021

This iteration the plan on the runtime side is to explore extending the existing find service in chromium which handles frame based search pretty well, we only need to add a way to filter out per-frame results rather than results from the entire document. The level of implementation for this will vary depending on where we are carrying out this result filter (at content layer or implement at the WebFramMain layer in electron). It would be best if we landed on an implementation on the native side for performance reasons and ease of use.

@rzhao271
Copy link
Contributor

I've been changing around the Fiddle to understand the search behaviour better.
For example, I added a setInterval which called the find and find next functions automatically.
I noticed that when it comes to iframes inside of webviews, the output is traversed irregularly:
recording (15)
Note that the traversal pattern also changes when I change the focus by clicking around.

Meanwhile, with iframes inside of iframes, all matches of the outer iframe are traversed first, before those of the inner iframe are traversed. Iframes inside of iframes are also loadable on Microsoft Edge, so here's how searching in nested iframes on Microsoft Edge (using their find widget, not setInterval) looks:
recording (16)

I thought that I would have to consider these cases for notebook output cells, but it turns out there's already another issue, where notebook output cells currently can't be searched: #94239.

Furthermore, I'm still unsure how focusing and getting the active match works in detail in the case of a single iframe with a search inputbox, and I plan on adding some logging statements later to get the values of some variables.

@rzhao271
Copy link
Contributor

Just for reference, here's another focus-related bug, but I'm unable to currently tell whether it's the same issue as this one: #113511 (comment)

@rzhao271
Copy link
Contributor

rzhao271 commented May 17, 2021

I added some logging.
The first set of logs represent the current actual behaviour. The second set of logs represent the behaviour when I comment out the line in Blink that sets the frame as focused.
What's interesting is that FinalUpdateReceived in both cases concludes there is a focused frame (the part that says if 2). I wouldn't be surprised if it's on the FindRequestManager side though rather than Blink, and I can check this more tomorrow.

[21536:0517/155313.623:ERROR:find_request_manager.cc(670)] FindRequestManager::SendFindRequest
[21536:0517/155313.623:ERROR:find_request_manager.cc(671)] request.options->new_session: 1
[27512:0517/155313.633:ERROR:find_in_page.cc(99)] FindInPage::Find calling InitNewSession
[27512:0517/155313.634:ERROR:find_in_page.cc(147)] FindInPage::Find calling StartScopingStringMatches
[27512:0517/155313.635:ERROR:find_task_controller.cc(198)] FindTaskController::RequestFindTask
[27512:0517/155313.647:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[27512:0517/155313.648:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[27512:0517/155313.648:ERROR:find_task_controller.cc(198)] FindTaskController::RequestFindTask
[21536:0517/155313.649:ERROR:find_in_page_client.cc(31)] FindInPageClient::SetNumberOfMatches #matches = 2
[27512:0517/155313.656:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[27512:0517/155313.656:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[27512:0517/155313.657:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[27512:0517/155313.657:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[27512:0517/155313.658:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[27512:0517/155313.665:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[27512:0517/155313.671:ERROR:find_task_controller.cc(198)] FindTaskController::RequestFindTask
[21536:0517/155313.672:ERROR:find_in_page_client.cc(31)] FindInPageClient::SetNumberOfMatches #matches = 8
[27512:0517/155313.672:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[27512:0517/155313.673:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[21536:0517/155313.684:ERROR:find_in_page_client.cc(31)] FindInPageClient::SetNumberOfMatches #matches = 10
[21536:0517/155313.684:ERROR:find_in_page_client.cc(31)] FindInPageClient::SetNumberOfMatches #matches = 10
[21536:0517/155313.685:ERROR:find_in_page_client.cc(64)] FindInPageClient::HandleUpdateType kFinalUpdate true
[21536:0517/155313.686:ERROR:find_request_manager.cc(339)] FindRequestManager::HandleFinalUpdateForFrame
[21536:0517/155313.686:ERROR:find_request_manager.cc(795)] FindRequestManager::FinalUpdateReceived
[21536:0517/155313.687:ERROR:find_request_manager.cc(824)] FindRequestManager::FinalUpdateReceived if 2
[21536:0517/155313.687:ERROR:find_request_manager.cc(852)] FindRequestManager::FinalUpdateReceived Sending find request
[21536:0517/155313.688:ERROR:find_request_manager.cc(670)] FindRequestManager::SendFindRequest
[21536:0517/155313.689:ERROR:find_request_manager.cc(671)] request.options->new_session: 0
[27512:0517/155313.695:ERROR:find_in_page.cc(107)] FindInPage::Find calling FindInternal
[27512:0517/155313.697:ERROR:find_in_page.cc(140)] FindInPage::Find calling IncreaseMatchCount
[21536:0517/155313.698:ERROR:find_in_page_client.cc(44)] FindInPageClient::SetActiveMatch
[21536:0517/155313.698:ERROR:find_in_page_client.cc(31)] FindInPageClient::SetNumberOfMatches #matches = 10
[21536:0517/155313.699:ERROR:find_in_page_client.cc(64)] FindInPageClient::HandleUpdateType kFinalUpdate true
[21536:0517/155313.700:ERROR:find_request_manager.cc(339)] FindRequestManager::HandleFinalUpdateForFrame
[21536:0517/155313.700:ERROR:find_request_manager.cc(795)] FindRequestManager::FinalUpdateReceived
[21536:0517/155313.701:ERROR:find_request_manager.cc(800)] FindRequestManager::FinalUpdateReceived All find matches in

[21424:0517/160906.253:ERROR:find_request_manager.cc(670)] FindRequestManager::SendFindRequest
[21424:0517/160906.253:ERROR:find_request_manager.cc(671)] request.options->new_session: 1
[23540:0517/160906.263:ERROR:find_in_page.cc(99)] FindInPage::Find calling InitNewSession
[23540:0517/160906.264:ERROR:find_in_page.cc(147)] FindInPage::Find calling StartScopingStringMatches
[23540:0517/160906.264:ERROR:find_task_controller.cc(198)] FindTaskController::RequestFindTask
[23540:0517/160906.277:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[23540:0517/160906.278:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[23540:0517/160906.279:ERROR:find_task_controller.cc(198)] FindTaskController::RequestFindTask
[21424:0517/160906.280:ERROR:find_in_page_client.cc(31)] FindInPageClient::SetNumberOfMatches #matches = 2
[23540:0517/160906.287:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[23540:0517/160906.288:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[23540:0517/160906.290:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[23540:0517/160906.294:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[23540:0517/160906.302:ERROR:find_task_controller.cc(198)] FindTaskController::RequestFindTask
[21424:0517/160906.303:ERROR:find_in_page_client.cc(31)] FindInPageClient::SetNumberOfMatches #matches = 6
[23540:0517/160906.304:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[23540:0517/160906.306:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[23540:0517/160906.318:ERROR:find_task_controller.cc(198)] FindTaskController::RequestFindTask
[21424:0517/160906.318:ERROR:find_in_page_client.cc(31)] FindInPageClient::SetNumberOfMatches #matches = 8
[23540:0517/160906.320:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[23540:0517/160906.325:ERROR:text_finder.cc(489)] TextFinder::DidFindMatch
[21424:0517/160906.326:ERROR:find_in_page_client.cc(31)] FindInPageClient::SetNumberOfMatches #matches = 10
[21424:0517/160906.333:ERROR:find_in_page_client.cc(31)] FindInPageClient::SetNumberOfMatches #matches = 10
[21424:0517/160906.334:ERROR:find_in_page_client.cc(64)] FindInPageClient::HandleUpdateType kFinalUpdate true
[21424:0517/160906.334:ERROR:find_request_manager.cc(339)] FindRequestManager::HandleFinalUpdateForFrame
[21424:0517/160906.335:ERROR:find_request_manager.cc(795)] FindRequestManager::FinalUpdateReceived
[21424:0517/160906.346:ERROR:find_request_manager.cc(824)] FindRequestManager::FinalUpdateReceived if 2 -> has focused frame
[21424:0517/160906.346:ERROR:find_request_manager.cc(852)] FindRequestManager::FinalUpdateReceived Sending find request
[21424:0517/160906.347:ERROR:find_request_manager.cc(670)] FindRequestManager::SendFindRequest
[21424:0517/160906.348:ERROR:find_request_manager.cc(671)] request.options->new_session: 0
[23540:0517/160906.350:ERROR:find_in_page.cc(107)] FindInPage::Find calling FindInternal
[23540:0517/160906.351:ERROR:find_in_page.cc(140)] FindInPage::Find calling IncreaseMatchCount
[21424:0517/160906.352:ERROR:find_in_page_client.cc(44)] FindInPageClient::SetActiveMatch
[21424:0517/160906.353:ERROR:find_in_page_client.cc(31)] FindInPageClient::SetNumberOfMatches #matches = 10
[21424:0517/160906.354:ERROR:find_in_page_client.cc(64)] FindInPageClient::HandleUpdateType kFinalUpdate true
[21424:0517/160906.356:ERROR:find_request_manager.cc(339)] FindRequestManager::HandleFinalUpdateForFrame
[21424:0517/160906.357:ERROR:find_request_manager.cc(795)] FindRequestManager::FinalUpdateReceived
[21424:0517/160906.358:ERROR:find_request_manager.cc(800)] FindRequestManager::FinalUpdateReceived All find matches in

@rzhao271
Copy link
Contributor

I looked more into two scenarios:

  1. Searching in a webview, with the code left as-is.
  2. Searching in an iframe, with the focus-changing code in TextFinder commented out .

During the initial search, when a user types their first character into the textbox, both scenarios seem to proceed in the same execution order.

However, when the user types a second character into the textbox, the behaviours differ.

Scenario 1

In scenario 1, new_session is true, and there is a focused frame, so FindInPage::FindInternal is called on the first pass:
https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/core/frame/find_in_page.cc;l=103

This calls into TextFinder::FindInternal, which then sets the kStartInSelection flag in find_options:
https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/core/editing/finder/text_finder.cc;l=191

The kStartInSelection flag is specifically what prevents the active match from jumping around when a user types more characters.

Scenario 2
In scenario 2, new_session is true, but because the focus code is commented out, when the user types the second character into the textbox, FindInPage::FindInternal is not called on the first pass.

Because it is not called on the first pass, there is no active match found, and the FindRequestManager eventually sends another call to FindInPage::Find with new_session = false:
https://source.chromium.org/chromium/chromium/src/+/main:content/browser/find_request_manager.cc;l=772
https://source.chromium.org/chromium/chromium/src/+/main:content/browser/find_request_manager.cc;l=812;
https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/core/frame/find_in_page.cc;l=103

Again, this calls into TextFinder::FindInternal, but this time, new_session = false, so the kStartInSelectionflag is not set, and the active match starts jumping around.
https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/core/editing/finder/text_finder.cc;l=191

Therefore, if we want to support searching in an iframe, we:

  1. Comment out the focus-related code, or find a way to focus the iframe without stealing focus from the inputbox: https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/core/editing/finder/text_finder.cc;l=263-267;drc=6531e02635a3f00784aacad5c641cd16532657e5, and if we do comment out the focus-related code,
  2. We might also have to find a way to change https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/core/editing/finder/text_finder.cc;l=191 to distinguish between new_session = false due to the iframe find requiring two SendFindRequests, and new_session = false due to an actual find-next operation.

A hypothesis I have is that focusing the frame is possible when searching in a webview, because a webview has its own WebContents that can be focused on, without stealing focus away from the inputbox.

@rzhao271
Copy link
Contributor

rzhao271 commented May 18, 2021

It looks like we don't have to worry about point 2 above, due to Editor::FindRangeOfString already handling the find-next case accidentally.
Assume we set the kStartInSelection flag and do a find-next operation. Then, that means we are looking for the same search string as before. Because kStartInSelection is set, we look for that string starting at the position where the current active match starts. But, that results in us finding the exact same range, so we hit the following condition which actually issues another search for us: https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/core/editing/editor.cc;l=853;drc=6531e02635a3f00784aacad5c641cd16532657e5.

Therefore, commenting out the frame-focusing lines, and setting kStartInSelection seems to work:
recording (17)

The downsides are:

  1. Maintainability. I feel it's lucky that this solution even works.
  2. Performance. Whereas usually, typing a second character for the webview search results in FindInPage::FindInternal being called on the first pass, now, FindInPage::FindInternal is always being called on the second pass, which is less efficient. Similarly, because kStartInSelection is always true, Editor::FindRangeOfString always ends up throwing out one match and having to find another one for each find-next operation. Edit: actually, one can comment out the line here, it seems: https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/core/frame/find_in_page.cc;l=103;drc=6531e02635a3f00784aacad5c641cd16532657e5 though it probably requires more testing.

@deepak1556
Copy link
Collaborator

Thanks for the debugging info, agree with the maintainability part. This fix depends on an internal detail that could be changed between versions and having to spend time on this again is not useful.

Conversely, based on the webview points in #96307 (comment) can you try this idea where we move the input box to a BrowserView https://github.com/electron/electron/blob/master/docs/api/browser-view.md#browserview and see if the correct behavior is achieved without the above patch. This lets the input container to be isolated in a different webContents compared to the frame that is being searched.

@rzhao271
Copy link
Contributor

I created a PR to get concurrent iframe searches working again: electron/electron#29241

Also, here's a demo showing how if the input container is in a different webContents, searching works again:
recording (18)

@deepak1556
Copy link
Collaborator

deepak1556 commented May 20, 2021

@rzhao271 as next step can you create a prototype PR in vscode that brings back ea8c6b3 but uses BrowserView for the search input. Thanks!

@rzhao271
Copy link
Contributor

This morning, I learned from @bpasero that there's a .setSelectionRange() method for the text input element, as well as .selectionStart and .selectionEnd attributes (https://developer.mozilla.org/en-US/docs/Web/API/HTMLInputElement/setSelectionRange).

I added a setTimeout to the Fiddle, and got it to focus the inputbox and place the cursor at the end of the search string a short while after each find call.

Screencap below:

recording (19)

Even if I were to add a debounce, this approach has the jumping search problem, where alternating the search string between "i" and "in" results in the active match jumping around, rather than staying in one place, due to the behaviour of kStartInSelection documented here: #96307 (comment).

Therefore, I think using a BrowserView would still be the best way forward for searching in iframes in the desktop app.

@bpasero
Copy link
Member

bpasero commented May 26, 2021

Thanks for giving this a try. Can we ship this to users to get some initial feedback and investigate the browser view based solution separately? Maybe get some coverage in insiders to hear back from users how bad this is?

@rzhao271
Copy link
Contributor

I added a debounce, and here's how that looks:

recording (21)

🐛 I typed Ctrl+A in the middle of one of the searches, and got the search frame to focus.

The jumping search problem still occurs (not shown in recording), though it's not as visible now, due to the delay between searching and the search results.

@bpasero
Copy link
Member

bpasero commented May 26, 2021

I like it 👍

@rzhao271
Copy link
Contributor

rzhao271 commented May 26, 2021

I still think this behaviour differs too much from the way the find widget currently works with webviews to be worth implementing.
It's closer to the search viewlet behaviour, but even that doesn't have the problem of focus being stolen away from the searchbox during searches.

@bpasero
Copy link
Member

bpasero commented May 28, 2021

@mjbvz any chance we could get ea8c6b3 into our builds? I have iframe based webviews enabled via settings ("webview.experimental.useIframes": true) and in markdown preview I am currently greeted with this:

image

My rationale is that I would like to selfhost on @rzhao271 changes for enabling find in iframes, but currently that is impossible.

@bpasero
Copy link
Member

bpasero commented Jun 1, 2021

Next steps:

  • @rzhao271 get the branch with the changes up to date
  • @rzhao271 input box debouncing into the prototype to reduce flicker
  • @bpasero investigate shortcomings of window.find
  • merge to main to enable for users with "webview.experimental.useIframes": true

rzhao271 pushed a commit that referenced this issue Jun 7, 2021
rzhao271 pushed a commit that referenced this issue Jun 23, 2021
rzhao271 added a commit that referenced this issue Jun 24, 2021
Add search for iframe based webviews. Fixes #96307

Co-authored-by: Matt Bierner <matb@microsoft.com>
@rzhao271 rzhao271 added the on-release-notes Issue/pull request mentioned in release notes label Jul 2, 2021
@github-actions github-actions bot locked and limited conversation to collaborators Aug 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
electron-12-update Issues and items related to electron 12 update feature-request Request for new features or functionality insiders-released Patch has been released in VS Code Insiders on-release-notes Issue/pull request mentioned in release notes on-testplan sandbox Running VSCode in a node-free environment web Issues related to running VSCode in the web webview Webview issues
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants
@bpasero @deepak1556 @rzhao271 @mjbvz and others