Shut off Open Search additions to "Other search engines" by default

[posita]

Test plan

1. Fresh profile
2. Visit brave://settings/search
3. Verify that the new Index other search engines preference is showing and is turned OFF by default
4. Visit a few sites (amazon.com, homedepot.com bestbuy.com) and search
5. Verify they DO NOT show up under Other search engines on brave://settings/searchEngines
6. Visit brave://settings/search
7. Enable the Index other search engines setting
8. Visit sites from step 4
9. Verify the sites DO show up under Other search engines on brave://settings/searchEngines

Description

As I browse, additional entries are silently inserted into Other search engines periodically. There is no notice that I can see. As far as I can tell, I have no opportunity to consent to this settings manipulation. As a user, even after reasonable inquiry, I have no idea that this is happening, much less how, or how to stop it.

Users should be empowered to turn this off, and I would argue it should be off by default. I should not have to manually and continually poll brave://settings/searchEngines to "discover" new or reappearing entires only to have to manually remove (and re-remove) them.

---

Also originally reported with this issue: Despite_ me deleting them every single time I notice their presence, Bing and Google are re-added to my Default search engines list. I haven't done enough testing to figure out what triggers this (i.e., whether it's upgrading Brave, relaunching it, or navigating to certain pages, etc.), but it keeps happening. This is disturbing, since I've signaled an intent that they should not appear. They should only come back if I signal intent to restore the default settings or add them explicitly. UPDATE: I will file a separate issue to track this when I observe it again and can provide more details.

[diracdeltas]

cc @bsclifton - not sure how much if any of this is intentional

[posita]

Following up with some additional investigation. I recently visited Home Depot's website and discovered that visit resulted in this addition to my search engines:

There are six characters after the ? in the URL which I have redacted. I don't know whether they are a tracking ID or what their purpose is. Due to the width of my screen, those six characters were not visible to me in the brave://settings/searchEngines screen. I had to click on the vertical ellipsis to expose the full URL (and capture the attached screenshot).

This is disconcerting, since it's not super crazy for me to type something like site.dom coupon code or similar into my nav bar. If I were to try that with homedepot.com after this clandestine addition, I would not only be directed to searching their site (which would be annoying and confusing), but I also may be being tracked in the process (which I would emphatically not want).

I'm making some assumptions here, but it doesn't seem like it would take much for an attacker to silently install tracking URLs with keywords that coincided with common search terms (e.g., where, why, etc.) and start hijacking a lot of queries. Something like this:

[posita]

… - not sure how much if any of this is intentional

I'm not sure intention matters here. This seems like an attack surface that needs to go away. At the very least, it needs to involve user consent before modifying state. In addition, I would strongly advocate for users to be able to turn this off entirely (and have that be the default).

Although, I may have misunderstood your comment. Are you saying this isn't behavior intended by Brave developers (and may be inherited from the underlying Chromium implementation)?

[bsclifton]

@posita (RE: your second question) this behavior is inherited from Chromium - to be clear, it's not something we added. It's Chromium's search integration using the Open Search spec. You'll notice the same behavior in Chromium or Chrome

Any web page (home depot, etc) that exposes a search will be detected and add itself under the Other search engines. The intention is that users can find one they like and then easily make it the default without having to click Add and put together the URL themselves (with the %s in it)

You bring up a good point - this behavior should be optional. We can absolutely put a toggle around this behavior and stop this from happening when disabled. I can help you reword the original top post, if that captures your concern? Please let us know 😄

@rebron @diracdeltas if this was a toggle, what would you think about having this disabled by default?

[bsclifton]

@posita quick follow up on the first question (apologies I missed that). With the Default search engines list, this is rebuilt on occasion. This behavior is also inherited from Chromium - I'd need to check the logic to be extra sure, but I believe it will rebuild these entries when a new profile is created OR when the list is updated (ex: new entry added). However, I didn't realize that it's adding it back - that is definitely a bug. If you can confirm that I can make a separate issue to track that

[posita]

… However, I didn't realize that it's adding it back - that is definitely a bug. If you can confirm that I can make a separate issue to track that

The truth is that it's something I've observed, but not carefully, so I'm not super confident about when it it occurs. Let me watch out for it, and if I see it again, and I know it's not one of the cases you mentioned, I'll file a separate issue to track that.

[posita]

…

You bring up a good point - this behavior should be optional. We can absolutely put a toggle around this behavior and stop this from happening when disabled. I can help you reword the original top post, if that captures your concern? Please let us know 😄

@rebron @diracdeltas if this was a toggle, what would you think about having this disabled by default?

@bsclifton, I tweaked the title and the original top post. Let me know if it concisely captures what you'd like. I can edit further, if desired.

It would be nice if this feature was on, that a user could at least know when these sites are added, but I think being able to shut this off and have that be the default is a more impactful first step.

[diracdeltas]

@rebron @diracdeltas if this was a toggle, what would you think about having this disabled by default?

@bsclifton that seems fine to me. i've only used this feature once and my sense is that most users don't know it exists.

[bsclifton]

Original post looks great, @posita 😄 Thanks for taking the time to explain clearly and update

I marked this with privacy label as this also gives clues to browsing history (I believe these persist even after clearing history). Marked as P3 but we can make a P2 if needed (cc: @rebron)

If we take this on, we'll want to update https://github.com/brave/brave-browser/wiki/Deviations-from-Chromium-(features-we-disable-or-remove) to make sure and include this

[stephendonner]

Verified PASSED using

Brave	1.30.44 Chromium: 93.0.4577.51 (Official Build) nightly (x86_64)
Revision	762d21050e2da59930c784c09b134d0b0b148188-refs/branch-heads/4577@{#915}
OS	macOS Version 11.5.2 (Build 20G95)

Steps:

1. clean profile
2. loaded brave://settings/search
3. verified that the new Index other search engines preference is showing and is turned OFF by default
4. visited a few sites (amazon.com, homedepot.com, bestbuy.com) and searched
5. verified they DO NOT show up under Other search engines on brave://settings/searchEngines
6. visited brave://settings/search
7. enabled the Index other search engines setting
8. visited sites from step 4
9. verified now the sites DO show up under Other search engines on brave://settings/searchEngines

Steps 1-5	Steps 6-9

[Madis0]

You might want to mark #867 as fixed as well, which is the issue that initially requested this.

[bsclifton]

@Madis0 thanks! Closed the other as a duplicate 😄

[UjCbFwtBayFM]

As someone who does fresh installs on a regular basis this is a terrible change, how do you switch back it on by default?

[posita]

As someone who does fresh installs on a regular basis this is a terrible change, how do you switch back it on by default?

Does "do[ing] fresh installs" mean that you are wiping settings as well? Or are you saying that this setting is not being preserved for you between updates/installs?

You can switch it on manually if you really like the feature, but it doesn't sound like that's what you want? It sounds like what you want is the old behavior (i.e., allow arbitrary sites to install arbitrary search engines without notice to the user). I'm pretty confident that behavior is antithetical to prioritizing privacy and security, so I think it's appropriate to have it off by default.

[UjCbFwtBayFM]

Does "do[ing] fresh installs" mean that you are wiping settings as well? Or are you saying that this setting is not being preserved for you between updates/installs?

Do you have any other definition for an install with no setting/config file kept?
On top of that the setting isn't persistent between launches. Having to re-add every single non default search engine (especially when you use quite a lot of them) and having Brave Search force-set as default SE every time you boot up Brave is annoying, to put it politely.
A better solution would have been to put a prompt+warning every time a site was indexed.

[posita]

Thanks for the clarifications @UjCbFwtBayFM, but I'm still not sure I understand what you're asking. Are you looking for a way to automate the setting the default/available search engines at install time? Or are you looking specifically to automate enabling the OpenSearch feature, where arbitrary websites (like homedepot.com) can add themselves as keyword-based search engines when users visit their sites?

I guess another way to ask my question is: When you visit brave://settings/searchEngines, which section are you most interested in manipulating at install time? Is it Default search engines or is it Other search engines, or does it matter to you?

[bsclifton]

Hi folks - trying to understand here also

@UjCbFwtBayFM this change is specifically to prevent sites like homedepot.com and bestbuy.com from being added into your Other search engines list brave://settings/searchEngines

I don't know about other folks, but before this change was in place, I had over 200 entries under Other search engines which had the following problems:

• Most were a site I visited and searched one time; like someone's blog or an eCommerce site
• These are considered in the omnibox when typing; there's a small perf hit when doing that
• Someone can look at all the entries under Other search engines and get an idea of the sites I visit, which is a minor privacy issue

@posita I think you captured the question perfectly 😄 Are we talking about Default search engines or Other search engines? also curious for more info about your use-case

The default search engines are hardcoded, based on region. If you wipe your profile often (ex: deleting the %USERPROFILE%/AppData/Local/BraveSoftware/Brave-Browser folder), you get reset back to the default unfortunately. If you are changing the default and it's not saving, that's a bug! I'd love to have more information to help find and fix this one. Do you have any Clear on exit settings set? (under brave://settings/clearBrowserData)

If you're working across a bunch of machines (and that's why you're having a new profile), I'd encourage using the Sync feature as this WILL sync your default search engine

[UjCbFwtBayFM]

I already said it but I guess I need to repeat it: first I want an option to have it enabled when I install or update it, or at bare minimum have a prompt so I can choose if I want website to be able to use this function. If you're able to have user permission prompt for things like location and stuff, you can do the same for OpenSearch.
Second I want it to stay enabled. I don't want to have to re-add every single search engine I use every single time I open Brave. Try adding manually 30+ engines multiple times per week, monitor how much time you'll waste on that. I'd also love for Brave Search to not be set as default engine at every start-up
Or should I go back to Chrome?

[posita]

Sorry @UjCbFwtBayFM, I'm still confused about the nature of the problem you're trying to solve, but I'd like to understand. It might be easier for folks like @bsclifton and me to understand if we had more details about your situation. How often are you doing fresh installs? I'm curious why that's happening. Perhaps we can help make that less likely?

I don't want to have to re-add every single search engine I use every single time I open Brave. … I'd also love for Brave Search to not be set as default engine at every start-up.

The OpenSearch entries aren't the same as the set of search engines that can be selected as the default. Many browsers set their own defaults. It sounds like that was always the case. What did you do before to set the default search engine? (This fix shouldn't have affected that.)

Or are you saying that you regularly install from scratch weekly and the way you populate the 30+ search engines you use is by surfing to those sites after installing? If I've described that accurately, I'd definitely look into Brave's sync feature as @bsclifton suggests. That will help you out a lot. It will also help address all the other settings you typically have to work through for each installation (e.g., extensions, bookmarks, privacy settings, etc.).

It sounds like we may be conflating several different issues at this point. @bsclifton, should @UjCbFwtBayFM open a new issue to move the discussion there? Or is this still a good place for it?

[bsclifton]

We definitely have a few different issues; thanks for explaining @UjCbFwtBayFM

Setting not staying enabled / default being reset

Basically, you shouldn't need to add 30+ engines multiple times per week. Or you shouldn't need to change your default search engine again. These settings are persisted to your profile. Same with the open search setting. If this is not being saved, we have a problem.

Do you have Brave set to clear on exit? Or are you creating new profiles? More information about your setup is appreciated. This is NOT expected behavior and I'd like to help diagnose so we can understand and attempt a fix.

Having an infobar or similar to present the open search choice to user

This is a good piece of feedback. Just like we show location prompt, etc.
cc: @rebron

Being able to set the open search preference on install

Are you installing Brave in an enterprise environment? We could expose a group policy setting for this new open search setting- so that folks in an enterprise setting can have the expected default. I have a blanket issue for group policy settings here - I'll make a note for this new setting also:
#9458

[UjCbFwtBayFM]

Or are you saying that you regularly install from scratch weekly and the way you populate the 30+ search engines you use is by surfing to those sites after installing? If I've described that accurately, I'd definitely look into Brave's sync feature as @bsclifton suggests. That will help you out a lot. It will also help address all the other settings you typically have to work through for each installation (e.g., extensions, bookmarks, privacy settings, etc.).

I don't use Sync. I tried it once, but it didn't (and I think it still doesn't) include Rewards and didn't work properly (didn't sync things that were supposed to be synced and adding unknown devices to it over time).

[posita]

I tried it once, but it didn't (and I think it still doesn't) include Rewards and didn't work properly (didn't sync things that were supposed to be synced and adding unknown devices to it over time).

If you haven't already, you might consider filing those as separate issues. It sounds like if those were addressed, it would largely get you where you need to go? Not sure. This conversation feels a bit like a moving target, although it sounds like @bsclifton is trying his best to translate from your needs as a user to the specific things that might need to be addressed on the application side.

[UjCbFwtBayFM]

If you haven't already, you might consider filing those as separate issues. It sounds like if those were addressed, it would largely get you where you need to go? Not sure. This conversation feels a bit like a moving target, although it sounds like @bsclifton is trying his best to translate from your needs as a user to the specific things that might need to be addressed on the application side.

I guess I could, but it'd loose the exposure they have here.

[AJolly]

Is there a group policy or registry setting I can set to automatically keep this on? Or better yet, to add a default search (kagi) to my browser?