Default filter lists unselected internally (i.e. not unselected by user)

[parkerjeremy44]

Prerequisites

• I read and understand the policy about what is a valid filter issue.
• I verified that this issue is not a duplicate. (Use this button to find out.)
• I forced an update of my filter lists. (Click the "Purge all caches" button while holding the 'Shift' key, then click the "Update now" button.)
• I did not remove any of the default filter lists, or I have verified that the issue was not caused by removing any of the default lists.
• I did not enable external filter lists, or I have verified that the issue still occurs without enabling external filter lists.
• I do not have custom filters/rules, or I have verified that the issue still occurs without custom filters/rules.
• I am not using uBlock Origin (uBO) along with other content blocker extensions.
• I have verified that the web browser's built-in blocker or DNS blocking (standalone or through a VPN) is not causing the issue.
• I did not answer truthfully to ALL the above checkpoints.

URL address of the web page

https://old.reddit.com/

Category

ads

Description

Ads are showing up now when they didn't before

Screenshot(s)

Screenshot(s)

Configuration

uBlock Origin: 1.49.0
Chromium: 112
filterset (summary): 
 network: 0
 cosmetic: 0
 scriptlet: 0
 html: 0
listset (total-discarded, last-updated): [none]
filterset (user): [array of 1 redacted]
trustedset: 
 added: [array of 68 redacted]
 removed: 
  edge-scheme
  wyciwyg-scheme
switchRuleset: 
 added: [array of 1 redacted]
modifiedUserSettings: [none]
modifiedHiddenSettings: [none]
supportStats: 
 allReadyAfter: 621 ms (selfie)
 maxAssetCacheWait: 4 ms
popupPanel: 
 blocked: 0

[gorhill]

You have no filter lists enabled:

• Restart the browser

Go to Filter lists pane. If it shows there are no filter lists enabled:

• Click Purge all caches
• Click Update now

If there are still no filter lists enabled, remove then reinstall uBO.

[gorhill]

There has been a bit too many reports of all filter lists being unselected for my taste.

It does seem something is going on, but whatever it is, it's probably very difficult to reproduce.

A new code path was added in 1.49.0 in order to unselect default lists which are no longer default when assets.json updates, but reading carefully through the code, I can't see how current default lists could be all unselected.

In any case, I've added more checks for unexpected conditions when assets.json updates, such that uBO will skip checking which default lists should be disabled when uBO detects one of those abnormal conditions: gorhill/uBlock@6294829

It's a shot in the dark, but given the severity of all lists being unselected, I will give it a try and publish an emergency release for this change.

[gorhill]

Cases beside the one reported here:

• 30/04: youtube.com: ads uAssets#17887
• 30/04: hao123.com: ads uAssets#17882
• 27/04: coolmathgames.com: ads uAssets#17813
• 27/04: youtube.com: ads uAssets#17808
• 27/04: https://www.reddit.com/r/uBlockOrigin/comments/1305guk/
• 26/04: 9anime.to: ads uAssets#17795
• 26/04: youtube.com: ads uAssets#17788
• 25/04: get.pixelexperience.org: ads uAssets#17784
• 25/04: https://www.reddit.com/r/uBlockOrigin/comments/12ys9e4/
• 25/04: youtube.com: ads uAssets#17781
• 25/04: youtube.com: ads uAssets#17778
• 25/04: youtube.com: ads uAssets#17777
• 25/04: https://www.reddit.com/r/uBlockOrigin/comments/12yabg9/ (probably)
• 24/04: wifi4games.com: ads uAssets#17770
• 24/04: youtube.com: detection uAssets#17763
• 23/04: https://forums.opera.com/topic/33598/ublock-origin/249 (probably)
• 23/04: https://www.reddit.com/r/uBlockOrigin/comments/12wlv2a/ (probably)
• 23/04: youtube.com: ads uAssets#17750
• 23/04: uBlock not working on any websites #2608
• 23/04: lowendtalk.com: breakage uAssets#17743
• 23/04: youtube.com: ads uAssets#17741
• 22/04: https://forums.opera.com/topic/33598/ublock-origin/248 (probably)
• 22/04: 9anime.to: ads uAssets#17730
• 22/04: youtube.com: ads uAssets#17724
• 21/04: thingiverse.com: detection uAssets#17704
• 20/04: https://forums.opera.com/topic/33598/ublock-origin/247 (probably)
• 20/04: youtube.com: ads uAssets#17701
• 19/04: https://www.reddit.com/r/uBlockOrigin/comments/12rv41x/

---

Given that uBO is used by tens of millions of people, this doesn't seem too worrying, as if this was an easy to reproduce issue, we would have been flooded by cases by now (I estimate over 3 millions users have their assets.json updated on any given day). But I feel something else than corrupted storage is going on since all other settings seem to have been properly preserved, except for the fact that all filter lists have been unselected in the reported cases.

[stephenhawk8054]

Can browser update affect extension's storage? The day the reports began to show, 2 of my chromium browsers (vanilla chromium and brave) started to update new version too.

[gorhill]

I can't tell really. That's the problem with those reports of broken uBO, it's quite rare given the user base, and when it's reported there is not much to work with, for example: #2341.

[stephenhawk8054]

So if I understand correctly, gorhill/uBlock@6294829 is for checking if default lists being turned off when launching? Can uBO auto force update in that case?

[gorhill]

That code is executed only when assets.json updates. It was originally added in this commit.

[ghajini]

sorry if iam making noise
isn't this same as #2136 except different browser ...

also user selected settings are also lost

eg i enabled these, but they are getting unselected when this happens

ignore generic cosmetic filters
block csp reports
iam an advanced user

[Wouter0100]

I've had another case myself, all of a sudden started w/o any changes. Purge all caches + update now + apply changes fixed it.

[gorhill]

I might have found the most likely explanation for the reported cases. It would have to do with corrupted storage, hence why it's rare. When I mimick corrupted storage to find out what uBO ends up doing, I do get zero filter lists loaded, and troubleshooting information looks like (I removed unimportant entries):

uBlock Origin: 1.49.1b1
Chromium: 112
filterset (summary): 
 network: 0
 cosmetic: 0
 scriptlet: 0
 html: 0
listset (total-discarded, last-updated): [none]
supportStats: 
 allReadyAfter: 115 ms
 maxAssetCacheWait: 4 ms

Especially, the section listset for reported cases is atypical, as it does not show that default lists were removed, which would be the case if the default lists were unselected.

The storage-reading code in uBO is ready to deal with not receiving the fetched items from storage, the issue is that uBO was not dealing with Promise rejection, which would cause the code paths dealing with failure to fetch items from storage to not be taken.

[u-RraaLL]

How feasible would it be to add a check for unselected (or not loaded at all) default filter lists in uBO's report widget that would advise users to purge and update lists before they proceed? And perhaps do the same with lists that haven't been updated in a while?

This could potentially reach a bigger percentage of uBO's userbase than the ones that have/are willing to make a github account.

[gorhill]

Being discussed: #2582

[parkerjeremy44]

You have no filter lists enabled:

• Restart the browser

Go to Filter lists pane. If it shows there are no filter lists enabled:

• Click Purge all caches
• Click Update now

If there are still no filter lists enabled, remove then reinstall uBO.

This worked! Thanks for your help.

[gorhill]

This worked! Thanks for your help.

Please report here if it ever happens again -- currently I suspect this just happens to people who had either uBO or their browser updated. I would like to know if this also happens when neither the browser or uBO updates to a new version.

[catshitz]

This worked! Thanks for your help.

Please report here if it ever happens again -- currently I suspect this just happens to people who had either uBO or their browser updated. I would like to know if this also happens when neither the browser or uBO updates to a new version.

Chromium: 112
filterset (summary): 
 network: 212258
 cosmetic: 206282
 scriptlet: 27143
 html: 0
listset (total-discarded, last-updated): 
 removed: 
  user-filters: null
  plowe-0: null
 added: 
  JPN-1: 11039-13, 2m
  RUS-0: 39724-13, 2m
  adguard-annoyance: 67523-511, 2m
  adguard-mobile: 8692-72, 2m
  adguard-social: 20328-863, 2m
  adguard-spyware-url: 999-3, 2m
  curben-phishing: 79218-2, 2m
  curben-pup: 194-0, 2m
  fanboy-cookiemonster: 46750-3745, 2m
  ublock-annoyances: 4957-202, 2m
 default: 
  CHN-0: 27633-109, 2m
  easylist: 65542-208, 2m
  easyprivacy: 30897-1401, 2m
  ublock-abuse: 70-1, 2m
  ublock-badware: 6867-154, 2m
  ublock-filters: 33990-570, 2m
  ublock-privacy: 322-4, 2m
  ublock-quick-fixes: 216-21, 2m
  ublock-unbreak: 1993-44, 2m
  urlhaus-1: 6769-3, 2m
filterset (user): [empty]
modifiedUserSettings: 
 cloudStorageEnabled: true
 contextMenuEnabled: false
modifiedHiddenSettings: [none]
supportStats: 
 allReadyAfter: 553 ms (selfie)
 maxAssetCacheWait: 22 ms

It happened to me too today, all filters disabled on all sites, clean the cache and re-update the filters can do a temp fix, but it's quite unstable (possibly when you re-open the chrome or restart the system), sometime it'll come back again, I think I may wanna clean re-install the whole extension.

[peace2000]

New case: uBlockOrigin/uAssets#17770 (Don't know if the log the user provided gives useful troubleshooting info, but linking just in case.

[mapx-]

@gorhill

What if once a day / session, uBO checks the status of filter lists and if it finds 0 filters in an enabled list, it tries to update all enabled lists? the same if it finds no list enabled (enable the default lists and update them)

[gorhill]

It's possible to use uBO without filter lists, that would be bad for people using uBO this way to have lists always unilaterally re-enabled.

Best is to get to the bottom of the issue and fix it.

[mapx-]

yes but you can test if there are no dynamic rules

[gorhill]

I don't want to hide symptoms of an issue, that would prevent us from knowing these issues exist and prevent us from investigating them. The cause needs to be identified and fixed. hiding the symptoms won't help toward this. And as usual, just adding more code to fix something is always a risk to cause more issues.

[stephenhawk8054]

When I mimick corrupted storage to find out what uBO ends up doing, I do get zero filter lists loaded

Could you reproduce the same symptom with older main releases (1.48, 1.47, 1.46 ... ), so we might know when it started to have this issue or it has been always there?

I remember we didn't have this many reports when previous versions updated.

[gorhill]

I could not really reproduce the exact same symptoms. What I reproduced left uBO in a broken state, where I could not do what most people are doing. i.e. purge and update all filter lists. The Filter lists pane was completely broken for me.

I looked into the reports quite a lot and they all share two key details: they all report that uBO was loaded from a selfie. I didn't reproduce this. The other one is that they all report that the internal variable which contains all the available filter lists is seemingly empty, not even user filter list is being reported.

When I look at the code, these two things can't happen.

The only explanation I see where this can happen, is that a selfie was created before uBO reached the point where it loads filter lists in launch sequence, and of course this is not supposed to happen.

Again, the only explanation I see for this to happen is that for selfie purpose, uBO started to use browser.alarms API in 1.49.0, and I suspect the API does not behave in ways I expect it to behave, i.e. I suspect that sometimes there is an alarm event fired when uBO is launched -- possibly from a an alarm event that was pending when uBO was exited the previous time. This would cause uBO to create a selfie while no filters are in memory and the available filter lists (including My filters list) have not yet been scanned.

For now I will get rid of alarms API usage.

---

There is an open issue re. alarms API reporting that alarms are not cleared upon extensions update: https://bugs.chromium.org/p/chromium/issues/detail?id=958809 -- this would be a case explaning the reported issues above. Though uBO didn't update after 1.49.0, there are threads out there pointing out that alarms fires erratically sometimes, possibly related to suspend/sleep. Possibly the browser itself updating could lead to the scenario where the alarm fires right at uBO launch.

In any case, beside reverting to avoid using alarms API, I've also added more checks to avoid generating a selfie when there is no filters loaded in memory.

I think I will go ahead and publish 1.49.2 with those changes.

[gorhill]

Similar reports are no longer showing up, and none of the last ones were for 1.49.2, so this probably confirm usage of browser.alarms was at the root of the issue. Closing, I will re-open if new reports for 1.49.2 with the same troubleshooting information signature are showing up.

[xvello]

Heya @gorhill, I'm wondering whether the fix for this issue could have created a regression for custom lists not being refreshed anymore. I see a significant drop in usage since the release of 1.49.2. I don't have enough telemetry to confirm that the drop is really uBO users, but the timing looks pretty suspicious.

I wrote more context in #2659

[gwarser]

If people wander here searching for solution to custom list being disabled in 1.49.2 - gorhill/uBlock@1a9a8aa

[gwarser]

w3c/webextensions#406

MDN states the following, which appears to be true in Firefox and Safari:

Alarms do not persist across browser sessions.

In Chrome, alarms do generally persist. Due to an open bug (https://crbug.com/1285798) they can be lost if the extension is reloaded and then there is a restart before any further changes are made to the registered alarms.