Skip to content

Commit

Permalink
Expand bidi-trie usage in static network filtering engine
Browse files Browse the repository at this point in the history
Related issues:
- uBlockOrigin/uBlock-issues#761
- uBlockOrigin/uBlock-issues#528

The previous bidi-trie code could only hold filters which
are plain pattern, i.e. no wildcard characters, and which
had no origin option (`domain=`), right and/or left anchor,
and no `csp=` option.

Example of filters that could be moved into a bidi-trie
data structure:

    &ad_box_
    /w/d/capu.php?z=$script,third-party
    ||liveonlinetv247.com/images/muvixx-150x50-watch-now-in-hd-play-btn.gif

Examples of filters that could NOT be moved to a bidi-trie:

    -adap.$domain=~l-adap.org
    /tsc.php?*&ses=
    ||ibsrv.net/*forumsponsor$domain=[...]
    @@||imgspice.com/jquery.cookie.js|$script
    ||view.atdmt.com^*/iview/$third-party
    ||postimg.cc/image/$csp=[...]

Ideally the filters above should be able to be moved to a
bidi-trie since they are basically plain patterns, or at
least partially moved to a bidi-trie when there is only a
single wildcard (i.e. made of two plain patterns).

Also, there were two distinct bidi-tries in which
plain-pattern filters can be moved to: one for patterns
without hostname anchoring and another one for patterns
with hostname-anchoring. This was required because the
hostname-anchored patterns have an extra condition which
is outside the bidi-trie knowledge.

This commit expands the number of filters which can be
stored in the bidi-trie, and also remove the need to
use two distinct bidi-tries.

- Added ability to associate a pattern with an integer
  in the bidi-trie [1].
    - The bidi-trie match code passes this externally
      provided integer when calling an externally
      provided method used for testing extra conditions
      that may be present for a plain pattern found to
      be matching in the bidi-trie.

- Decomposed existing filters into smaller logical units:
    - FilterPlainLeftAnchored =>
        FilterPatternPlain +
        FilterAnchorLeft
    - FilterPlainRightAnchored =>
        FilterPatternPlain +
        FilterAnchorRight
    - FilterExactMatch =>
        FilterPatternPlain +
        FilterAnchorLeft +
        FilterAnchorRight
    - FilterPlainHnAnchored =>
        FilterPatternPlain +
        FilterAnchorHn
    - FilterWildcard1 =>
        FilterPatternPlain + [
          FilterPatternLeft or
          FilterPatternRight
        ]
    - FilterWildcard1HnAnchored =>
        FilterPatternPlain + [
          FilterPatternLeft or
          FilterPatternRight
        ] +
        FilterAnchorHn
    - FilterGenericHnAnchored =>
        FilterPatternGeneric +
        FilterAnchorHn
    - FilterGenericHnAndRightAnchored =>
        FilterPatternGeneric +
        FilterAnchorRight +
        FilterAnchorHn
    - FilterOriginMixedSet =>
        FilterOriginMissSet +
        FilterOriginHitSet
    - Instances of FilterOrigin[...], FilterDataHolder
      can also be added to a composite filter to
      represent `domain=` and `csp=` options.

- Added a new filter class, FilterComposite, for
  filters which are a combination of two or more
  logical units. A FilterComposite instance is a
  match when *all* filters composing it are a
  match.

Since filters are now encoded into combination of
smaller units, it becomes possible to extract the
FilterPatternPlain component and store it in the
bidi-trie, and use the integer as a handle for the
remaining extra conditions, if any.

Since a single pattern in the bidi-trie may be a
component for different filters, the associated
integer points to a sequence of extra conditions,
and a match occurs as soon as one of the extra
conditions (which may itself be a sequence of
conditions) is fulfilled.

Decomposing filters which are currently single
instance into sequences of smaller logical filters
means increasing the storage and CPU overhead when
evaluating such filters. The CPU overhead is
compensated by the fact that more filters can now
moved into the bidi-trie, where the first match is
efficiently evaluated. The extra conditions have to
be evaluated if and only if there is a match in the
bidi-trie.

The storage overhead is compensated by the
bidi-trie's intrinsic nature of merging similar
patterns.

Furthermore, the storage overhead is reduced by no
longer using JavaScript array to store collection
of filters (which is what FilterComposite is):
the same technique used in [2] is imported to store
sequences of filters.

A sequence of filters is a sequence of integer pairs
where the first integer is an index to an actual
filter instance stored in a global array of filters
(`filterUnits`), while the second integer is an index
to the next pair in the sequence -- which means all
sequences of filters are encoded in one single array
of integers (`filterSequences` => Uint32Array). As
a result, a sequence of filters can be represented by
one single integer -- an index to the first pair --
regardless of the number of filters in the sequence.

This representation is further leveraged to replace
the use of JavaScript array in FilterBucket [3],
which used a JavaScript array to store collection
of filters. Doing so means there is no more need for
FilterPair [4], which purpose was to be a lightweight
representation when there was only two filters in a
collection.

As a result of the above changes, the map of `token`
(integer)  => filter instance (object) used to
associate tokens to filters or collections of filters
is replaced with a more efficient map of `token`
(integer) to filter unit index (integer) to lookup a
filter object from the global `filterUnits` array.

Another consequence of using one single global
array to store all filter instances means we can reuse
existing instances when a logical filter instance is
parameter-less, which is the case for FilterAnchorLeft,
FilterAnchorRight, FilterAnchorHn, the index to these
single instances is reused where needed.

`urlTokenizer` now stores the character codes of the
scanned URL into a bidi-trie buffer, for reuse when
string matching methods are called.

New method: `tokenHistogram()`, used to generate
histograms of occurrences of token extracted from URLs
in built-in benchmark. The top results of the "miss"
histogram are used as "bad tokens", i.e. tokens to
avoid if possible when compiling filter lists.

All plain pattern strings are now stored in the
bidi-trie memory buffer, regardless of whether they
will be used in the trie proper or not.

Three methods have been added to the bidi-trie to test
stored string against the URL which is also stored in
then bidi-trie.

FilterParser is now instanciated on demand and
released when no longer used.

***

[1] https://github.com/gorhill/uBlock/blob/135a45a878f5b93bc538f822981e3a42b1e9073f/src/js/strie.js#L120
[2] e94024d
[3] https://github.com/gorhill/uBlock/blob/135a45a878f5b93bc538f822981e3a42b1e9073f/src/js/static-net-filtering.js#L1630
[4] https://github.com/gorhill/uBlock/blob/135a45a878f5b93bc538f822981e3a42b1e9073f/src/js/static-net-filtering.js#L1566
  • Loading branch information
gorhill committed Oct 21, 2019
1 parent 9f06985 commit 7971b22
Show file tree
Hide file tree
Showing 12 changed files with 2,415 additions and 1,969 deletions.
4 changes: 2 additions & 2 deletions src/js/background.js
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,8 @@ const µBlock = (( ) => { // jshint ignore:line

// Read-only
systemSettings: {
compiledMagic: 21, // Increase when compiled format changes
selfieMagic: 22, // Increase when selfie format changes
compiledMagic: 23, // Increase when compiled format changes
selfieMagic: 23, // Increase when selfie format changes
},

restoreBackupSettings: {
Expand Down
1 change: 0 additions & 1 deletion src/js/document-blocked.js
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@ let details = {};
(async ( ) => {
const response = await messaging.send('documentBlocked', {
what: 'listsFromNetFilter',
compiledFilter: details.fc,
rawFilter: details.fs,
});
if ( response instanceof Object === false ) { return; }
Expand Down
2 changes: 2 additions & 0 deletions src/js/hntrie.js
Original file line number Diff line number Diff line change
Expand Up @@ -560,6 +560,8 @@ HNTrieContainer.prototype.HNTrieRef = class {
this.container = container;
this.iroot = iroot;
this.size = size;
this.needle = '';
this.last = -1;
}

add(hn) {
Expand Down
3 changes: 0 additions & 3 deletions src/js/logger-ui.js
Original file line number Diff line number Diff line change
Expand Up @@ -659,7 +659,6 @@ const viewPort = (( ) => {
}
if ( filteringType === 'static' ) {
divcl.add('canLookup');
div.setAttribute('data-filter', filter.compiled);
} else if ( filteringType === 'cosmetic' ) {
divcl.add('canLookup');
divcl.toggle('isException', filter.raw.startsWith('#@#'));
Expand Down Expand Up @@ -1465,7 +1464,6 @@ const reloadTab = function(ev) {

const fillSummaryPaneFilterList = async function(rows) {
const rawFilter = targetRow.children[1].textContent;
const compiledFilter = targetRow.getAttribute('data-filter');

const nodeFromFilter = function(filter, lists) {
const fragment = document.createDocumentFragment();
Expand Down Expand Up @@ -1524,7 +1522,6 @@ const reloadTab = function(ev) {
if ( targetRow.classList.contains('networkRealm') ) {
const response = await messaging.send('loggerUI', {
what: 'listsFromNetFilter',
compiledFilter: compiledFilter,
rawFilter: rawFilter,
});
handleResponse(response);
Expand Down
14 changes: 9 additions & 5 deletions src/js/messaging.js
Original file line number Diff line number Diff line change
Expand Up @@ -66,14 +66,18 @@ const onMessage = function(request, sender, callback) {

case 'listsFromNetFilter':
µb.staticFilteringReverseLookup.fromNetFilter(
request.compiledFilter,
request.rawFilter,
callback
);
request.rawFilter
).then(response => {
callback(response);
});
return;

case 'listsFromCosmeticFilter':
µb.staticFilteringReverseLookup.fromCosmeticFilter(request, callback);
µb.staticFilteringReverseLookup.fromCosmeticFilter(
request
).then(response => {
callback(response);
});
return;

case 'reloadAllFilters':
Expand Down
3 changes: 3 additions & 0 deletions src/js/redirect-engine.js
Original file line number Diff line number Diff line change
Expand Up @@ -737,6 +737,9 @@ RedirectEngine.prototype.loadBuiltinResources = function() {
store(name, reader.result);
resolve();
};
reader.onabort = reader.onerror = ( ) => {
resolve();
};
reader.readAsDataURL(blob);
});
};
Expand Down
44 changes: 25 additions & 19 deletions src/js/reverselookup.js
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,9 @@ let messageId = 1;

const onWorkerMessage = function(e) {
const msg = e.data;
const callback = pendingResponses.get(msg.id);
const resolver = pendingResponses.get(msg.id);
pendingResponses.delete(msg.id);
callback(msg.response);
resolver(msg.response);
};

/******************************************************************************/
Expand All @@ -55,6 +55,9 @@ const stopWorker = function() {
worker.terminate();
worker = null;
needLists = true;
for ( const resolver of pendingResponses.values() ) {
resolver();
}
pendingResponses.clear();
};

Expand Down Expand Up @@ -127,44 +130,42 @@ const initWorker = function() {

/******************************************************************************/

const fromNetFilter = async function(compiledFilter, rawFilter, callback) {
if ( typeof callback !== 'function' ) {
return;
}
const fromNetFilter = async function(rawFilter) {
if ( typeof rawFilter !== 'string' || rawFilter === '' ) { return; }

if ( compiledFilter === '' || rawFilter === '' ) {
callback();
const µb = µBlock;
const writer = new µb.CompiledLineIO.Writer();
if ( µb.staticNetFilteringEngine.compile(rawFilter, writer) === false ) {
return;
}

await initWorker();

const id = messageId++;
const message = {
worker.postMessage({
what: 'fromNetFilter',
id: id,
compiledFilter: compiledFilter,
compiledFilter: writer.last(),
rawFilter: rawFilter
};
pendingResponses.set(id, callback);
worker.postMessage(message);
});

return new Promise(resolve => {
pendingResponses.set(id, resolve);
});
};

/******************************************************************************/

const fromCosmeticFilter = async function(details, callback) {
if ( typeof callback !== 'function' ) { return; }

if ( details.rawFilter === '' ) {
callback();
const fromCosmeticFilter = async function(details) {
if ( typeof details.rawFilter !== 'string' || details.rawFilter === '' ) {
return;
}

await initWorker();

const id = messageId++;
const hostname = µBlock.URI.hostnameFromURI(details.url);
pendingResponses.set(id, callback);

worker.postMessage({
what: 'fromCosmeticFilter',
id: id,
Expand All @@ -182,6 +183,11 @@ const fromCosmeticFilter = async function(details, callback) {
) === 2,
rawFilter: details.rawFilter
});

return new Promise(resolve => {
pendingResponses.set(id, resolve);
});

};

/******************************************************************************/
Expand Down
Loading

6 comments on commit 7971b22

@uBlock-user
Copy link
Contributor

@uBlock-user uBlock-user commented on 7971b22 Feb 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gorhill After this commit, a filter like *image_widget_c4w274ug7kv01.png is ignored by uBlock Origin.

[rdk@on uBlock-git]$ git bisect bad && ./tools/make-firefox.sh 7971b223855d3afae2ca57565f27da0631a1b045 is the first bad commit commit 7971b223855d3afae2ca57565f27da0631a1b045 Author: Raymond Hill rhill@raymondhill.net Date: Mon Oct 21 08:15:58 2019 -0400

Expand bidi-trie usage in static network filtering engine

Related issues:...

Broken in 1.24 (1.22.5rc5) https://github.com/gorhill/uBlock/commit/7971b223855d3afae2ca57565f27da0631a1b045 Huge commit.

found by @gwarser

Discussion src - https://www.reddit.com/r/uBlockOrigin/comments/ex8ggg/need_help_with_a_reddit_filter/

@gorhill
Copy link
Owner Author

@gorhill gorhill commented on 7971b22 Feb 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any image_widget reported in the logger at https://www.reddit.com/r/nba/.

If there is an issue, I need a formal issue created along with a reproducible case.

@gorhill
Copy link
Owner Author

@gorhill gorhill commented on 7971b22 Feb 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found another image on the page and I can reproduce with *communityIcon_1podsfdai4301.png -- the logo of the page.

@uBlock-user
Copy link
Contributor

@uBlock-user uBlock-user commented on 7971b22 Feb 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any image_widget reported in the logger at https://www.reddit.com/r/nba/.

Try at https://new.reddit.com/r/nba/

The widget is only present in new reddit design.

@gorhill
Copy link
Owner Author

@gorhill gorhill commented on 7971b22 Feb 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I have identified and fixed the issue. I need to assess the severity to decide whether this must be an emergency fix.

@gorhill
Copy link
Owner Author

@gorhill gorhill commented on 7971b22 Feb 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see no filters affected in EasyList, EasyPrivacy -- searching with /^(\|\|?)?\*[0-9a-z%]{2,}[^0-9a-z%$]+[0-9a-z%]{2,}/. I see two filters affected in uBlock filters:

||*ontent.streamp1ay.me^$all
*pre-roll$media,redirect=noopmp4-1s,domain=camwhores.tv,important

Given how trivial the fix is, I lean toward an emergency fix since the next release is not going to happen soon.

Please sign in to comment.