Approximate license detection

[pombredanne]

In addition to exact detection we should add support for approximate detection, as planned in the roadmap.

[pombredanne]

There is a weird heisenbug on Windows tests from Appveyor... cannot reproduce it anywhere else:

rules_tokens_ids = [[32, 8, 110, 108, 104, 10, ...]]
dictionary = {'1': 0, '2': 1, '3': 2, '4': 3, ...}
tokens_by_tid = ('1', '2', '3', '4', '5', 'a', ...)
frequencies_by_tid = (1, 1, 1, 1, 1, 3, ...), length = 9, with_checks = True

    def renumber_token_ids(rules_tokens_ids, dictionary, tokens_by_tid, frequencies_by_tid, length=9, with_checks=True):
        """
        Return updated index structures with new token ids such that the most common
        aka. 'junk' tokens have the lowest ids.

        `rules_tokens_ids` is a mapping of rule_id->sequence of token ids

        These common tokens are based on a curated list of frequent words and
        further refined such that:
         - no rule text sequence is composed entirely of these common tokens.
         - no or only a few rule text sub-sequence of `length` tokens (aka.
         ngrams) is not composed entirely of these common tokens.

        The returned structures are:
        - old_to_new: mapping of (old token id->new token id)
        - len_junk: the highest id of a junk token
        - dictionary (token string->token id)
        - tokens_by_tid (token id->token string)
        - frequencies_by_tid (token id->frequency)
        """
        # keep track of very common junk tokens: digits and single letters
        very_common = set()
        for tid, token in enumerate(tokens_by_tid):
            # DIGIT TOKENS: Treat tokens composed only of digits as common junk
            # SINGLE ASCII LETTER TOKENS: Treat single ASCII letter tokens as common junk

            # TODO: ensure common numbers as strings are always there (one, two, and first, second, etc.)
>           if token.isdigit() or (len(token) == 1 and token in string.lowercase):
E           UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 26: ordinal not in range(12

[pombredanne]

The issue for Windows @ appveyor is likely due to default encoding: https://www.google.com/search?q=sys.getdefaultencoding%20problem%20on%20windows

[pombredanne]

This is all merged in develop now. And I am closing