Regex for source in asn.py has a problem

[Neiby]

We've noticed that occasionally we get multiline output for the source field. Instead of only including a source, like RIPE, it also includes the remarks the follow it. I think it's because the ending of the regex is wrong. My guess is that it should end in \n but currently ends in \<.

ASN_ORIGIN_HTTP = {
    'radb': {
        'url': 'http://www.radb.net/query',
        'form_data_asn_field': 'keywords',
        'form_data': {
            'advanced_query': '1',
            'query': 'Query',
            # '-T option': 'inet-rtr',
            'ip_option': '',
            '-i': '1',
            '-i option': 'origin'
        },
        'fields': {
            'description': r'(descr):[^\S\n]+(?P<val>.+?)\n',
            'maintainer': r'(mnt-by):[^\S\n]+(?P<val>.+?)\n',
            'updated': r'(changed):[^\S\n]+(?P<val>.+?)\n',
            'source': r'(source):[^\S\n]+(?P<val>.+?)\<',
        }
    },
}

If I pop that regex as-is into a regex tester using RADB output, it fails to match anything for source. I think a later function assumes that the source is the last thing in the output and just grabs the entire rest of the output, which can include remarks.

Changing that regex to r'(source):[^\S\n]+(?P<val>.+?)\n' seems to fix the problem.

I was testing against output like the following:

route:          147.162.0.0/15
descr:          Aggregated GARR routes
origin:         AS137
remarks:        Universita' di Padova
remarks:        Universita' di Palermo
mnt-by:         GARR-LIR
created:        2002-04-24T11:36:36Z
last-modified:  2013-03-01T09:11:01Z
source:         RIPE
remarks:        ****************************
remarks:        * THIS OBJECT IS MODIFIED
remarks:        * Please note that all data that is generally regarded as personal
remarks:        * data has been removed from this object.
remarks:        * To view the original object, please query the RIPE Database at:
remarks:        * http://www.ripe.net/whois
remarks:        ****************************

You can use this simple test code to verify. After fixing my local copy of asn.py, this code returns what I would expect, but you can try it before fixing the regex to see the problem.

from ipwhois.net import Net
from ipwhois.asn import ASNOrigin

DUMMY_NET = "193.201.40.0"
mynet = Net(DUMMY_NET)
obj = ASNOrigin(mynet)
lookup = obj.lookup('AS137')

for net in lookup['nets']:
    print(net)

It looks like this bug was introduced in this commit:

566dc3e#diff-ab3ec2b1715ed3779d3c1dfd582f725be6c203f1230720f636863dd78269bd83

[secynic]

I didn't see this issue but I pushed a fix to the dev branch to account for both scenarios