Adding custom threat lists to database

[TheFrozenFire]

I am integrating gglsbl as a backend for checking of URLs that are sent in our customers' messaging. This library has been extremely helpful in doing so - props for that.

A question I have is whether anyone has attempted to integrate data from other sources into their database. For instance, we will be checking URLs against feeds such as OpenPhish, and industry-specific feeds we've gained access to.

It's not super obvious from the code whether the functions for converting a URL to an entry which can be added to the database are present, though it seems like it might be possible using the function to get the hashes of a URL, plus the functions to add threat entries to the database.

One area of confusion for me is how I might compute "threat prefixes". Is that just intended to be the first four hash values of a URL's hash list?

[afilipovich]

This library is compatible only with Google Safe Browsing API data source.
URLs are transformed into hashes and hash prefixes as described in this spec:
https://developers.google.com/safe-browsing/v4/urls-hashing

Conceptually it is possible to add support for other URL blacklist providers, but it would require a major overhaul. In fact it would be easier to make a separate library for other feeds as they list URLs in clear text while Google provides only irreversible hashes which require extra transformations and lookups.

[TheFrozenFire]

My intention would be to generate the same sort of hashes that Google does, and then add those to the database. I'm not looking to add support for other feed providers to the library itself, but rather to add some helper functionality for generating the same hash format as Google does, and for associating those hashes with threat lists.

[afilipovich]

Gotcha. You can use this class to translate URL to hash: https://github.com/afilipovich/gglsbl/blob/master/gglsbl/protocol.py#L168