Enhancement: Postman improve scope validation

[domwhewell-sage]

This PR adds some validation to the postman download module that way it checks if the workspace, description, environment variables, collection requests contain any in-scope urls, dns names or email address's

And if not it will ignore the code repository event

I have removed the if statement from the postman search module as the download module will verify if it is in scope or not.

[TheTechromancer]

Nice work on this.

I like the idea of validating on based on hostname, but I'm wondering if it might be faster and more effective, instead of iterating through the JSON object, to search the postman response with the yara rule we already have for extracting in-scope hostnames. This would be quite a bit faster and would also catch urls/emails embedded in xml blobs or other misc data.

I feel bad bringing this up after you wrote all this code. Let me know what you think and I'll see if we have a helper extracting in-scope hostnames with yara.

[domwhewell-sage]

Oh yes that sounds good!

[domwhewell-sage]

I don't see any helpers extracting hostnames with Yara, but there is a regex one for emails.

Would the one for extracting hostnames be able to accept the full response as a string and yield DNS names to iterate over?

[TheTechromancer]

Yes sorry for the delay, I'm working on one and it should be finished soon.

[TheTechromancer]

@domwhewell-sage the helper is merged:

    async def extract_in_scope_hostnames(self, s):
        """
        Given a string, uses yara to extract hostnames matching scan targets
        Examples:
            >>> await self.scan.extract_in_scope_hostnames("http://www.evilcorp.com")
            ... {"www.evilcorp.com"}
        """

[domwhewell-sage]

Ok I've added a for loop to loop over extracted emails and hostnames from workspaces, collections and environments which will discard CODE_REPOSITORY events if they are not in scope or download them if they are.

I have added a workspace to the test that should be detected as out-of-scope (As it does not contain any hostnames/emails related to the target)