Contributing

First of all, thank you for taking the time!

The easiest way you can get involved is by using awspx and reporting bugs you encounter through the Github issue tracker. But if you'd like to do a bit more than that there are a couple of other ways you can help out:

Feature suggestions

Chances are, if you've had an idea for something that would be really useful to you, it would probably be useful to many others. Submitting feature suggestions through the Github issue tracker is a great way of exposing ideas to the community. Not only does this help with prioritization, but it provides others with an opportunity to do something where they might be short of their own ideas. We don't have a template just yet but ask that you are detailed as you can be - until we figure out what works and what doesn't.

Code contributions

Of course, if you have the means to, you needn't stop at an idea. Whether its a fix or a feature, submitting a Pull Request (PR) helps in a big way. If you're keen to get your hands dirty, but aren't sure how, checkout some of the issues others have opened. Again, we don't have a template for PRs but ask that you set the target branch to develop.

Anything else you can think of

This project is in its early days and there's still plenty that can be done. Whether its submitting a fix, identifying bugs, suggesting enhancements, creating or updating documentation, refactoring smell code, or even extending this list — all contributions help and are more than welcome. Please feel free to use your judgement and do whatever you think would benefit the community most.

Background information

Documenting everything would be incredibly tedious, so we're not going to. Instead, we'll start by just providing some neccessary background information for obvious extensions requiring further explanation. We'll update this section as and when we need to, in the interim feel free to post questions using the Github issue tracker.

How to add or extend service ingestors

Depending on the service you want to add support for, you may need to write a Resources-based ingestor, or you may need to write a custom ingestor using the familiar boto3 get, list, describe syntax.

How to add a new service with boto3 Resources

The base Ingestor class uses boto3's Resources interface to ingest resources and infer relationships between them. Currently, this interface is implemented for the following services (awspx support has been added for a few of them already):

• cloudformation
• cloudwatch
• dynamodb
• ec2
• glacier
• iam
• opsworks
• s3
• sns
• sqs

Presently, only awspx's EC2 and S3 ingestors use this architecture. If you would like to add one of the other services in this list (for example, CloudFormation), you can start with the following template class declaration, placing it in lib/aws/ingestor.py:

class CloudFormation:
    pass

To test your ingestor, run awspx ingest with the following arguments:

awspx ingest --services CloudFormation

It is likely that the first run will crash with an error. These errors usually result from the ingestor failing to determine a Name or Arn property for one or more resources in the service. With verbose output (-v), you will be able to see the structure of the data the ingestor fails on. Using this output, you will need to amend the methods _get_resource_arn and _get_resource_name in the Ingestor base class. Continue until all ingested resources have both a Name and an Arn.

Once fixed, you can start considering which resources you want to ingest and which relationships are worth showing. These two aspects of ingestion are represented by run and associates respectively.

An ingestor's run attribute is a list of resources it will ingest. An empty run attribute, however, indicates that everything should be ingested. These values are represented identically to resource names in lib/aws/resources.py. They will be converted to the correct boto method automatically.

An ingestor's associates attribute is a list of tuples, representing associative relationships between resource. The ingestor will infer relationships between resources based on boto3's Resource model, e.g. if an instance object has an attribute called snapshots, there is a relationship between instances and snapshots. If associates is not defined, all inferred relationships will be mapped. Thus, associates is a good way to prune excessive edges. See the EC2 ingestor for a good example.

Adding a new service without boto3 Resources

When adding services not supported by boto3 Resources, you will have to fall back to using boto3 service clients, which are thin wrappers over AWS CLI commands. You will need to manually discover the different resource types and their associations. The Lambda service ingestor is an example of such an ingestor.

Enriching services

While the above methods will ingest a large amount of data about resources and their relationships, many AWS services require additional enrichment for us to get all the information that's useful for mapping out the environment and drawing attack paths. For these cases, we need to do some enrichment.

The S3 ingestor provides a good example of this. Because bucket policies and ACLs are not pulled by the standard ingestor logic, we have two additional methods to pull this information for each bucket. Consider the method for getting bucket policies:

def get_bucket_policies(self):

    sr = self.session.resource(self.__class__.__name__.lower())

    for bucket in self.get("AWS::S3::Bucket").get("Resource"):
        try:

            bucket.set("Policy", json.loads(sr.BucketPolicy(
                bucket.get('Name')).policy))
        except:  # no policy for this bucket
            pass

This method uses additional, S3-specific methods to get the policy for each ingested bucket, which are enumerated using methods provided by Ingestor's parent class, Elements (see lib/graph/base.py). The same procedure is followed to get bucket ACLs, and to get instance user data in EC2.