Skip to content

Cyber Hygiene Common Vulnerabilities and Exposures (CVE) synchronization Python library

License

Notifications You must be signed in to change notification settings

cisagov/cyhy-cvesync

Folders and files

NameName
Last commit message
Last commit date

Latest commit

fe943de · Oct 29, 2024
Oct 25, 2024
Oct 29, 2024
Oct 23, 2024
Jun 7, 2023
Sep 13, 2023
Oct 23, 2024
May 3, 2019
May 5, 2022
Oct 23, 2024
Feb 25, 2022
Jul 29, 2024
Jun 7, 2019
Nov 9, 2022
Oct 23, 2024
Jun 24, 2019
Oct 29, 2024
Oct 29, 2024
Oct 22, 2024
Feb 5, 2021
Sep 28, 2020
Apr 15, 2021
Oct 18, 2024
Oct 23, 2024
Jan 10, 2020

Repository files navigation

cyhy-cvesync

GitHub Build Status CodeQL Coverage Status Known Vulnerabilities

cyhy-cvesync is Python library that can retrieve JSON files containing Common Vulnerabilities and Exposures (CVE) data (such as those from the National Vulnerability Database (NVD)) and import the data into a MongoDB collection.

Pre-requisites

Starting a Local MongoDB Instance for Testing

Important

This requires Docker to be installed in order for this to work.

You can start a local MongoDB instance in a container with the following command:

pytest -vs --mongo-express

Note

The command pytest -vs --mongo-express not only starts a local MongoDB instance, but also runs all the cyhy-cvesync unit tests, which will create various collections and documents in the database.

Sample output (trimmed to highlight the important parts):

<snip>
MongoDB is accessible at mongodb://mongoadmin:secret@localhost:32881 with database named "test"
Mongo Express is accessible at http://admin:pass@localhost:8081

Press Enter to stop Mongo Express and MongoDB containers...

Based on the example output above, you can access the MongoDB instance at mongodb://mongoadmin:secret@localhost:32881 and the Mongo Express web interface at http://admin:pass@localhost:8081. Note that the MongoDB containers will remain running until you press "Enter" in that terminal.

Example Usage

Once you have a MongoDB instance running, the sample Python code below demonstrates how to initialize the CyHy database, fetch CVE data from a source, and then load the data into to your database.

import asyncio
from cyhy_cvesync import DEFAULT_CVE_URL_PATTERN
from cyhy_cvesync.cve_sync import process_urls
from cyhy_db import initialize_db
from cyhy_db.models import CVEDoc

async def main():
    # Initialize the CyHy database
    await initialize_db("mongodb://mongoadmin:secret@localhost:32881", "test")

    # Count number of CVE documents in DB before sync
    cve_count_before = await CVEDoc.find_all().count()
    print(f"CVE documents in DB before sync: {cve_count_before}")

    # Fetch CVE data from the default source for a single year and sync it to the database
    created_cve_docs_count, updated_cve_docs_count, deleted_cve_docs_count = await process_urls([DEFAULT_CVE_URL_PATTERN.format(year=2024)], cve_data_gzipped=True)

    print(f"Created CVE documents: {created_cve_docs_count}")
    print(f"Updated CVE documents: {updated_cve_docs_count}")
    print(f"Deleted CVE documents: {deleted_cve_docs_count}")

    # Count number of CVE documents in DB after sync
    cve_count_after = await CVEDoc.find_all().count()
    print(f"CVE documents in DB after sync: {cve_count_after}")

asyncio.run(main())

Output:

CVE documents in DB before sync: 20
Processing CVE feed ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:24
Deleting outdated CVE docs ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Created CVE documents: 11624
Updated CVE documents: 0
Deleted CVE documents: 0
CVE documents in DB after sync: 11644

Environment Variables

Variable Description Default
MONGO_INITDB_ROOT_USERNAME The MongoDB root username mongoadmin
MONGO_INITDB_ROOT_PASSWORD The MongoDB root password secret
DATABASE_NAME The name of the database to use for testing test
MONGO_EXPRESS_PORT The port to use for the Mongo Express web interface 8081

Pytest Options

Option Description Default
--mongo-express Start a local MongoDB instance and Mongo Express web interface n/a
--mongo-image-tag The tag of the MongoDB Docker image to use docker.io/mongo:latest
--runslow Run slow tests n/a

Contributing

We welcome contributions! Please see CONTRIBUTING.md for details.

License

This project is in the worldwide public domain.

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

About

Cyber Hygiene Common Vulnerabilities and Exposures (CVE) synchronization Python library

Resources

License

Security policy

Stars

Watchers

Forks

Packages

No packages published