Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pulling changes #2

Merged
merged 89 commits into from
Apr 28, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
85d15d2
Switch data_source dependency to GitPython
tardyp Mar 29, 2021
5d29b6f
Resolves #399: Invalid IDs in msr2019
Pushpit07 Apr 4, 2021
0a0c855
Updated RegEx for is_cve in helpers.py to make it more accurate
Pushpit07 Apr 5, 2021
942e1ae
Merge pull request #428 from Pushpit07/rectify_invalid_id_in_msr2019
sbs2001 Apr 5, 2021
f254b0d
Fixes #312 : Added solution for ModuleNotFoundError (#427)
Pushpit07 Apr 5, 2021
54b0c62
remove duplicate import in importers
Apr 6, 2021
a31dc4a
Merge pull request #430 from sify21/rm_dup_import
sbs2001 Apr 6, 2021
5300ce9
Use latest pytest
sbs2001 Apr 6, 2021
ae1c211
Improve nix tests.
rolfschr Mar 13, 2021
1c20128
Use xdist for running upstream tests
sbs2001 Apr 6, 2021
4613b12
Fix tests invocation in upstream tests
sbs2001 Apr 6, 2021
d10e6c1
Organise upstream tests.
sbs2001 Apr 6, 2021
f76b1ce
Update py dep to use version 1.10.0
sbs2001 Apr 6, 2021
326422e
Add pytest.ini
sbs2001 Apr 6, 2021
1221e29
Run upstream tests only periodically
sbs2001 Apr 8, 2021
17b0d74
Remove dephell_specifier and use universal-versions in safetydb
sbs2001 Mar 18, 2021
d316abc
Refactor npm importer to use universal_versions
sbs2001 Mar 24, 2021
82aee93
Refactor elixir importer to use universal_versions
sbs2001 Mar 25, 2021
7dc7a1e
Refactor Istio to use universal_versions
sbs2001 Mar 25, 2021
bf90402
Refactor rust importer to use universal_versions
sbs2001 Mar 25, 2021
a401218
Refactor nginx importer to use universal_versions
sbs2001 Mar 25, 2021
d57a5d6
Refactor apache tomcat importer to use universal versions
sbs2001 Mar 25, 2021
323a4b9
Refactor apache kakfa importer to use universal versions
sbs2001 Mar 25, 2021
2cbc23e
Refactor ruby importer to use universal versions
sbs2001 Mar 25, 2021
6f643c9
Refactor github importer and fix doctests in tomcat
sbs2001 Mar 30, 2021
5e6b29e
Refactor oval data sources to use universal versions
sbs2001 Mar 30, 2021
0ac1ab3
Refactor tests
sbs2001 Mar 31, 2021
bda5f0d
Update packaging dep
sbs2001 Mar 31, 2021
049f05c
Remove debug statements, improve doc, simplify code
sbs2001 Apr 2, 2021
653f949
Refactor universal versions to univers
sbs2001 Apr 2, 2021
7c4bbf4
Update attrs dep
sbs2001 Apr 2, 2021
0ca5e16
Install univers from pypi
sbs2001 Apr 8, 2021
5152a97
Merge pull request #421 from nexB/drop_dephell_specifier
sbs2001 Apr 8, 2021
459a16c
Bump django from 3.0.13 to 3.0.14
dependabot[bot] Apr 8, 2021
73e8baf
Merge pull request #435 from nexB/dependabot/pip/django-3.0.14
sbs2001 Apr 9, 2021
8aafb63
Fix ubuntu_usn importer for invalid CVE (#432)
AmitGupta7580 Apr 12, 2021
9be7cee
use binaryornot instead of handmade function
tardyp Apr 15, 2021
6e76309
Add --in-place option.
rolfschr Apr 18, 2021
2da0c14
Update nix deps in place during github workflow.
rolfschr Apr 18, 2021
9ff0192
Merge pull request #444 from ngi-nix/fix-nix-workflow
sbs2001 Apr 19, 2021
b5a48a9
expose find_all_cve helper
Hritik14 Apr 15, 2021
525729b
Merge pull request #439 from Hritik14/cve_regex_helper
sbs2001 Apr 22, 2021
3e094c9
Add migrations and code to infer patched package.
sbs2001 Apr 4, 2021
7021526
Refactor the import process to implement patched_package
sbs2001 Apr 9, 2021
d684871
Refactor alpine importer and it's tests fwrt new models
sbs2001 Apr 9, 2021
8bc91ff
Refactor debian importer and it's tests wrt to new models
sbs2001 Apr 9, 2021
cea3738
Refactor apache kafka and it's tests wrt new models
sbs2001 Apr 9, 2021
b0ccf83
Refactor apache tomcata and it's importers for new models
sbs2001 Apr 9, 2021
5344e95
Refactor archlinux importer wrt new models
sbs2001 Apr 9, 2021
6607c29
Refactor elixr security importer wrt new models
sbs2001 Apr 9, 2021
d430a21
Refactor gentoo importer wrt new models and update univers
sbs2001 Apr 9, 2021
46fd0a0
Use AffectedPackageWithPatched dataclass and refactor the Advisory in…
sbs2001 Apr 9, 2021
1508274
Refactor github importer wrt new models
sbs2001 Apr 9, 2021
2b3334f
Refactor istio importer wrt new models
sbs2001 Apr 9, 2021
1cae542
Refactor kaybee importer wrt new models
sbs2001 Apr 9, 2021
f73fbc9
Refactor nginx importer wrt new models
sbs2001 Apr 9, 2021
e97c415
Refactor npm importer wrt new models
sbs2001 Apr 10, 2021
c1c0d14
Refactor openssl wrt new models
sbs2001 Apr 10, 2021
e9cd094
Refactor postgresql importer
sbs2001 Apr 10, 2021
db17f09
Refactor msr2019 importer
sbs2001 Apr 10, 2021
9f6bb60
Refactor redhat importer
sbs2001 Apr 10, 2021
92fd1ad
Refactor retire dot net
sbs2001 Apr 10, 2021
30b0b49
Refactor ruby importer
sbs2001 Apr 10, 2021
c852afc
Refactor rust importer and simplify it's tests
sbs2001 Apr 10, 2021
d1f5485
Refactor safetydb wrt new models and simplify tests
sbs2001 Apr 12, 2021
527324f
Disable suse backport importer and refactor suse score importer
sbs2001 Apr 12, 2021
29f1185
Refactor ubuntu usn importer and tests
sbs2001 Apr 12, 2021
c12e985
Refactor ubuntu
sbs2001 Apr 12, 2021
cbd6c73
Fix more tests and refactor importer_runner
sbs2001 Apr 12, 2021
9246553
Update fixtures
sbs2001 Apr 15, 2021
36f4d1d
Fix model relations and patch inference in importers
sbs2001 Apr 16, 2021
5999a8a
Simplify patch inference, fix arch importer bug and remove useless code
sbs2001 Apr 19, 2021
d70a931
Fix codestyle
sbs2001 Apr 19, 2021
5973bbc
Add docstrings for model properties
sbs2001 Apr 19, 2021
cc5bbb4
Improve naming and docs
sbs2001 Apr 22, 2021
fd15724
Merge pull request #436 from sbs2001/add-patched-pkg
sbs2001 Apr 22, 2021
cac1aa3
add `blank=True` to fields of Vulnerability and Package (#433)
sify21 Apr 22, 2021
74a1afa
Merge branch 'main' into gitpython
sbs2001 Apr 22, 2021
13eaa01
Merge pull request #409 from tardyp/gitpython
sbs2001 Apr 22, 2021
deb44c6
Fix views
sbs2001 Apr 22, 2021
7a26cb9
:sparkles: Improve postgres importer
sbs2001 Apr 23, 2021
663f43e
Fix model properties to avoid dups
sbs2001 Apr 23, 2021
d041a2b
Ignore wildcard versions in gentoo and remove redundant docstring
sbs2001 Apr 23, 2021
2744afc
Fix and improve istio importer
sbs2001 Apr 26, 2021
412a429
Drop VulnerabilityReference's source field
sbs2001 Apr 26, 2021
f9233cc
Fix openssl importer.
sbs2001 Apr 27, 2021
65443e4
:fire: Remove redundant files(Procfile, apps.py)
sbs2001 Apr 27, 2021
114a970
Fix debian importer.
sbs2001 Apr 27, 2021
254cb96
:bug: Fix safetydb and disable it by default (license issue)
sbs2001 Apr 27, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/test-import-using-nix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,6 @@ jobs:
- name: run checks & test import
run: |
cd etc/nix
./get-latest-pypi-deps-db.sh --in-place
nix --print-build-logs flake check
./test-import-using-nix.sh alpine
5 changes: 3 additions & 2 deletions .github/workflows/upstream_test.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
on:
workflow_dispatch: # allow manual execution
push:
schedule:
# run on every 9 o'clock
- cron: '0 9 * * *'
Expand Down Expand Up @@ -39,9 +38,11 @@ jobs:
sudo apt install python3-dev postgresql libpq-dev build-essential libxml2-dev libxslt1-dev
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install pytest-xdist

- name: Run tests
run: pytest -v -m webtest
run: |
python -m pytest -v vulnerabilities/tests/test_upstream.py -n 2
env:
# The hostname, username used to communicate with the PostgreSQL service container
POSTGRES_HOST: localhost
Expand Down
1 change: 0 additions & 1 deletion Procfile

This file was deleted.

3 changes: 3 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,9 @@ compose. For this you need to have the following installed.
Use ``sudo docker-compose up`` to start VulnerableCode. Then access
VulnerableCode at http://localhost:8000/ or at http://127.0.0.1:8000/

**Important**: Don't forget to run ``sudo docker-compose up -d --no-deps --build web`` to sync your instance after every ``git pull``.


Use ``sudo docker-compose exec web bash`` to access the VulnerableCode
container. From here you can access ``manage.py`` and run management commands
to import data as specified below.
Expand Down
8 changes: 6 additions & 2 deletions etc/nix/flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@
name = "vulnerablecode-${version}";
src = vulnerablecode-src;
dontConfigure = true; # do not use ./configure
propagatedBuildInputs = [ pythonEnv postgresql ];
propagatedBuildInputs = [ pythonEnv postgresql gitMinimal];

postPatch = ''
# Make sure the pycodestyle binary in $PATH is used.
Expand Down Expand Up @@ -136,7 +136,11 @@
checkPhase = ''
# Run pytest on the installed version. A running postgres
# database server is needed.
(cd ${vulnerablecode} && pytest)
(
cd ${vulnerablecode}
black -l 100 --check .
pytest -m "not webtest"
)

# Launch the webserver and call the API.
${vulnerablecode}/manage.py runserver &
Expand Down
14 changes: 12 additions & 2 deletions etc/nix/get-latest-pypi-deps-db.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,16 @@ COMMIT=$(sed '1q;d' <<< "$DATA")
DATE=$(sed '2q;d' <<< "$DATA")
SHA256=$(nix-prefetch-url --unpack --type sha256 "https://github.com/$USER_SLASH_REPO/tarball/$COMMIT" | tail -n 1)

NIX_REV_ATTR="pypiDataRev = \"$COMMIT\"; # $DATE"
NIX_SHA_ATTR="pypiDataSha256 = \"$SHA256\";"

echo ""
echo "pypiDataRev = \"$COMMIT\"; # $DATE"
echo "pypiDataSha256 = \"$SHA256\";"
echo $NIX_REV_ATTR
echo $NIX_SHA_ATTR

if [[ "$1" = "--in-place" ]] ; then
# Replace the values in the flake.
PATTERN="\s*\n?\s*\"[^\n]+" # <space><newline><space>"content...<newline>
perl -i.bak1 -0777 -pe "s/pypiDataRev =$PATTERN/$NIX_REV_ATTR/" flake.nix
perl -i.bak2 -0777 -pe "s/pypiDataSha256 =$PATTERN/$NIX_SHA_ATTR/" flake.nix
fi
2 changes: 2 additions & 0 deletions pytest.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[pytest]
DJANGO_SETTINGS_MODULE = vulnerablecode.settings
18 changes: 10 additions & 8 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
aiohttp==3.7.4
asgiref==3.2.7
attrs==19.3.0
attrs==20.3.0
backcall==0.1.0
beautifulsoup4==4.7.1
binaryornot==0.4.4
cached-property==1.5.1
cffi==1.14.0
contextlib2==0.5.5
decorator==4.4.2
dephell-specifier==0.2.1
univers==21.4.16.6
dj-database-url==0.4.2
Django==3.0.13
Django==3.0.14
django-filter==2.2.0
djangorestframework==3.11.2
django-widget-tweaks==1.4.8
Expand All @@ -22,7 +23,7 @@ jedi==0.17.0
lxml==4.6.3
more-itertools==8.0.2
packageurl-python==0.9.3
packaging==19.2
packaging==20.9
parso==0.7.0
pexpect==4.8.0
pickleshare==0.7.5
Expand All @@ -31,15 +32,16 @@ prompt-toolkit==3.0.5
black==20.8b1
psycopg2==2.8.4
ptyprocess==0.6.0
py==1.8.0
py==1.10.0
pycparser==2.20
pygit2==1.5.0
gitpython==3.1.14
Pygments==2.7.4
pyparsing==2.4.5
pytest==5.3.2
pytest==6.2.3
pytest-django==4.1.0
django-widget-tweaks==1.4.8
pytest-dependency==0.4.0
pytest-django==3.7.0
pytest-django==4.1.0
pytest-mock==1.13.0
python-dateutil==2.8.1
pytz==2019.3
Expand Down
2 changes: 1 addition & 1 deletion vulnerabilities/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ class PackageAdmin(admin.ModelAdmin):

@admin.register(PackageRelatedVulnerability)
class PackageRelatedVulnerabilityAdmin(admin.ModelAdmin):
list_filter = ("is_vulnerable", "package__type", "package__namespace")
list_filter = ("package__type", "package__namespace")
search_fields = ["vulnerability__vulnerability_id", "package__name"]


Expand Down
2 changes: 1 addition & 1 deletion vulnerabilities/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ class VulnerabilityReferenceSerializer(serializers.ModelSerializer):

class Meta:
model = VulnerabilityReference
fields = ["source", "reference_id", "url", "scores"]
fields = ["reference_id", "url", "scores"]


class MinimalPackageSerializer(serializers.HyperlinkedModelSerializer):
Expand Down
8 changes: 0 additions & 8 deletions vulnerabilities/apps.py

This file was deleted.

Loading