Migrate all existing importers to importer improver structure.

[Hritik14]

Old structure:

• Overview: https://github.com/nexB/vulnerablecode/blob/main/docs/source/how-to-guides/add_new_importer.rst
• Codebase: https://github.com/nexB/vulnerablecode/tree/ed2131656b1b3030c2f9eb28e25c10dbbedc8e1d

New structure:

• Overview: Separate import and improve operations #525
• Codebase: https://github.com/nexB/vulnerablecode

After migrating each importer make sure to

• Modify tests under vulnerabilities/tests
• Enable the modified test by removing it from collect_ignore
• Enable doctests for the importer by removing it from ignore list

Following need migration to the new framework:

• nginx Separate import and improve operations #525
• alpine alpine_linux importer - improver migration #620
• nginx tests Add tests for nginx importer #643
• github github importer - improver migration #604
• openssl openssl importer - improver migration #615
• NVD importer - improver migration #666
• Pysec Add PYSEC importer #607
• gitlab Add support for new GitLab community advisories #438
• redhat importer - improver migration #718
• debian importer - improver migration #716
• archlinux
• debian_oval importer - improver migration #717
• Migrate ubuntu importer to importer-improver model #741
• Migrate postgresql importer #969
• npm npm importer - improver migration #605
• retiredotnet importer - improver migration #606
• Migrate apache_httpd #971
• Collect Mozilla #78
• Migrate gentoo importer #1055
• Migrate istio importer #1059
• Migrate apache_kafka #972 ( PR under progress )
• Migrate apache_tomcat #970 ( PR under progress )
• Collect xen #103 ( PR under progress )
• Migrate suse_scores importer #1052 ( PR under progress )
• Migrate ubuntu_usn importer #1051 ( PR under progress )
• Migrate project_kb_msr2019 #1040 ( PR under progress )
• Migrate elixir_security #1060 ( PR under progress )
• osv Add OSV importer #608 ( PR under progress ) - it's not an old importer that needs to be migrated we are writing a new importer.
• safety_db: this should be deleted and dropped as this is not open source
• Migrate kaybee importer #1011 - This requires some model changes we will do it when all importers are migrated ( PR under progress )
• Migrate suse backports #1054 - The backport data has been removed we will deprecate this importer and add a new importer to integrate data from CVRF, OVAL or CSAF

[tardyp]

Hello,

May I suggest that those big refactors are made in a development branch?

We are rebasing regularly on main branch to follow the dev (and help on the proxy compatibility).
We and were very surprised finding out that all the tests are currently broken.

I would suggest to reset to ed21316 which is the last commit which has passed CI.

[Hritik14]

Hello @tardyp

May I suggest that those big refactors are made in a development branch?

We have gone through a major infrastructural change and any updates on the old codebase will need to be migrated. Accepting updates on the old structure would add to debts and slower migration. What do you think @pombredanne ?

We are rebasing regularly on main branch to follow the dev (and help on the proxy compatibility).

We've marked all proxy compatibility related issues with the networking label and are constantly communicating regarding the development in our gitter channel.

... and were very surprised finding out that all the tests are currently broken.

Other than nix, other active tests should be running. Could you post a log of broken active tests ?

[pombredanne]

@tardyp sorry for the inconvenience but there is no development branch... only one main branch and tags from there. What we could do is tag the last commit before this merge.

[tardyp]

As maybe most of the potential users, it is difficult to really be contributing to opensource projects without actually getting short/middle term value from it.

I've not really dug into the current state nor looked at the details of this refacto, we can only trust you in the fact that this is necessary. I cannot really follow the status in gitter. chatrooms are not very efficient in getting a summary of the activities

Now we saw that there is a big refacto, and that most of the importers are just disabled.

In the current situation vulnerablecode disabled all of its value, and I am very sadned by that as this will not help new users to try it before contributing to it.

From a semi outsider point of view I can report my experience trying to make good use of it.

It is usual for opensource projects to not support proxy as it is hard to test, so that part is not an issue.

We however were surprised to see that a lot of importers just crashed when running them. We were asking ourself whether there was a production instance beyond ourself.

We contributed the fixes as much as we could including the github tag API. I don't know how someone could do a full import without that, as in my experience, the svn way to retrieve the info is just timing out for big projects.

Maybe I was a bit too enthousiastic and shouldn't have ignored the work-in-progress part in the project description.

[pombredanne]

I've not really dug into the current state nor looked at the details of this refacto, we can only trust you in the fact that this is necessary.

Yes, it was essential as otherwise we were creating either misleading or incorrect data or plain ignoring some.

I cannot really follow the status in gitter. chatrooms are not very efficient in getting a summary of the activities

Fair enough... FWIW this has been discussed almost weekly and reported in meeting notes for quite a while.

Now we saw that there is a big refacto, and that most of the importers are just disabled.

Yes, that's a temporary thing until we have progressively ported each of them to the new design. It is going to take some work and is not something that can be done in a day. But we are progressively porting each of them to the new design.

It is usual for opensource projects to not support proxy as it is hard to test, so that part is not an issue.

Actually part of the new design is to have one place where network operations are done and have all importers use it. This way it will be possible to have a single place where to configure a proxy. Today the network operations are all scattered in too many places.

We contributed the fixes as much as we could including the github tag API. I don't know how someone could do a full import without that, as in my experience, the svn way to retrieve the info is just timing out for big projects.

Yes this is much appreciated and the SVN way was a failed experiment indeed.

Maybe I was a bit too enthousiastic and shouldn't have ignored the work-in-progress part in the project description.

Bear with us! Let me tag the last release before this merge so we have a stable base for now.

[pombredanne]

@tardyp I pushed this tag that you can stick to for now https://github.com/nexB/vulnerablecode/releases/tag/v22.01

[TG1999]

We have either migrated the importers listed here, or deprecated or added a follow up issue for same closing this, Feel free to re-open if needed.

[Hammad-1]

Hi @tardyp @pombredanne @Hritik14 @TG1999 @tdruez
I am Hammad, a current MS student in Artificial Intelligence. The proposed project aligns well with my technical expertise and interests, making me eager to explore its codebase and learn about its architecture and functionality. I am eager to contribute to the project For GSOC 2023. My immediate plans include setting up the project locally and familiarizing myself with its codebase. Fix some small issues. I would appreciate guidance in ensuring that I am heading in the right direction. Thank you for your consideration.
A little bit about me:
I'm an experienced Python backend engineer skilled in developing web applications with Django/Django Rest Framework. Proficient in creating efficient data scraping and automation tools using Python web scraping stack, and working with scientific computing libraries like Numpy, Pandas, Matplotlib, and Scikit-learn. Committed to delivering high-quality code and efficient solutions for complex problems.
Looking forward to work with you.
Thanks,
Hammad

[pombredanne]

@Hammad-1 Thank you for your enthusiasm. This issues is not a forum for discussion beyond the matter at hand here!
It would be best to discuss your GSoC interest on the public Gitter channel!

[Hammad-1]

@pombredanne Sure