Leo/bsi b soup #94

adamjanovsky · 2021-06-09T14:08:22Z

No description provided.

…inks to the certified products page.

…ems to work properly, now working on a good demo script

adamjanovsky · 2021-06-09T14:16:34Z

…one, last step is to integrate it in the current class architecture

…here is no need to search infinitely for subpages, the handler will have one level of sub-handler so it shouldn't be hard to navigate through all of the handler.

…oop, leading to a recursive/infinite process

adamjanovsky

Some intermediate review. Once done with your work, be sure to get rid of non-descriptive comments and make the code (at least somewhat) PEP-8 compliant.

adamjanovsky · 2021-06-22T15:24:46Z

sec_certs/bsi_parse.py

-
-def process(root_url: str):
-    browser = BsiBrowser(root_url)
+def process(base_url: str):


It feels that this whole method should be part of BsiBrowser.parse() method.

adamjanovsky · 2021-06-22T15:26:01Z

sec_certs/bsi_parse.py

-def process(root_url: str):
-    browser = BsiBrowser(root_url)
+def process(base_url: str):
+    browser = BsiBrowser(base_url)
    browser.parse()
    tmp_list = []
    for handler in browser.handler_list:


This whole for loop can be written as a list comprehension and you can call tmp_list.extend([the_comprehension])

adamjanovsky · 2021-06-22T15:27:12Z

sec_certs/bsi_parse.py

+
+
+# To end the processing, a last class will be used with final links
+# to retrieve simple data written on the page


 class Bsitmp(ComplexSerializableType):


Name of class is not really explanatory. Should be named something like BSICertificate

Eventually, you probably won't be calling BSICertificate(link) but some alternative constructor BSICertificate.from_url(url)

adamjanovsky · 2021-06-22T15:28:37Z

sec_certs/bsi_parse.py

+        for arch_handler in handler.handler_list:
+            for arch_url in arch_handler.link_list:
+                arch_temp = Bsitmp(arch_url)
+                tmp_list.append(arch_temp)


Same here, use comprehension and .extend() notation

adamjanovsky · 2021-06-22T15:29:01Z

sec_certs/bsi_parse.py

-def process(root_url: str):
-    browser = BsiBrowser(root_url)
+def process(base_url: str):
+    browser = BsiBrowser(base_url)
    browser.parse()
    tmp_list = []


this could then be instance variable

adamjanovsky · 2021-06-22T15:32:21Z

sec_certs/bsi_parse.py

+            [BsiHandler("https://www.bsi.bund.de/" + a['href'])
+             for a in self.soup.find_all('a', href=True, recursive=True, title=re.compile('Archive'))]
+
+        if len(self.handler_list) != 0:


if self.handler_list:

adamjanovsky · 2021-06-22T15:33:25Z

sec_certs/bsi_parse.py

 test_url = "https://www.bsi.bund.de/SharedDocs/Zertifikate_CC/CC/Digitale_Signatur_Kartenlesegeraete/1046.html" \
           ";jsessionid=AF4A9C0B7D27992808B8EA8C426454BE.internet461?nn=513452 "

 root_url = "https://www.bsi.bund.de/EN/Topics/Certification/certified_products/digital_signature" \


declare with typing.Final type and underscored uppercase to denote constants. Furthemore, these can be moved to BSIParser class afaik.

adamjanovsky · 2021-06-22T15:36:36Z

sec_certs/bsi_parse.py

+    valid_until: str
+    soup: BeautifulSoup
+    id: str
+    pdf_links: list[str]


this won't work. You need to use List[str] from typing module.

adamjanovsky · 2021-06-23T13:30:34Z

sec_certs/bsi_parse.py

+        self.handler_list = \
+            [BsiHandler("https://www.bsi.bund.de/" + a['href'])
+             for a in self.soup.find_all('a', href=True, recursive=True)
+             if 'c-navigation-teaser' in str(a.get('class'))]


You can search directly by link class with find_all(class_='c-navigation-teaser')

…e use the same logic as bsi_parse.py, but adapt it to the layout - attempt to use a dictionary in bsi_parse.py, last fix in the constructors to match the requirement of the serialization. - issue encountered : - the websites seems to block some requests, after putting some delay between each request, it seems to be working better - the last field used as a key in the dictionary wasn't retrieved, testing a new one

…g this kind of key : 'BSI - smart cards and similar devices - BSI-DSZ-CC-0954-2015' todo: do the same thing on ANSSI

…ictionary, with the key being the reference of the product

J08nY · 2022-07-13T21:48:42Z

This is now done in #247.

KeleranV added 2 commits June 3, 2021 09:59

Adding a file to parse a page from the BSI website and retrieve the l…

e9594d2

…inks to the certified products page.

Progress in the architecture of the parsing process, the 3 classes se…

8cd8a5b

…ems to work properly, now working on a good demo script

adamjanovsky marked this pull request as draft June 9, 2021 14:23

KeleranV added 7 commits June 14, 2021 12:10

The function needed to navigate through the whole website should be d…

84a55f9

…one, last step is to integrate it in the current class architecture

rework of the architecture according to the website structure since t…

917443c

…here is no need to search infinitely for subpages, the handler will have one level of sub-handler so it shouldn't be hard to navigate through all of the handler.

rework of the process function to match the new class architecture

79723a0

rework of the process function to match the new class architecture

b98cf16

rework of the process function to match the new class architecture

b28c200

fixes the infinite loop issue, the wrong variable was used in a for l…

e0c2bad

…oop, leading to a recursive/infinite process

rework of the code to improve its quality

275e3d5

adamjanovsky commented Jun 22, 2021

View reviewed changes

adamjanovsky commented Jun 23, 2021

View reviewed changes

KeleranV added 5 commits June 28, 2021 13:01

full rollback before refactoring. WIP

587af45

working refactor of the code

89b943b

fix for the dictionary, a product can be found in the dictionary usin…

9dc9564

…g this kind of key : 'BSI - smart cards and similar devices - BSI-DSZ-CC-0954-2015' todo: do the same thing on ANSSI

last improvement, the anssi browser now stores the information in a d…

1ba242a

…ictionary, with the key being the reference of the product

J08nY closed this Jul 13, 2022

adamjanovsky deleted the leo/BSI_BSoup branch July 27, 2022 10:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Leo/bsi b soup #94

Leo/bsi b soup #94

adamjanovsky commented Jun 9, 2021

adamjanovsky commented Jun 9, 2021 •

edited

Loading

adamjanovsky left a comment

adamjanovsky Jun 22, 2021

adamjanovsky Jun 22, 2021

adamjanovsky Jun 22, 2021

adamjanovsky Jun 22, 2021

adamjanovsky Jun 22, 2021

adamjanovsky Jun 22, 2021

adamjanovsky Jun 22, 2021

adamjanovsky Jun 22, 2021

adamjanovsky Jun 22, 2021 •

edited

Loading

adamjanovsky Jun 23, 2021 •

edited

Loading

J08nY commented Jul 13, 2022

Leo/bsi b soup #94

Leo/bsi b soup #94

Conversation

adamjanovsky commented Jun 9, 2021

adamjanovsky commented Jun 9, 2021 • edited Loading

As of 9 June, the following is done:

As of 9 June, the following remains to be done:

Feedback from 1. July

adamjanovsky left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamjanovsky Jun 22, 2021 • edited Loading

Choose a reason for hiding this comment

adamjanovsky Jun 23, 2021 • edited Loading

Choose a reason for hiding this comment

J08nY commented Jul 13, 2022

adamjanovsky commented Jun 9, 2021 •

edited

Loading

adamjanovsky Jun 22, 2021 •

edited

Loading

adamjanovsky Jun 23, 2021 •

edited

Loading