add docstrings #309
add docstrings #309
Conversation
lordlabuckdas
force-pushed
the
docstrings
branch
from
36e2c64
to
e6bc07e
Compare
…ython 3.6 and 3.7
| """Base class for all core functions of the cloner | ||
|
|
||
| :param root: Website root URL | ||
| :type root: str |
There was a problem hiding this comment.
do we need type if the type annotated in function definition?
There was a problem hiding this comment.
Yes, Sphinx mandates it and the type disappears from the parameter description in docs when we remove its :type from the docstring.
snare/utils/snare_helpers.py
Outdated
| self.logger = logging.getLogger(__name__) | ||
| self.meta = {} | ||
|
|
||
| def convert(self, path): | ||
| def convert(self, path: str) -> None: | ||
| """Convert all pages to a Snare-friendly form and write meta info |
There was a problem hiding this comment.
what is Snare-friendly format? :)
There was a problem hiding this comment.
I meant to convey the renaming of files to their hashes but I now realize it sounds very vague. I have clarified it in 58eafef.
snare/cloner.py
Outdated
| @@ -111,7 +151,16 @@ async def process_link(self, url, level, check_host=False): | |||
| self.logger.error("ValueError while processing the %s link", url) | |||
| return res | |||
|
|
|||
| async def replace_links(self, data, level): | |||
| async def replace_links(self, data: Union[bytes, str], level: int) -> BeautifulSoup: | |||
| """Replace website links to make them suitable for serving | |||
There was a problem hiding this comment.
what does it mean "suitable"? try to be more concrete, maybe even with example
There was a problem hiding this comment.
Have clarified it in 58eafef. Will add detailed descriptions to this and other functions in the next series of commits.
|
|
||
|
|
||
| class HeadlessCloner(BaseCloner): | ||
| async def fetch_data(self, browser, current_url, level, try_count): | ||
| async def fetch_data( |
There was a problem hiding this comment.
consider adding the docstring for the class as well
lordlabuckdas
force-pushed
the
docstrings
branch
from
59dd4d2
to
58eafef
Compare
No description provided.