feat(struc): database modularization and code improvement #137

luabida · 2023-06-09T20:35:17Z

TODOs:

Base modules

Database base class
FTP File base class

Databases

Functions:

List all files
Describe file
Download method
Endpoints

Docs:

docstrings

Tests:

Listing files & downloading

Parsing

dbc -> dbf
dbf -> parquet
parquet -> dataframe

luabida · 2023-06-09T20:43:14Z

This PR aims to improve the code for pysus databases, making it easier to maintain, to test and to add new functionalities to the code. The preview below is one of the benefits of the new code:

fccoelho · 2023-06-12T22:00:17Z

Could you please add some docstrings to the code?

fccoelho · 2023-06-20T11:37:13Z

pysus/ftp/__init__.py

@@ -26,8 +26,16 @@ class File:
    def __init__(
        self, path: str, name: str, size: int, date: datetime.datetime
    ) -> None:
+        try:
+            name, extension = name.split(".")


maybe it is more robust to os.path.splitext(name) here.

Nice, that's exactly what I needed. Thanks!

fccoelho · 2023-06-20T11:38:05Z

pysus/ftp/__init__.py

+            name, extension = name.split(".")
+            self.name = name
+            self.extension = extension
+            self.basename = (".").join([name, extension])


you can use os.path.join here

fccoelho · 2023-06-20T11:40:57Z

pysus/ftp/databases.py

+        if file.name.startswith("SRC"):
+            dis_code = file.name[:3]
+        elif file.name == "LEIBR22":
+            dis_code = "LEIV"  # MISPELLED FILE NAME


This is not really a mispelling the V corresponds to Visceral Leishmaniasis, but still does not follow the pattern of the other file names, so it is ok to add this special treatment here.

luabida · 2023-06-20T20:02:03Z

pysus/ftp/databases.py

+        ER="AIH Rejeitada com erro",
+        SP="Serviços Profissionais",
+        CH="Cadastro Hospitalar",
+        CM="",  # TODO


@fccoelho do you know, by any chance, what CM means here? I couldn't find any reference to it. My guess is Cadastro Municipal, but I'm not sure

No Idea, I have checked the documentation in the ftp but there is no mention of it. Leave it blank for now.

…by 35x

luabida · 2023-08-21T20:25:36Z

@fccoelho I've added a 5 seconds limit to every test because they were taking too much time and stopping the CI before the end of the tests. They should be rather mocked or split into smaller tests in the future. I've also started splitting the dependencies using extras, so now the preprocessing methods would have to install pysus[preprocessing]

luabida · 2023-09-02T16:46:16Z

In [1]: db = SINAN()

In [2]: len(db.files)
Out[2]: 757

In [3]: !ls -la ~/pysus
total 8
drwxrwxr-x  2 bida bida 4096 set  2 13:39 .
drwxr-x--- 58 bida bida 4096 set  2 13:39 ..

In [4]: file = db.files[0]

In [5]: db.describe(file)
Out[5]:
{'name': 'ACBIBR06.dbc',
 'disease': 'Acidente de trabalho com material biológico',
 'year': 2006,
 'size': '28.3 kB',
 'last_update': '01-16-2023 02:15PM'}

In [6]: file.download()
Out[6]: '/home/bida/pysus/ACBIBR06.dbc'

In [7]: file = db.files[1]

In [8]: await file.async_download()

In [9]: !ls -la ~/pysus
total 696
drwxrwxr-x  2 bida bida   4096 set  2 13:41 .
drwxr-x--- 58 bida bida   4096 set  2 13:39 ..
-rw-rw-r--  1 bida bida  28326 set  2 13:40 ACBIBR06.dbc
-rw-rw-r--  1 bida bida 673314 set  2 13:41 ACBIBR07.dbc

In [10]: db.get_files(dis_codes=["ZIKA", "DENG"])
Out[10]:
[DENGBR00.dbc,
 DENGBR01.dbc,
 DENGBR02.dbc,
 DENGBR03.dbc,
 DENGBR04.dbc,
 DENGBR05.dbc,
 DENGBR06.dbc,
 DENGBR07.dbc,
 DENGBR08.dbc,
 DENGBR09.dbc,
 DENGBR10.dbc,
 DENGBR11.dbc,
 DENGBR12.dbc,
 DENGBR13.dbc,
 DENGBR14.dbc,
 DENGBR15.dbc,
 DENGBR16.dbc,
 DENGBR17.dbc,
 DENGBR18.dbc,
 DENGBR19.dbc,
 DENGBR20.dbc,
 DENGBR21.dbc,
 ZIKABR16.dbc,
 ZIKABR17.dbc,
 ZIKABR18.dbc,
 ZIKABR19.dbc,
 ZIKABR20.dbc,
 ZIKABR21.dbc,
 ZIKABR22.dbc,
 DENGBR22.dbc,
 ZIKABR23.dbc]

In [11]: db.get_files(dis_codes=["ZIKA", "DENG"], years=2017)
Out[11]: [DENGBR17.dbc, ZIKABR17.dbc]

In [12]: file = db.get_files(dis_codes=["ZIKA", "DENG"], years=2017)[1]

In [13]: file
Out[13]: ZIKABR17.dbc

In [14]: file.download()
Out[14]: '/home/bida/pysus/ZIKABR17.dbc'

In [15]: file.info
Out[15]:
{'size': '646938',
 'type': 'file',
 'modify': datetime.datetime(2021, 11, 23, 18, 3)}

In [16]: db.format(file)
Out[16]: ('ZIKA', '17')

luabida · 2023-09-02T18:39:37Z

❯ ipython -i pysus/online_data/SINAN.py

In [1]: download("ZIKA", "2023")
Out[1]: ['/home/bida/pysus/ZIKABR23.dbc']

In [2]: download("ZIKA", ["2023", "22"])
Out[2]: ['/home/bida/pysus/ZIKABR22.dbc', '/home/bida/pysus/ZIKABR23.dbc']

In [3]: download("ZIKA", ["2023", "22", 21])
Out[3]:
['/home/bida/pysus/ZIKABR21.dbc',
 '/home/bida/pysus/ZIKABR22.dbc',
 '/home/bida/pysus/ZIKABR23.dbc']

In [4]: download(["ZIKA", "CHIK"], ["2023", "22", 21])
Out[4]:
['/home/bida/pysus/CHIKBR21.dbc',
 '/home/bida/pysus/CHIKBR22.dbc',
 '/home/bida/pysus/ZIKABR21.dbc',
 '/home/bida/pysus/ZIKABR22.dbc',
 '/home/bida/pysus/CHIKBR23.dbc',
 '/home/bida/pysus/ZIKABR23.dbc']

In [5]: get_available_years("DENG")
Out[5]:
[DENGBR00.dbc,
 DENGBR01.dbc,
 DENGBR02.dbc,
 DENGBR03.dbc,
 DENGBR04.dbc,
 DENGBR05.dbc,
 DENGBR06.dbc,
 DENGBR07.dbc,
 DENGBR08.dbc,
 DENGBR09.dbc,
 DENGBR10.dbc,
 DENGBR11.dbc,
 DENGBR12.dbc,
 DENGBR13.dbc,
 DENGBR14.dbc,
 DENGBR15.dbc,
 DENGBR16.dbc,
 DENGBR17.dbc,
 DENGBR18.dbc,
 DENGBR19.dbc,
 DENGBR20.dbc,
 DENGBR21.dbc,
 DENGBR22.dbc]

luabida · 2023-09-02T18:51:40Z

❯ ipython -i pysus/online_data/SIM.py

In [1]: download("SP", 2020)
Out[1]: ['/home/bida/pysus/DOSP2020.dbc']

In [2]: download("MS", 2015)
Out[2]: ['/home/bida/pysus/DOMS2015.dbc']

In [3]: download(["SC", "PE"], [2015, "04"])
Out[3]:
['/home/bida/pysus/DOPE2004.dbc',
 '/home/bida/pysus/DOPE2015.dbc',
 '/home/bida/pysus/DOSC2004.dbc',
 '/home/bida/pysus/DOSC2015.dbc']

luabida · 2023-09-02T19:25:21Z

In [1]: get_available_years("AC")
Out[1]:
[DNRAC1994.dbc,
 DNRAC94.DBC,
 DNRAC95.DBC,
 DNRAC1995.dbc,
 DNAC1996.DBC,
 DNAC1997.DBC,
 DNAC1998.DBC,
 DNAC1999.DBC,
 DNAC2000.DBC,
 DNAC2001.DBC,
 DNAC2002.DBC,
 DNAC2003.DBC,
 DNAC2004.DBC,
 DNAC2005.DBC,
 DNAC2006.DBC,
 DNAC2007.DBC,
 DNAC2008.DBC,
 DNAC2009.DBC,
 DNAC2010.DBC,
 DNAC2011.DBC,
 DNAC2012.DBC,
 DNAC2013.dbc,
 DNAC2014.dbc,
 DNAC2015.dbc,
 DNAC2016.dbc,
 DNAC2017.dbc,
 DNAC2018.dbc,
 DNAC2019.dbc,
 DNAC2020.dbc,
 DNAC2021.dbc]

luabida · 2023-09-02T19:37:52Z

❯ ipython -i pysus/online_data/SIH.py

In [1]: sih.groups
Out[1]:
{'RD': 'AIH Reduzida',
 'RJ': 'AIH Rejeitada',
 'ER': 'AIH Rejeitada com erro',
 'SP': 'Serviços Profissionais',
 'CH': 'Cadastro Hospitalar',
 'CM': ''}

In [2]: download(["SC", "PE"], [2015, "04"], [1,2,3], ["RD", "RJ"])
Out[2]:
['/home/bida/pysus/RDPE0104.dbc',
 '/home/bida/pysus/RDPE0204.dbc',
 '/home/bida/pysus/RDPE0304.dbc',
 '/home/bida/pysus/RDSC0104.dbc',
 '/home/bida/pysus/RDSC0204.dbc',
 '/home/bida/pysus/RDSC0304.dbc']

luabida · 2023-09-02T19:51:34Z

In [1]: sia.groups
Out[1]:
{'AB': 'Laudo de Acompanhamento a Cirurgia Bariátrica',
 'ABO': 'Acompanhamento Pós Cirurgia Bariátrica',
 'ACF': 'Confeção de Fístula Arteriovenosa',
 'AD': 'Laudos Diversos',
 'AM': 'Laudo de Medicamentos',
 'AMP': '',
 'AN': 'Laudo de Nefrologia',
 'AQ': 'Laudo de Quimioterapia',
 'AR': 'Laudo de Radioterapia',
 'ATD': 'Tratamento Dialítico',
 'BI': 'Boletim Individual',
 'IMPBO': '',
 'PA': 'Produção Ambulatorial',
 'PAM': '',
 'PAR': '',
 'PAS': '',
 'PS': 'Psicossocial',
 'SAD': 'Atenção Domiciliar'}

In [2]: download(["AM", "RJ"], [2001, "04"], [1,2,3], ["aq", "PA", "BI"])
Out[2]: ['/home/bida/pysus/PAAM0304.dbc', '/home/bida/pysus/PARJ0304.dbc']

In [3]: download(["AM", "RJ"], [2001, "04", 2010], [1,2,3], ["aq", "PA", "BI"])
Out[3]:
['/home/bida/pysus/PAAM0110.dbc',
 '/home/bida/pysus/PAAM0210.dbc',
 '/home/bida/pysus/PAAM0304.dbc',
 '/home/bida/pysus/PARJ0110.dbc',
 '/home/bida/pysus/PARJ0210.dbc',
 '/home/bida/pysus/PARJ0304.dbc',
 '/home/bida/pysus/PARJ0310.dbc']

github-actions · 2023-09-19T19:42:39Z

🎉 This PR is included in version 0.10.0 🎉

The release is available on:

0.10.0
GitHub release

Your semantic-release bot 📦🚀

luabida added the enhancement label Jun 9, 2023

fccoelho reviewed Jun 20, 2023

View reviewed changes

luabida commented Jun 20, 2023

View reviewed changes

luabida added 10 commits August 21, 2023 17:17

feat(struc): database modularization and code improvement

0a82ceb

SIM database

a122360

SIM database

0ae0ff3

SINASC database

a1b2a32

Docstrings

b5c5414

Finish SIM and minor improvements

19805b4

SIH Database & removing dateparser in File to increase listing speed …

34f68b1

…by 35x

Saving work

1f9ae92

minor fix on zfill_year

114b858

Setting a time limit of 5 seconds to every test

a63de45

luabida force-pushed the modularization branch from efe1695 to a63de45 Compare August 21, 2023 20:20

redo poetry lock

92f0ad2

luabida force-pushed the modularization branch from 2cba0b9 to 92f0ad2 Compare August 21, 2023 20:27

minor fix on dockerfile

8bdedb6

luabida force-pushed the modularization branch from 490b14a to b4756e4 Compare August 21, 2023 21:22

skip tests that take over 5 seconds

e5d2df2

luabida force-pushed the modularization branch from b4756e4 to e5d2df2 Compare August 22, 2023 11:35

luabida added 3 commits August 22, 2023 11:00

Improve SINAN.get_files to accept multiple queues

31a5336

Include asynchronous FTP download method

0aba430

Improve FTP objects when listing ftp content

d3c69cd

luabida force-pushed the modularization branch 2 times, most recently from c2ceea0 to 73a6239 Compare September 2, 2023 16:38

luabida force-pushed the modularization branch 2 times, most recently from 15b1d8b to 436dc2b Compare September 2, 2023 16:57

Implement download methods on File class

54949a7

luabida force-pushed the modularization branch from 436dc2b to 54949a7 Compare September 2, 2023 17:23

luabida added 2 commits September 2, 2023 14:55

Fix some docstrings

3685255

Replace online_data.SINAN methods with new class

0ce5968

Replace online_data.SIM download method with new class

ff28b85

luabida force-pushed the modularization branch from 94266b5 to ff28b85 Compare September 2, 2023 18:54

Replace online_data.sinasc download method with new class

16f0411

luabida added 2 commits September 2, 2023 16:38

Replace online_data.SIH download method with new class

5a8a9e7

Replace online_data.SIA download method with new class

6f9f999

luabida force-pushed the modularization branch from 9241650 to 6f9f999 Compare September 2, 2023 19:51

luabida marked this pull request as ready for review September 2, 2023 19:52

luabida added 2 commits September 2, 2023 17:25

add humanize pckg

2ef2441

Start removing references from online_data.__init__

b77e09b

luabida merged commit d7e6d27 into AlertaDengue:master Sep 2, 2023

github-actions bot added the released label Sep 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(struc): database modularization and code improvement #137

feat(struc): database modularization and code improvement #137

luabida commented Jun 9, 2023 •

edited

Loading

luabida commented Jun 9, 2023

fccoelho commented Jun 12, 2023

fccoelho Jun 20, 2023

luabida Jun 20, 2023

fccoelho Jun 20, 2023

fccoelho Jun 20, 2023

luabida Jun 20, 2023

fccoelho Jun 20, 2023

luabida commented Aug 21, 2023

luabida commented Sep 2, 2023

luabida commented Sep 2, 2023 •

edited

Loading

luabida commented Sep 2, 2023

luabida commented Sep 2, 2023

luabida commented Sep 2, 2023

luabida commented Sep 2, 2023

github-actions bot commented Sep 19, 2023

feat(struc): database modularization and code improvement #137

feat(struc): database modularization and code improvement #137

Conversation

luabida commented Jun 9, 2023 • edited Loading

Base modules

Databases

Functions:

Docs:

Tests:

Listing files & downloading

Parsing

luabida commented Jun 9, 2023

fccoelho commented Jun 12, 2023

fccoelho Jun 20, 2023

Choose a reason for hiding this comment

luabida Jun 20, 2023

Choose a reason for hiding this comment

fccoelho Jun 20, 2023

Choose a reason for hiding this comment

fccoelho Jun 20, 2023

Choose a reason for hiding this comment

luabida Jun 20, 2023

Choose a reason for hiding this comment

fccoelho Jun 20, 2023

Choose a reason for hiding this comment

luabida commented Aug 21, 2023

luabida commented Sep 2, 2023

luabida commented Sep 2, 2023 • edited Loading

luabida commented Sep 2, 2023

luabida commented Sep 2, 2023

luabida commented Sep 2, 2023

luabida commented Sep 2, 2023

github-actions bot commented Sep 19, 2023

luabida commented Jun 9, 2023 •

edited

Loading

luabida commented Sep 2, 2023 •

edited

Loading