Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(sinan): remove unnecessary cwd's in FTP_SINAN #123

Merged
merged 4 commits into from
Mar 28, 2023

Conversation

luabida
Copy link
Collaborator

@luabida luabida commented Mar 24, 2023

listing files in ftp server is very dependent on server connection. On tests with airflow, it showed inconstancy. Therefore, this PR tries to reduce the amount of server requests in the FTP_SINAN class

@luabida luabida requested a review from fccoelho March 24, 2023 20:21
@luabida luabida marked this pull request as draft March 24, 2023 20:32
@luabida luabida force-pushed the generators-methods branch from 3ed32f5 to 324ba1d Compare March 24, 2023 20:39
@luabida
Copy link
Collaborator Author

luabida commented Mar 24, 2023


  _     ._   __/__   _ _  _  _ _/_   Recorded: 17:40:51  Samples:  1930
 /_//_/// /_\ / //_// / //_'/ //     Duration: 19.230    CPU time: 2.457
/   _/                      v4.4.0

Program: /home/luabida/Projetos/EGH/EpiGraphHub/containers/airflow/dags/brasil/sinan.py

19.227 <module>  sinan.py:1
├─ 18.069 task_flow_for  sinan.py:59
│  ├─ 17.605 FTP_SINAN.__init__  pysus/online_data/__init__.py:650
│  │  └─ 17.605 FTP.nlst  ftplib.py:547
│  │        [60 frames hidden]  ftplib, socket, .., re, sre_compile, ...
│  └─ 0.249 _TaskDecorator.__call__  airflow/decorators/base.py:310
│        [199 frames hidden]  airflow, abc, .., enum, logging, copy...
├─ 0.426 <module>  airflow/decorators/__init__.py:17
│     [1778 frames hidden]  airflow, flask, werkzeug, http, re, s...
├─ 0.411 <module>  epigraphhub/data/brasil/sinan/__init__.py:1
│     [3 frames hidden]  epigraphhub, ..
│        0.407 <module>  pysus/online_data/__init__.py:1
│        └─ 0.236 <module>  pandas/__init__.py:3
│              [584 frames hidden]  pandas, pyarrow, copy, .., textwrap, ...
└─ 0.233 <module>  airflow/__init__.py:18
      [871 frames hidden]  airflow, importlib, sqlalchemy, textw...

@luabida luabida marked this pull request as ready for review March 24, 2023 20:45
@luabida luabida force-pushed the generators-methods branch from da04656 to 5cadf11 Compare March 24, 2023 21:06
@luabida luabida force-pushed the generators-methods branch from 5cadf11 to 44a615d Compare March 24, 2023 21:40
@luabida luabida merged commit 5199685 into AlertaDengue:master Mar 28, 2023
@github-actions
Copy link

🎉 This PR is included in version 0.9.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants