Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[greynoisefeed] remove dedup function, add batching for bundle create and submit #2812

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

bradchiappetta
Copy link
Contributor

Proposed changes

  • Working on performance improvements
  • Removes the dedup function as it seems unnecessary and is very slow at high volumes
  • Add batch bundle submission process to allow for work to start processing while still being submitted
  • Change default interval to 24h (GreyNoise recommended interval)
  • Revamp the wait period process to ensure import of feed isn't happening on connector start up when last_run is less then defined interval

Related issues

Checklist

  • I consider the submitted work as finished
  • I tested the code for its functionality using different use cases
  • I added/update the relevant documentation (either on github or on notion)
  • Where necessary I refactored code to improve the overall quality

Further comments

@helene-nguyen helene-nguyen self-requested a review November 20, 2024 07:41
@helene-nguyen helene-nguyen added the partner used to identify PR from patner label Dec 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
partner used to identify PR from patner
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants