Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: StreamAlert SQS Classified Logs Missing Prefix #885

Closed
Ryxias opened this issue Jan 22, 2019 · 3 comments · Fixed by #960
Closed

bug: StreamAlert SQS Classified Logs Missing Prefix #885

Ryxias opened this issue Jan 22, 2019 · 3 comments · Fixed by #960

Comments

@Ryxias
Copy link
Contributor

Ryxias commented Jan 22, 2019

Background

Similar to #886

During deployment on test account we noticed that the streamalert_classified_logs SQS queue is missing a prefix.

Description

All SA resources require a prefix. Missing the prefix causes trouble when there are multiple SA deployments on a single AWS Account. Terraform will detect that a duplicate SQS resource already exists during deployment and error.

Steps to Reproduce

Deploy SA to the staging account.

Desired Change

Add the prefix to the SQS queue name.

@Ryxias Ryxias added this to the 2.2.0 milestone Jan 22, 2019
@Ryxias Ryxias added the bug label Jan 22, 2019
@Ryxias Ryxias self-assigned this Jan 22, 2019
@ryandeivert ryandeivert changed the title StreamAlert SQS Classified Logs Missing Prefix bug: StreamAlert SQS Classified Logs Missing Prefix Jan 22, 2019
@Ryxias
Copy link
Contributor Author

Ryxias commented Jan 25, 2019

Leaving this for myself so I don't forget:
If you use python manage.py deploy --function classifier rule you have to deploy twice. The reason for this is the SQS queue is created on the first run, but its URL is generated during creation, so Terraform doesn't know that the URL is changed and fails to update the classifier's SQS_QUEUE_URL environment variable.

@Ryxias
Copy link
Contributor Author

Ryxias commented Jan 25, 2019

Confirmed on stage:
There will be data loss if the SQS queue is not replaced quickly. This is because the Classifier does not actually crash when it runs into the QueueDoesNotExist/NonExistentQueue error. It actually performs 5 retries using backoff before giving up. When it gives up, it exits gracefully, resulting in no Lambda error, but results in the Kinesis iterator popping records off nonetheless.

@ryandeivert ryandeivert modified the milestones: 2.2.0, 2.3.0 Apr 16, 2019
@Ryxias Ryxias modified the milestones: 2.3.0, 3.0.0 Jul 9, 2019
@Ryxias
Copy link
Contributor Author

Ryxias commented Jul 9, 2019

Changing this to a 3.0.0 release. There is no way to do this gracefully without introducing a breaking change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants