Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return nil for strange FT 500 errors #12948

Merged
merged 0 commits into from
Jun 9, 2023
Merged

Conversation

LindseySaari
Copy link
Contributor

@LindseySaari LindseySaari commented Jun 9, 2023

Summary

On occasion, the v0/feature_toggles endpoint returns 500 errors in large groups. See this example of 368 errors. This also happens over a random cadence that doesn't match up to any specific patterns (We've checked RDS connectivity issues, networking, timeouts, etc)

I added some logging around the issue and the response is that session[:token] is ALWAYS empty and when we do the Session.find(session[:token]), it returns 1 for some reason. It's possible this is a bug in the underlying Redis Namespace gem, etc.

The conditional added here will check if the expected response is not a hash AND if the session[:token] is empty, it will return nil. This will cover the case that we are experiencing.

After doing some investigation, this has been occurring for 7+ months according to Sentry. See comment in ticket here, so it doesn't appear to be an issue that was introduced during the EKS rollout.

Screenshot 2023-06-09 at 2 41 13 PM Screenshot 2023-06-09 at 2 41 31 PM

Related issue(s)

https://app.zenhub.com/workspaces/platform-tech-team-1-633af4074573d06c3cda142a/issues/gh/department-of-veterans-affairs/va.gov-team/57316

Research

It's also possible that this is somehow related to the session reset (maybe a race condition where the session is being reset, but the feature_toggles endpoint is hit at the same time?) During the this example where FT errors are seen, there are MANY reset_session logs.

@LindseySaari LindseySaari merged commit 48726b2 into master Jun 9, 2023
@LindseySaari LindseySaari deleted the Feature-Toggle-errors branch June 9, 2023 19:06
ryan-mcneil pushed a commit that referenced this pull request Dec 11, 2023
* return nil for strange FT case

* rubocops
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants