Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: handle server error when refresh token requests come same time #3207

Merged

Conversation

sawadashota
Copy link
Contributor

Related issue(s)

Handle server error when refresh token requests come same time.
Hydra usually handles this as ErrInactiveToken. But sometimes, when requests come very bad timing, this will be ErrServerError with very poor error log ({msg: 'An error occurred', error: {debug: 'server_error', status: 'Internal Server Error'}} ).

As stack trace log (debug mode), it turns out that Persister.deleteSessionBySignature returns sqlconn 's error without any handling. If it's forsite 's error, fosite will handle error as ErrInvalidRequest at RefreshTokenGrantHandler.handleRefreshTokenEndpointStorageError.

StackTrace(debug mode)
github.com/ory/x/sqlcon.handlePostgres
	/go/pkg/mod/github.com/ory/x@v0.0.368/sqlcon/error.go:54
github.com/ory/x/sqlcon.HandleError
	/go/pkg/mod/github.com/ory/x@v0.0.368/sqlcon/error.go:75
github.com/ory/hydra/persistence/sql.(*Persister).deleteSessionBySignature
	/project/persistence/sql/persister_oauth2.go:255
github.com/ory/hydra/persistence/sql.(*Persister).DeleteRefreshTokenSession
	/project/persistence/sql/persister_oauth2.go:330
github.com/ory/fosite/handler/oauth2.(*RefreshTokenGrantHandler).handleRefreshTokenReuse
	/go/pkg/mod/github.com/ory/fosite@v0.42.2/handler/oauth2/flow_refresh.go:206
github.com/ory/fosite/handler/oauth2.(*RefreshTokenGrantHandler).HandleTokenEndpointRequest
	/go/pkg/mod/github.com/ory/fosite@v0.42.2/handler/oauth2/flow_refresh.go:69
github.com/ory/fosite.(*Fosite).NewAccessRequest
	/go/pkg/mod/github.com/ory/fosite@v0.42.2/access_request_handler.go:108
github.com/ory/hydra/oauth2.(*Handler).TokenHandler
	/project/oauth2/handler.go:584
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2047
github.com/julienschmidt/httprouter.(*Router).Handler.func1
	/go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:275
github.com/julienschmidt/httprouter.(*Router).ServeHTTP
	/go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387
github.com/urfave/negroni.Wrap.func1
	/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:46
github.com/urfave/negroni.HandlerFunc.ServeHTTP
	/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:29
github.com/urfave/negroni.middleware.ServeHTTP
	/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2047
github.com/ory/hydra/x.RejectInsecureRequests.func1
	/project/x/tls_termination.go:90
github.com/urfave/negroni.HandlerFunc.ServeHTTP
	/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:29
github.com/urfave/negroni.middleware.ServeHTTP
	/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38
github.com/ory/x/metricsx.(*Service).ServeHTTP
	/go/pkg/mod/github.com/ory/x@v0.0.368/metricsx/middleware.go:275
github.com/urfave/negroni.middleware.ServeHTTP
	/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2047
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1
	/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:198
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2047
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2047
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func1
	/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:68
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2047
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2047
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func1
	/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:165
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2047
github.com/ory/x/prometheusx.Metrics.instrumentHandlerStatusBucket.func1
	/go/pkg/mod/github.com/ory/x@v0.0.368/prometheusx/metrics.go:108

Checklist

  • I have read the contributing guidelines.
  • I have referenced an issue containing the design document if my change introduces a new feature.
  • I am following the contributing code guidelines.
  • I have read the security policy.
  • I confirm that this pull request does not address a security vulnerability.
    If this pull request addresses a security. vulnerability,
    I confirm that I got green light (please contact security@ory.sh) from the maintainers to push the changes.
  • I have added tests that prove my fix is effective or that my feature works.
  • I have added or changed the documentation.

Further Comments

@sawadashota sawadashota requested a review from aeneasr as a code owner July 28, 2022 03:24
@sawadashota sawadashota changed the title fix(oauth2): Handle server error when refresh token requests come same time fix: Handle server error when refresh token requests come same time Jul 28, 2022
@sawadashota sawadashota changed the title fix: Handle server error when refresh token requests come same time fix: handle server error when refresh token requests come same time Jul 28, 2022
Exec())
if err := p.Connection(ctx).
RawQuery(fmt.Sprintf("DELETE FROM %s WHERE signature=?", OAuth2RequestSQL{Table: table}.TableName()), signature).
Exec(); errors.Is(err, sql.ErrNoRows) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can never be true because Exec() is no longer wrapped in sqlcon.HandleError

} else if err := sqlcon.HandleError(err); err != nil {
if errors.Is(err, sqlcon.ErrConcurrentUpdate) {
return errors.Wrap(fosite.ErrSerializationFailure, err.Error())
} else if strings.Contains(err.Error(), "Error 1213") { // InnoDB Deadlock?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we handle this in sqlcon.HandleError instead :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed at 1ad6caa!

Signed-off-by: sawadashota <shota@sslife.tech>
Signed-off-by: sawadashota <shota@sslife.tech>
@aeneasr aeneasr force-pushed the handle_delete_session_by_signature_error branch from 1ad6caa to 4574761 Compare July 29, 2022 06:50
@codecov
Copy link

codecov bot commented Jul 29, 2022

Codecov Report

Merging #3207 (713e5ca) into master (ed6eb30) will decrease coverage by 0.23%.
The diff coverage is 66.27%.

❗ Current head 713e5ca differs from pull request most recent head 4574761. Consider uploading reports for the commit 4574761 to get more accurate results

@@            Coverage Diff             @@
##           master    #3207      +/-   ##
==========================================
- Coverage   79.55%   79.31%   -0.24%     
==========================================
  Files         112      111       -1     
  Lines        7971     8077     +106     
==========================================
+ Hits         6341     6406      +65     
- Misses       1225     1258      +33     
- Partials      405      413       +8     
Impacted Files Coverage Δ
cmd/server/handler.go 63.76% <ø> (ø)
cmd/token_user.go 13.69% <0.00%> (ø)
driver/registry_base.go 87.14% <ø> (-0.05%) ⬇️
x/oauth2cors/cors.go 89.47% <ø> (+0.38%) ⬆️
x/sqlx.go 53.70% <ø> (-26.30%) ⬇️
persistence/sql/persister_oauth2.go 79.91% <25.00%> (-1.94%) ⬇️
client/client.go 74.74% <62.85%> (-6.51%) ⬇️
client/handler.go 77.77% <68.96%> (-1.42%) ⬇️
persistence/sql/migratest/exptected_data.go 100.00% <100.00%> (ø)
... and 1 more

Help us with your feedback. Take ten seconds to tell us how you rate us.

@aeneasr aeneasr merged commit e66ba3c into ory:master Jul 29, 2022
@sawadashota sawadashota deleted the handle_delete_session_by_signature_error branch July 29, 2022 09:26
grantzvolsky pushed a commit that referenced this pull request Jul 31, 2022
grantzvolsky pushed a commit that referenced this pull request Aug 1, 2022
aeneasr pushed a commit that referenced this pull request Aug 1, 2022
aeneasr pushed a commit that referenced this pull request Aug 18, 2022
aeneasr pushed a commit that referenced this pull request Sep 5, 2022
aeneasr pushed a commit that referenced this pull request Sep 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants