fix(conn): conn to have state machine #3559

ndyakov · 2025-10-24T10:10:31Z

Add connection state tracking instead of multiple flags
Resolve race conditions on handoff
Improve pool performance

This PR fixes issues related to us having multiple different atomic flags in the pool connections for handoff, usability, etc. Since I tried to slightly changed the get and set operations of the pool, this correctness (which negatively impacted the performance in the first place) does not have real world drawbacks and improves developer experience.

…ne-conn

Copilot

Pull Request Overview

This PR refactors connection management to use a state machine architecture, replacing the previous boolean flags (usable, Inited, used) with a formal state machine that provides better concurrency control and clearer state transitions.

Key Changes:

Introduced a new ConnStateMachine with explicit states: CREATED → INITIALIZING → IDLE ⇄ IN_USE, with UNUSABLE for handoff/reauth operations
Replaced atomic boolean flags with atomic state transitions using CAS operations and FIFO waiting queues
Updated connection initialization logic to handle concurrent initialization attempts properly

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`internal/pool/conn_state.go`	New file implementing the connection state machine with FIFO waiting queue
`internal/pool/conn_state_test.go`	Comprehensive tests for state machine transitions and concurrency
`internal/pool/conn.go`	Refactored connection to use state machine instead of boolean flags
`internal/pool/pool.go`	Updated pool operations to use state machine transitions
`internal/pool/pool_single.go`	Added comments explaining SingleConnPool's non-thread-safe nature
`redis.go`	Refactored `initConn` to handle state transitions and concurrent initialization
`maintnotifications/pool_hook.go`	Updated to check handoff state before usability
`maintnotifications/pool_hook_test.go`	Updated tests to use state machine API
`maintnotifications/handoff_worker.go`	Added call to clear handoff state before closing
`maintnotifications/errors.go`	Added new error type and documentation comments
`internal/auth/streaming/pool_hook.go`	Simplified reauth logic using state machine's AwaitAndTransition
`internal/auth/streaming/pool_hook_state_test.go`	New tests verifying reauth respects state machine

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

redis.go

internal/pool/conn.go

internal/pool/conn_state.go

internal/pool/conn.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull Request Overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

maintnotifications/errors.go

internal/pool/conn.go

internal/pool/conn_state.go

internal/pool/pool.go

internal/pool/conn_state_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…ne-conn

Copilot

Pull Request Overview

Copilot reviewed 34 out of 37 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

redis_test.go

internal/pool/conn_check.go

internal/pool/pool.go

internal/pool/conn.go

htemelski-redis

Overall looks good.
There aside from the other comments, can we preallocate all errors.New(...) too?

htemelski-redis · 2025-11-06T10:27:15Z

internal/auth/streaming/pool_hook.go

-						attempt++
-					}
-				}
+			_, err := stateMachine.AwaitAndTransition(ctx, []pool.ConnState{pool.StateIdle}, pool.StateUnusable)


Can we preallocate this case too?

htemelski-redis · 2025-11-06T10:39:19Z

internal/pool/conn.go

+		// Trying to make usable - transition from UNUSABLE to IDLE
+		// This should only work from UNUSABLE or INITIALIZING states
+		_, err := cn.stateMachine.TryTransition(
+			[]ConnState{StateInitializing, StateUnusable},


Same as above, can we preallocate?

htemelski-redis · 2025-11-06T10:39:32Z

internal/pool/conn.go

+		// Trying to make unusable - transition from IDLE to UNUSABLE
+		// This is typically for acquiring the connection for background operations
+		_, err := cn.stateMachine.TryTransition(
+			[]ConnState{StateIdle},


htemelski-redis · 2025-11-06T10:39:41Z

internal/pool/conn.go

+	defer cancel()
+	finalState, err := cn.stateMachine.AwaitAndTransition(
+		waitCtx,
+		[]ConnState{StateCreated, StateIdle, StateUnusable},


ndyakov added 4 commits October 23, 2025 14:05

wip

27591cd

wip, used and unusable states

606264e

Merge remote-tracking branch 'origin/master' into ndyakov/state-machi…

0a75466

…ne-conn

polish state machine

5721512

ndyakov requested a review from htemelski-redis October 24, 2025 10:10

ndyakov mentioned this pull request Oct 24, 2025

fix(init): conn init should be thread safe #3555

Closed

ndyakov added 4 commits October 24, 2025 14:13

correct handling OnPut

663a60e

better errors for tests, hook should work now

7526e67

fix linter

92433e6

improve reauth state management. fix tests

21bd243

ndyakov requested a review from Copilot October 24, 2025 12:07

Copilot AI reviewed Oct 24, 2025

View reviewed changes

ndyakov and others added 11 commits October 24, 2025 15:20

Update internal/pool/conn.go

3f29463

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update internal/pool/conn.go

de2f8ba

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

better timeouts

94fa920

empty endpoint handoff case

cfcf37d

fix handoff state when queued for handoff

3a53e1b

try to detect the deadlock

c4ed467

try to detect the deadlock x2

9ad6288

delete should be called

03b0003

improve tests

84e856e

fix mark on uninitialized connection

a2c7a25

Merge branch 'master' into ndyakov/state-machine-conn

23d0e0f

ndyakov requested a review from Copilot October 25, 2025 00:33

Copilot AI reviewed Oct 25, 2025

View reviewed changes

ndyakov and others added 5 commits October 25, 2025 21:44

Update internal/pool/conn_state_test.go

ffbe1e5

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update internal/pool/conn_state_test.go

65a6ece

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update internal/pool/pool.go

0964dcc

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update internal/pool/conn_state.go

bc42307

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update internal/pool/conn.go

33696fb

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

ndyakov added 6 commits October 29, 2025 16:06

verify pass auth on conn creation

7201275

fix assertion

62eecaa

fix unsafe test

43eeae7

fix benchmark test

2965e3d

improve remove conn

59da35b

re doesn't support requirepass

09a2f07

ndyakov force-pushed the ndyakov/state-machine-conn branch from 4a452b0 to b55cdad Compare October 30, 2025 16:23

wait more in e2e test

fc2da24

ndyakov force-pushed the ndyakov/state-machine-conn branch from b55cdad to fc2da24 Compare October 30, 2025 16:27

Merge branch 'master' into ndyakov/state-machine-conn

5f0b58b

ndyakov force-pushed the ndyakov/state-machine-conn branch from 1642bb5 to a7b2f91 Compare October 30, 2025 17:11

flaky test

d207749

ndyakov force-pushed the ndyakov/state-machine-conn branch from a7b2f91 to d207749 Compare October 30, 2025 17:19

ndyakov added 3 commits October 30, 2025 19:44

Merge remote-tracking branch 'origin/master' into ndyakov/state-machi…

ef3e06f

…ne-conn

add missed method in interface

5fa97c8

fix test assertions

d91800d

ndyakov force-pushed the ndyakov/state-machine-conn branch from 68199f2 to d91800d Compare October 30, 2025 20:20

ndyakov and others added 5 commits November 4, 2025 18:00

Merge branch 'master' into ndyakov/state-machine-conn

c5ca81d

silence logs and faster hooks manager

a39dd4c

address linter comment

c3dbc8c

Merge branch 'master' into ndyakov/state-machine-conn

3b65139

Merge branch 'master' into ndyakov/state-machine-conn

95af71c

ndyakov requested a review from Copilot November 5, 2025 15:29

Copilot AI reviewed Nov 5, 2025

View reviewed changes

redis_test.go Show resolved Hide resolved

internal/pool/conn_check.go Outdated Show resolved Hide resolved

internal/pool/pool.go Outdated Show resolved Hide resolved

internal/pool/conn.go Outdated Show resolved Hide resolved

ndyakov and others added 5 commits November 5, 2025 21:35

fix flaky test

18b46a1

use read instad of control

4673c62

use pool size for semsize

2b8023c

CAS instead of reading the state

41024f7

Merge branch 'master' into ndyakov/state-machine-conn

3cb5ab3

htemelski-redis reviewed Nov 6, 2025

View reviewed changes

fix(conn): conn to have state machine #3559

Are you sure you want to change the base?

fix(conn): conn to have state machine #3559

Conversation

ndyakov commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

htemelski-redis left a comment

Choose a reason for hiding this comment

Uh oh!

htemelski-redis Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

htemelski-redis Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

htemelski-redis Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

htemelski-redis Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ndyakov commented Oct 24, 2025 •

edited

Loading