-
Notifications
You must be signed in to change notification settings - Fork 617
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add connectionbroker package #1850
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,105 @@ | ||
// Package connectionbroker is a layer on top of remotes that returns | ||
// a gRPC connection to a manager. The connection may be a local connection | ||
// using a local socket such as a UNIX socket. | ||
package connectionbroker | ||
|
||
import ( | ||
"sync" | ||
|
||
"github.com/docker/swarmkit/api" | ||
"github.com/docker/swarmkit/remotes" | ||
"google.golang.org/grpc" | ||
) | ||
|
||
// Broker is a simple connection broker. It can either return a fresh | ||
// connection to a remote manager selected with weighted randomization, or a | ||
// local gRPC connection to the local manager. | ||
type Broker struct { | ||
mu sync.Mutex | ||
remotes remotes.Remotes | ||
localConn *grpc.ClientConn | ||
} | ||
|
||
// New creates a new connection broker. | ||
func New(remotes remotes.Remotes) *Broker { | ||
return &Broker{ | ||
remotes: remotes, | ||
} | ||
} | ||
|
||
// SetLocalConn changes the local gRPC connection used by the connection broker. | ||
func (b *Broker) SetLocalConn(localConn *grpc.ClientConn) { | ||
b.mu.Lock() | ||
defer b.mu.Unlock() | ||
|
||
b.localConn = localConn | ||
} | ||
|
||
// Select a manager from the set of available managers, and return a connection. | ||
func (b *Broker) Select(dialOpts ...grpc.DialOption) (*Conn, error) { | ||
b.mu.Lock() | ||
localConn := b.localConn | ||
b.mu.Unlock() | ||
|
||
if localConn != nil { | ||
return &Conn{ | ||
ClientConn: localConn, | ||
isLocal: true, | ||
}, nil | ||
} | ||
|
||
return b.SelectRemote(dialOpts...) | ||
} | ||
|
||
// SelectRemote chooses a manager from the remotes, and returns a TCP | ||
// connection. | ||
func (b *Broker) SelectRemote(dialOpts ...grpc.DialOption) (*Conn, error) { | ||
peer, err := b.remotes.Select() | ||
if err != nil { | ||
return nil, err | ||
} | ||
|
||
cc, err := grpc.Dial(peer.Addr, dialOpts...) | ||
if err != nil { | ||
b.remotes.ObserveIfExists(peer, -remotes.DefaultObservationWeight) | ||
return nil, err | ||
} | ||
|
||
return &Conn{ | ||
ClientConn: cc, | ||
remotes: b.remotes, | ||
peer: peer, | ||
}, nil | ||
} | ||
|
||
// Remotes returns the remotes interface used by the broker, so the caller | ||
// can make observations or see weights directly. | ||
func (b *Broker) Remotes() remotes.Remotes { | ||
return b.remotes | ||
} | ||
|
||
// Conn is a wrapper around a gRPC client connection. | ||
type Conn struct { | ||
*grpc.ClientConn | ||
isLocal bool | ||
remotes remotes.Remotes | ||
peer api.Peer | ||
} | ||
|
||
// Close closes the client connection if it is a remote connection. It also | ||
// records a positive experience with the remote peer if success is true, | ||
// otherwise it records a negative experience. If a local connection is in use, | ||
// Close is a noop. | ||
func (c *Conn) Close(success bool) error { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's a little counter-intuitive to close an "unsuccessful" connection. Would it be better to move There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I think There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added an observation if There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Unfortunately not, because We could split it out into a separate If it's weird for There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Could this be moved to where failure is observed? If There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
We can do that, but as I mentioned above, I think it's better this way. We had a several bugs before where certain code paths returned early and skipped adjusting the weight. If weight adjustment is part of closing the connection, it can't be skipped accidentally without leaking a connection. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Make sense. We have similar problem in class Swarm where operation failure should adjust node preference. It's hard to do that on all the calls. |
||
if c.isLocal { | ||
return nil | ||
} | ||
|
||
if success { | ||
c.remotes.ObserveIfExists(c.peer, -remotes.DefaultObservationWeight) | ||
} else { | ||
c.remotes.ObserveIfExists(c.peer, remotes.DefaultObservationWeight) | ||
} | ||
|
||
return c.ClientConn.Close() | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the use case for this method? It seems to me that this breaks separation of concerns.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The agent needs access to
Remotes
to update the set of managers based on the list it gets from the dispatcher.https://github.com/docker/swarmkit/pull/1826/files#diff-6ea70e0a4de93ebcb83000a028edc163L344
The alternative to exposing
Remotes
through this method is to implementObserve
,Weights
, andRemove
as connectionbroker methods, and that feels like it would pollute the connectionbroker method set, instead of keeping that functionality specific toRemotes
.Or we could pass the underlying
Remotes
into the agent separately, but I don't like that much either.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good to me.