proposal: Go 2: sends on closed channels do not panic

[jeromefroe]

Motivation

A common pattern I've seen is using a lock to protect a channel and a variable which indicates whether or not the channel is closed. For example:

type Queue struct {
  sync.RWMutex

  closed bool
  ch chan T
}

func New(cap int) *Queue {
  return &Queue{ ch: make(chan T, cap) } 
}

func (q *Queue) Enqueue(t T) error {
  q.RLock()
  defer q.RUnlock()
  if q.closed {
    return errors.New("queue is closed")
  }
  q.ch <- t
}

func (q *Queue) Close() {
  q.Lock()
  defer q.Unlock()
  if q.closed {
    return
  }
  close(q.closed)
  q.closed = true
}

Using this approach one has to acquire two locks in each call to Enqueue. The first lock being the read lock for q.closed and the second being the internal lock used by the channel. However, the channel already has a field closed to indicate whether the channel has been closed and a lock which protects it, so the extra lock is only necessary because there is no publicly accessible API for determining if the channel is closed.

---

Proposal

To support cases like the one above, where a channel may be closed concurrently with sends on it, I'd like to propose that Go add support for conditionally sending values on a channel by returning a bool from send operations to indicate whether the send was successful. Sends to a channel could then do the following:

 if ok := q.ch <- t; !ok {
  return errChanClosed
}

Callers that do, in fact, want the send to panic could explicitly call panic themselves. While those that want to treat such a case as an error will have that ability as well.

Thanks!

[robpike]

Under this proposal, incorrect code that broke before will silently fail, that is, appear to run correctly while still being incorrect. That seems retrograde.

[zigo101]

how about just like type assertion, if ok is present, then not panic, otherwise, panic just like Go 1.

[robpike]

If a program sends on a closed channel, it's not managing resources correctly. You could argue that your proposal makes it easier to do that, but I would counter that it makes it too easy to construct poor concurrent structures.

[cznic]

Also, let's make integer division by zero silently return a random number. /s

[jeromefroe]

Under this proposal, incorrect code that broke before will silently fail, that is, appear to run correctly while still being incorrect. That seems retrograde.

I think there are a few approaches that could be considered in order to address this issue. The first is that static analysis tools could be used to highlight to users where in their code they are not checking the return value of the send. There's a precedence for this already with linters such as errcheck that find unchecked errors. A second, more heavy-handed approach, would be for the compiler to reject any code that does not check the return value of a send, so that ch <- t becomes illegal. Then callers would be forced to check the return value or at the very least explicitly throw it away.

how about just like type assertion, if ok is present, then not panic, otherwise, panic just like Go 1.

My only hesitation with this approach is that I can't think of any other places in the language where _ = <operation> and <operation> have different semantics when the operation has a single return value. For example, in the case of a type assertion _ = t.(type) and t.(type) have the same semantics the latter form is invalid. I'm not saying I'm personally opposed to this idea though as it would address the issue above of old code now silently failing, just that I'm not sure if the broader community would find it appealing given that it introduces an arbitrary exception.

If a program sends on a closed channel, it's not managing resources correctly. You could argue that your proposal makes it easier to do that, but I would counter that it makes it too easy to construct poor concurrent structures.

I'm not sure that I agree that it would make it much easier to construct poor concurrent structures given that it's already possible to emulate this behavior with an additional lock around the channel, as the Queue example above shows. Since channels are often a point of synchronization, I think it's natural for them to serve as a signal of events in the lifecycle of a program. Allowing sends to fail gracefully would then allow users to propagate information about resources being closed.

[cznic]

My only hesitation with this approach is that I can't think of any other places in the language where _ = and have different semantics.

_ = f() != f() when f has any other number of return values than 1, eg. as in func f() {}.

For example, in the case of a type assertion _ = t.(type) and t.(type) have the same semantics.

The later form is invalid.

[jeromefroe]

_ = f() != f() when f has any other number of return values than 1, eg. as in func f() {}

My apologies, I should have been more explicit, I was referring to function calls that have a single return value. Updated my comment to reflect that.

The later form is invalid.

You're right! Thanks! I had blindly assumed it would work the same as a function call. Updated my comment appropriately.

[jech]

Here's an actual example where this feature would be useful.

I've got some code where a set of goroutines are managing data that is made available over an HTTP interface. When the HTTP code receives a request, if finds a channel to the relevant goroutine in a global sync.Map, then sends a request for the data to the channel it found.

When data is deleted, the goroutine that manages it first removes the channel from the sync.Map, then closes the channel, and then dies. Hence, the channel found by the HTTP code is usually open. However, there's a race — if the channel is closed between the point when it is fetched and the point where it is written to, then the HTTP goroutine panics.

I don't see any clean way to solve this without locking. Currently, I don't bother, and live with the occasional panic.

[zigo101]

@jech you don't need to close the channels, you can use an extra channel (called it exited) for each element in the sync.Map. You can close the exited instead and then you can use a select block to send data, though a select block is not as efficient as a single send operation.

Here are some examples: http://www.tapirgames.com/blog/golang-channel-closing

[jech]

you can use an extra channel

Sure, I can make the structure of my code increasingly complex. Or I could simply write

ch, ok <- &Request{...}
if !ok {
    return 0, ErrDataUnavailable
}

in the reader passed to http.ServeContents.

[robpike]

As others have pointed out, the language already has all the tools you need to achieve your goal, and despite your protests it's really not much more complex to use them.

And as I said before, your proposal makes it too easy to write bad concurrent structures. It's analogous to how reentrant locks result in bad locking protocols.

[jech]

...despite your protests... your proposal makes it too easy to write bad concurrent structures.

Just to avoid any blame getting misdirected -- there's two bodies here. The proposal is due to @jeromefroe, who remained impeccably polite throughout the discussion. It's only me that's rudely protesting.

[zigo101]

Just realized that, for send operations as case conditions, this proposal makes it is hard to keep compatibility. For solo send operations, it is not a big problem.

Good or bad, whether the ok is present will also make effect on the behaviors of select blocks.

[jeromefroe]

And as I said before, your proposal makes it too easy to write bad concurrent structures. It's analogous to how reentrant locks result in bad locking protocols.

Forgive me if I'm beating a dead horse, but it's not clear to me why this proposal makes it too easy to write bad concurrent structures. Admittedly though, I don't have much experience with reentrant locks, so I didn't quite see how the analogy applies.

My thought with this proposal was that since channels serve as synchronization points, one can use a closed channel as a way to cancel in-flight goroutines. For example, a program may have a source of goroutines (a server which creates a new goroutine for each connection being one example) that perform some processing and then put a value on a channel. When an event occurs the program may want to close the source, so no new goroutines are created, and the channel as well, so that any in-flight goroutines are cleaned up.

[jech]

I wrote previously:

ch, ok <- &Request{...}
if !ok {
    return 0, ErrDataUnavailable
}

Looking at this some more, there's a race condition: if the (asynchronous) channel is closed just after the request is written, the request is discarded with no error indication to the caller.

So I'm slowly starting to feel that @robpike is right — this proposal would lead to bad code.

[jeromefroe]

Looking at this some more, there's a race condition: if the (asynchronous) channel is closed just after the request is written, the request is discarded with no error indication to the caller.

Wouldn't that be equivalent to closing a buffered channel that has values in it though? The values are still in the channel and available to be pulled off.

[gopherbot]

Change https://golang.org/cl/86515 mentions this issue: sync: add Chan, in order to avoid panic when send on closed channel

[gopherbot]

Change https://golang.org/cl/86555 mentions this issue: sync: add Chan, in order to avoid panic when send on closed channel

[gopherbot]

Change https://golang.org/cl/86555 mentions this issue: sync: add syncchan, in order to avoid panic when send on closed channel

[bronze1man]

I just found another problem to this "sends on closed channels panic" problem when working with "M receivers, N senders and X closers channel" problem.

"close the channel" and "send message to it with panic recover" in two different goroutines is a data race condition to the golang current data race detecter.
So "send message to it with panic recover" workaroud is not equal to "close an additional signal channel" workaroud.

And the "close an additional signal channel" workaroud is difficult to work right when I need read all the data from the channel after the channel closed.

So if I can use sync.Mutex to solve my problem,I will not use channel. It is more complex then I first thought. The workaround https://golang.org/cl/86555 looks good to me.

[ianlancetaylor]

There are good arguments against this proposal above, and few supporters. Declining.

[niamster]

@robpike (et al)
I'm not quite sure you are 100% correct.
Are there guarantees of message delivery when message is sent and channel is closed just afterwards? I assume yes (didn't dig into the src code though).
However having a status code whether message was sent or not(whether channel is full or closed) is essential, especially when working with 3p libraries. Please, don't tell me that those libraries shall be fixed. I do agree, but the time to market might be essential in several cases.
"SyncChan" is a nice alternative but it might not (an won't be) respected by everyone.

[gwd]

I don't understand any of the arguments against this proposal.

Closing a channel is currently an accepted way of sending a message from the sender to the receiver, saying that they won't be sending any more data down that channel. But what about if you want to send a message from the receiver to the sender, that you won't be reading any more data from the channel?

Take https://github.com/emersion/go-imap, which has Fetch() send messages it receives down a channel as it receives them, as in the following example code:

	messages := make(chan *imap.Message, 10)
	done := make(chan error, 1)
	go func() {
		done <- c.Fetch(seqset, items, messages)
	}()

	for msg := range messages {
		if prs, err := mdb.IsMsgIdPresent(msg.Envelope.MessageID); err != nil {
			// STOP PROCESSING
		} else if prs {
                       // ...
		}
	}

At the moment, the only reliable way to implement // STOP PROCESSING is to loop over all the rest of the messages, reading and discarding them.

Now yes, Fetch() could take Yet Another Channel, whose only purpose is to say, "stop processing"; but that's ugly and complicated; having the receiver simply close the channel, while allowing the sender to check that the send received and return an error (stopping processing) would be simple, clean, and most of all, symmetric.

Let me respond to comments:

Under this proposal, incorrect code that broke before will silently fail, that is, appear to run correctly while still being incorrect. That seems retrograde.

No, it won't (at least, not the current version): Old code will be ch <- t, which will panic; new code will be ch, ok <- t (or something like that) which won't.

If a program sends on a closed channel, it's not managing resources correctly. You could argue that your proposal makes it easier to do that, but I would counter that it makes it too easy to construct poor concurrent structures.

That assumes one specific use of a channel. I think the above way of using channels is perfectly legitimate.

I don't see the difference between saying "if a program sends on a closed channel, it's not managing resources correctly" and saying "if a program receives on a closed channel, it's not managing resources correctly". "Correctly" depends on the model.

As others have pointed out, the language already has all the tools you need to achieve your goal, and despite your protests it's really not much more complex to use them.

The same thing could be said about receiving on closed channels.

All the other objections have been answered as far as I can tell. So the only concrete objection boils down to this unsubstantiated assertion:

And as I said before, your proposal makes it too easy to write bad concurrent structures. It's analogous to how reentrant locks result in bad locking protocols.

Please consider re-opening this issue.

[DeedleFake]

I think a lot of the confusion about this could be resolved by removing two-way channels. If you only have a <-chan or a chan<- in a given place, then it becomes obvious which direction the data should flow, including closes. If you want the receiver to indicate to the sender that data should stop, have the receiver close a second channel and use a select on the sender's end. This is a pattern that context encourages, but without context it seems unfortunately non-obvious due to channels defaulting to being multi-directional. With a two-way channel, it seems very odd that if I use it in both directions than I can never close it without risking a panic, especially considering all the language around closing being a type of send operation.

[gwd]

@DeedleFake Not sure you understood my comment. In my example, there's no confusion -- I did my research, and discovered that while I can do an atomic "receive if not closed" operation, I can't do an atomic "send if not closed" operation. The channel is one way. The producer was asked to send a bunch of data down the channel. The consumer consumes some of it, and realizes he wants to stop the whole transaction. Why should I have an entire second channel on the off chance I need to cancel the operation?

Or alternately -- why bother allowing a "receive if not closed" operation? After all, you could just have two channels: one for the data, one for the stop signal, and select between them.

Given that "receive if not closed" is already implemented, really I don't understand the resistance to implementing "send if not closed". The same arguments for one are arguments for the other; the same arguments against one are arguments against the other.

[DeedleFake]

My comment wasn't a direct response to yours, per se. It was more just a general comment on this discussion. I've been hit by this as well and it took me a while to realize the reason was that I was trying to use channels in both directions, and that the type system seems to default towards encouraging that, even though you really probably shouldn't do that.

Or alternately -- why bother allowing a "receive if not closed" operation? After all, you could just have two channels: one for the data, one for the stop signal, and select between them.

That wouldn't work the same way, though, even if both are being used the same direction as each other. If you close a buffered channel, the receiving end doesn't see that it's closed until the buffer's empty. If you used two channels, one of which is buffered, and selected, then you'd have a random chance of noticing the closed channel in the select.

The biggest problem I usually run into with sending on closed channels comes from trying to have multiple senders on a single channel. In that case, you do need to use a second channel, because even if you signaled the receiving end by closing the channel and used a select in the senders with a second channel just for them, they'll panic. In other words

// Sender:
select {
case <-doneSending:
  return
case mainChannel <- data:
  // This panics in secondary senders when mainChannel gets closed.
}

// Receiver:
for data := range mainChannel {
  // Do stuff.
}

The best ways to do this seem to be to select on the receiving end, too, but that causes the same, aforementioned problem.

[ianlancetaylor]

@gwd

Closing a channel is currently an accepted way of sending a message from the sender to the receiver, saying that they won't be sending any more data down that channel. But what about if you want to send a message from the receiver to the sender, that you won't be reading any more data from the channel?

The key point here is that closing a channel is considered to be a send operation. We need a synchronous point at which you say "there will be no more data on this channel" in order to implement range over a channel.

You are suggesting repurposing close as a message from the receiver to the sender. But such a message is inherently not synchronous, so there is no need to send that message on the same channel. It can be sent on a different channel. You call that "ugly and complicated," but I don't agree. The sender uses a single select statement when sending. Nothing else is required. Conventionally these days people use a context.Context value for this.

I don't see the difference between saying "if a program sends on a closed channel, it's not managing resources correctly" and saying "if a program receives on a closed channel, it's not managing resources correctly". "Correctly" depends on the model.

Yes. The model is: close is an operation performed by the sender to let the receiver know that no more data will follow. You are suggesting that the language should support a different model. But I think we all agree that the language already provides mechanisms for other models, and (without close) does not provide a mechanism for synchronously telling the receiver that there is no more data.

[DeedleFake]

You are suggesting repurposing close as a message from the receiver to the sender.

Like I said earlier, I think the confusion arises primarily in the case of two-way channels. In that case, the language allows you to send data back and forth, so both ends are inherently both senders and receivers simultaneously. Even if you think of a close as a send operation, it does seem odd that it's the one send operation that's not allowed from both ends even on a two-way channel and that attempting it is likely to cause a panic.

[geraldss]

If a select has 2 or more branches (including possibly a default branch) and (a) one branch is a send on a closed channel that will panic, and (b) at least one other branch can proceed without a panic, is the select guaranteed to choose a non-panicking branch? I couldn't find anything in the spec to guarantee this.

[ianlancetaylor]

In general we don't use issues for questions, especially not closed issues. Please see https://golang.org/wiki/Questions. Thanks.

The answer to your question is that a closed channel is never ready to send a value on, so the select statement will never choose that case.

[geraldss]

Thanks @ianlancetaylor . From the spec of July 31, 2019:

A send on a closed channel proceeds by causing a run-time panic.

If one or more of the communications can proceed, a single one that can proceed is chosen via a uniform pseudo-random selection.

I posted here because the spec as written supports the original proposal here.