Gateway stops indexing new blocks

[j1010001]

Problem

Over the last few days we had multiple occurences of EVM GW stopping block indexing.
examples:

1. https://flow-foundation.slack.com/archives/C014WBGR1J9/p1727268188292399
2. https://flow-foundation.slack.com/archives/C014WBGR1J9/p1727227388238869

The Flow process is still running, restarting the flow process on the Gateway temporarily resolves the problem.

[peterargue]

We captured some data from the most recent hang. Besides the halt in indexing, the key indicator there is an issue is that goroutines start to climb:

The goroutines profile shows they are waiting to get a lock creating new subscriptions on the blocks provider

This lock is also held when notifying subscriptions about published messages:

flow-evm-gateway/models/stream.go

Lines 20 to 27 in 44112e1

	func (p *Publisher[T]) Publish(data T) {
	p.mux.RLock()
	defer p.mux.RUnlock()
	for s := range p.subscribers {
	s.Notify(data)
	}
	}

The subscription Notify method previously had a blocking write of the error to the error channel

flow-evm-gateway/models/stream.go

Lines 58 to 63 in 0fde502

	func (b *Subscription[T]) Notify(data T) {
	err := b.callback(data)
	if err != nil {
	b.err <- err
	}
	}

That has since been fixed (PR)

flow-evm-gateway/models/stream.go

Lines 60 to 70 in 44112e1

	func (b *Subscription[T]) Notify(data T) {
	err := b.callback(data)
	if err != nil {
	select {
	case b.err <- err:
	default:
	// TODO: handle this better!
	panic(fmt.Sprintf("failed to send error to subscription %v", err))
	}
	}
	}

Here is the code that deadlocks:

flow-evm-gateway/api/stream.go

Lines 148 to 170 in 47ecbea

	subs := models.NewSubscription(callback(notifier, rpcSub))
	l := logger.With().
	Str("gateway-subscription-id", fmt.Sprintf("%p", subs)).
	Str("ethereum-subscription-id", string(rpcSub.ID)).
	Logger()
	publisher.Subscribe(subs)
	go func() {
	defer publisher.Unsubscribe(subs)
	for {
	select {
	case err := <-subs.Error():
	fmt.Println(err)
	return
	case err := <-rpcSub.Err():
	l.Debug().Err(err).Msg("client unsubscribed")
	return
	}
	}
	}()

The deadlock can happen if the client disconnects, and before Unsubscribe() is called, the same subscription encounters an error. At that point, there is no listener so the open call to Notify() blocks, which also blocks Unsubscribe(), preventing either goroutine from exiting. New subscriptions then block calling Subscribe(), which is the source of the goroutine leak. Finally, the block indexer also uses this same publisher, so calls to Publish() within the block ingestion logic also blocks.