Skip to content

Commit

Permalink
feat(eventbus): warn applications whenever events aren't read
Browse files Browse the repository at this point in the history
Adds logging for the EventBus, Mainly a periodical warning for applications whenever they
are slow in reading or do not read EventBus notifications. It came up after debugging a
very tricky case where a bugged Server side application wasn't reading `EvtConnectednessChanged`
events, which manifested in the server's inability to accept new streams as connection handling
logic deadlocked on _sync_ Notifee's  `Connected` callbacks which emit the events.

P.S. This change should save some time for future application developers.
  • Loading branch information
Wondertan committed Jan 28, 2023
1 parent 34dc115 commit e974590
Showing 1 changed file with 25 additions and 1 deletion.
26 changes: 25 additions & 1 deletion p2p/host/eventbus/basic.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,14 @@ import (
"reflect"
"sync"
"sync/atomic"
"time"

logging "github.com/ipfs/go-log/v2"
"github.com/libp2p/go-libp2p/core/event"
)

var log = logging.Logger("eventbus")

// /////////////////////
// BUS

Expand Down Expand Up @@ -347,7 +351,27 @@ func (n *node) emit(evt interface{}) {
}

for _, ch := range n.sinks {
ch <- evt
select {
case ch <- evt:
default:
n.emitWithWarn(evt, ch)
}
}
n.lk.Unlock()
}

func (n *node) emitWithWarn(evt any, sink chan any) {
// warn periodically, otherwise a single log may get lost
t := time.NewTicker(time.Millisecond * 100)
defer t.Stop()

for {
select {
case sink <- evt:
return
case <-t.C:
// warn node operator or dev about slow event consumption by application
log.Warn("SLOW EVENT CONSUMER OF TYPE %s", n.typ)
}
}
}

0 comments on commit e974590

Please sign in to comment.