Memory leak

[C-monC]

Hi,

I'm restreaming rtsp cameras over webrtc using pion. It works, I can view the streams but it leaks about a mb every 10 seconds.
Here is the go routine I use to make webrtc.NewTrackLocalStaticSample tracks which I add to peer connections.
I can try make a minimal example if that's necessary. It seems like the inflating function is at pion/rtp/codecs/h264_packet.go line 72.

func rtspConsumer(camera models.Camera) {
	rtspURL := camera.GetRTSPURL(false)
	fmt.Println("RTSP URL is: ", rtspURL)
	outboundVideoTrack, err := webrtc.NewTrackLocalStaticSample(webrtc.RTPCodecCapability{
		MimeType: "video/h264",
	}, fmt.Sprint(camera.ID), fmt.Sprint(camera.ID))
	if err != nil {
		PostException("ERROR from rtspCons - making track: %v", err)
	}
	annexbNALUStartCode := func() []byte { return []byte{0x00, 0x00, 0x00, 0x01} }
	ThisStream := RunningStreams{CameraID: camera.ID, Track: outboundVideoTrack, CameraChan: make(chan string, 4)}
	GlobalStreams.addStream(&ThisStream)
	waitBeforeRetry := false
	for {
		fmt.Println("rerunning main loop in stream: ", rtspURL)
		if waitBeforeRetry {
			fmt.Println("waiting 20 seconds")
			time.Sleep(time.Second * 20)
			waitBeforeRetry = false
		}
		session, err := rtsp.DialTimeout(rtspURL, 20*time.Second)

		if err != nil {
			fmt.Println("Dialling rtsp failed", err)
			waitBeforeRetry = true
			continue
		}
		session.RtspTimeout = 20 * time.Second
		session.RtpTimeout = 20 * time.Second
		session.RtpKeepAliveTimeout = 5 * time.Second

		codecs, err := session.Streams()
		if err != nil {
			fmt.Println("Retrieving codecs failed: ", err)
			waitBeforeRetry = true
			continue
		}
		for i, t := range codecs {
			log.Println("Stream", i, "is of type", t.Type().String())
		}
		if codecs[0].Type() != av.H264 {
			panic("RTSP feed must begin with a H264 codec")
		}
		if len(codecs) != 1 {
			log.Println("Ignoring all but the first stream.")
		}

		var previousTime time.Duration
		for {
			pkt, err := session.ReadPacket()
			if err != nil {
				fmt.Println("error reading packet ", err)
				break
			}

			if pkt.Idx != 0 {
				//audio or other stream, skip it
				continue
			}

			pkt.Data = pkt.Data[4:]

			// For every key-frame pre-pend the SPS and PPS
			if pkt.IsKeyFrame {
				pkt.Data = append(annexbNALUStartCode(), pkt.Data...)
				pkt.Data = append(codecs[0].(h264parser.CodecData).PPS(), pkt.Data...)
				pkt.Data = append(annexbNALUStartCode(), pkt.Data...)
				pkt.Data = append(codecs[0].(h264parser.CodecData).SPS(), pkt.Data...)
				pkt.Data = append(annexbNALUStartCode(), pkt.Data...)
			}

			bufferDuration := pkt.Time - previousTime
			previousTime = pkt.Time
			if err = outboundVideoTrack.WriteSample(media.Sample{Data: pkt.Data, Duration: bufferDuration}); err != nil && err != io.ErrClosedPipe {
				fmt.Println(err)
				break
			}
		}

		if err = session.Close(); err != nil {
			log.Println("session Close error", err)
			waitBeforeRetry = true
		}

		time.Sleep(5 * time.Second)
	}
}

I collected the heap after running the program for a while.
Early on:

after a few minutes:

If I leave the programming running for 2 days it ends up with around 2gb of ram usage.

The datachannel has a similar problem but it is not as severe.

[Sean-Der]

That isn't good @C-monC!

I look at Payload and nothing is obvious to me yet. The only time we store things outside the scope of the function it is a set (and not an append).

Is it possible to recreate this just using a payloader? Would you be able to capture the H264 packets and reproduce this?

[Sean-Der]

I am also available on Slack if you want to debug quicker.

[C-monC]

I've taken a single payload and sent it in a forever loop to the payloader and there is no increase in memory.

Does this mean somehow the calls to H264Payloader.Payload are increasing? I am not sure how to interpret the heap dump.
I checked the goroutine count now as well. There are only 2 rtspConsumers for 2 rtsp streams (The go routine indirectly calling Payload).

Moving to slack, thank you.

[Sean-Der]

Glad to see this is fixed! Me and @C-monC discussed this a little bit on Slack, I believe it was fixed by forcing garbage collection to run more often?

@C-monC is that right? Just want to give context for others who read this in the future :)

[C-monC]

Unfortunately I am not sure what fixed it. After many debugging/profiling hours it just went away. The Payload node in the graph gets capped at about 15mb now.
Potentially it could be that a buggy client was not reading the stream/data-channels causing them to backlog? Also the connection speed is very low here. If the server upload is 2mb/s and the client can only receive at 1mb/s would the payload buffer increase?

Thanks for the help @Sean-Der. I'll update this thread if I figure out what happened.

[JamesVal]

Hey there,

@Sean-Der I've stumbled across this issue and am not sure if my findings are still relevant (we are on the v3 version of the package). My application sounded similar enough, we read frame data from a camera and send it over webRTC using WriteSample. In my debugging, I noticed that the frame size was always different everytime. I came across this issue: golang/go#23199 related to sync.Pool which I think is being utilized.

Could the fact that my frame sizes are always different cause a similar type of leak here? Is this enough information to go off of? I adjusted my code to always send the same frame over and over and the leak seemed to have gone away (the RAM usage would eventually stop climbing and hover around the same number). I don't know Golang well enough but am hoping this can be useful information to provide.

---

For better context.. here's how we're using it: WriteSample(media.Sample{Data: byteArr, Timestamp: time.Now(), Duration: time.Second / time.Duration(fpsConfigurationWebRTC)});

I meant the length of the byteArr is always changing