Apparent memory leak of spawned tasks

[jebrosen]

I believe there is some form of memory leak, and this was discussed some in the Discord channel #tokio-users recently. This graph should illustrate it pretty well:

The orange 6.2MB are a single allocation by tracing-subscriber, the green are RawTasks that are allocated via spawn, and the blue are something in broadcast. The workload is just running while true; do target/release/cli get k; done for a while (perhaps after setting a value for k, but I believe it doesn't actually matter).

As far as I can tell, the spawned tasks have actually completed. A different implementation (using watch instead of broadcast) fixes it. Another project using broadcast does not have a similar issue. My hunch is that there is some kind of reference cycle, maybe in part because the shutdown coordination is bidirectional, but I have not looked more closely yet.

[Darksonn]

If you modify Tokio to print when Arc<WaitNode> is created and dropped inside broadcast, you will see that it creates one for every connection, but none of them are dropped until the application closes.

[carllerche]

Thanks for the report. I will try to look later today. My guess, w/o looking, is that it is a similar problem as the old semaphore had. If that is a case, it will need a similar rework as tokio-rs/tokio#2325

[hawkw]

@carllerche do you think broadcast can easily be modified to use the semaphore?

[carllerche]

Open PR here: tokio-rs/tokio#2509

[carllerche]

I have been running the reproduction using the PR linked above and memory is stable for me.

[carllerche]

Fixed w/ the tokio release.