Drop abuse of pool.RunJobs

[joe-elliott]

Is your feature request related to a problem? Please describe.
Generally the jury is still out on the RunJobs queue, but it is abused in 2 places that needs to be corrected in the short term:

tempo/tempodb/tempodb.go

Line 237 in 3bd19f6

foundBytes, err := rw.pool.RunJobs(derivedCtx, copiedBlocklist, func(ctx context.Context, payload interface{}) ([]byte, error) {

tempo/tempodb/retention.go

Line 22 in 9503c34

// todo: continued abuse of runJobs. need a runAllJobs() method or something

These should be replaced by any of a million better patterns to limit concurrency.

EDIT:
Ignore the first link. See discussion below.

[mdisibio]

Thoughts on also changing blocklist polling? We have had issues then the blocklist is large (thousands).

[joe-elliott]

Yup, that first link is wrong. It's to the only place I want to actually keep RunJobs. It should be:

https://github.com/grafana/tempo/blob/master/tempodb/tempodb.go#L326

which is blocklist polling.

[joe-elliott]

Fixed in the linked PR.