pusher: retry shallow receipts

[acud]

This change makes pusher push chunks for which we've received shallow receipts multiple times to try and get them stored deeper. This is a naive implementation that can be elaborate later on to attempt different traces somehow.

unit tests still TODO

---

This change is 

[mrekucci]

Reviewed 2 of 2 files at r1.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @acud)

---

pkg/pusher/pusher.go, line 161 at r1 (raw file):

						po := swarm.Proximity(ch.Address().Bytes(), storerPeer.Bytes())
						s.metrics.ReceiptDepth.WithLabelValues(strconv.Itoa(int(po))).Inc()
						delete(retryCounter, ch.Address().ByteString())

Shouldn't be the delete operation also guarded with the mtx mutex?

[mrekucci]

Reviewed 2 of 2 files at r2.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @acud)

[Eknir]

increment after the check with the counter. Otherwise you will effectively only retry retryCount-1

[acud]

since we already retried, this is the correct thing to do. you can double check

[janos]

This PR somehow revealed a data race on the logger that is shared between goroutines https://github.com/ethersphere/bee/pull/2049/checks?check_run_id=2820412618#step:10:5875.