[v23.3.x] [CORE-2399] Schema Registry: Improve retries and tombstoning

[vbotbuildovich]

Backport #17905 to branch v23.3.x. Note that this issue was created as a placeholder, since the original PR's commit(s) could not be automatically cherry-picked.
Here are the commands to execute:

git checkout v23.3.x
git checkout -b vbotbuildovich/backport-17905-v23.3.x-150
git cherry-pick -x 23c12f56882a45d5cbe870ea3cf453d1352da125 60ceea9177158ef1e2a145eb70274cb9f961391b a2c9a16a94b2c4e31e6dc2ba2afe5f96b8df020a ae8f74a560059f0de01685dd40eb06686e1fe2b7 640cb843b9eb368366a346d7873f611ec14c8152 813c4beb53361dd75edd3c82fa69f0f51d4506aa 6a48b96a2ae4edc36b685e3a7b2283d6950b9e8a

git push origin vbotbuildovich/backport-17905-v23.3.x-150
gh pr create \
  --title "[v23.3.x] [CORE-2399] Schema Registry: Improve retries and tombstoning" \
  --base "v23.3.x" \
  --label "kind/backport" \
  --head "vbotbuildovich/backport-17905-v23.3.x-150" \
  --draft \
  --repo "redpanda-data/redpanda" \
  --milestone "v23.3.x-next" \
  --body "Backport of PR https://github.com/redpanda-data/redpanda/pull/17905 "

JIRA Link: [CORE-2445](https://redpandadata.atlassian.net/browse/CORE-2445)

[BenPope]

Fixed by #18624