v1.18: fix: set allocation size to 0 for transactions known to fail (backport of #2966)

[mergify]

Problem

Transactions with invalid system instructions are known to fail but still attempt to reserve allocation space in the cost model despite not actually persisting any new allocated accounts.

Summary of Changes

• Set allocation size to 0 for transactions known to fail
• Clamp total estimated allocation size to the max tx allocation limit since any allocations beyond this limit will fail

Fixes #

---

This is an automatic backport of pull request #2966 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 443246d has failed:

On branch mergify/bp/v1.18/pr-2966
Your branch is up to date with 'origin/v1.18'.

You are currently cherry-picking commit 443246dee0.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   cost-model/src/cost_model.rs

no changes added to commit (use "git add" and/or "git commit -a")

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[bw-solana]

LGTM