Remove flaky test

[benmoss]

It was not really testing anything but the library. I tried to make it less flaky but couldn't, I don't totally understand something about the way the http listener behaves.

Fixes #5474

[knative-metrics-robot]

The following is the coverage report on the affected files.
Say /test pull-knative-eventing-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/adapter/v2/cloudevents.go	89.7%	87.2%	-2.6

[codecov]

Codecov Report

Merging #5508 (a4cf357) into main (393e411) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #5508   +/-   ##
=======================================
  Coverage   82.71%   82.71%           
=======================================
  Files         197      198    +1     
  Lines        6098     6098           
=======================================
  Hits         5044     5044           
  Misses        729      729           
  Partials      325      325           

Impacted Files	Coverage Δ
pkg/adapter/v2/cloudevents.go	81.91% <0.00%> (-2.13%)	⬇️
pkg/kncloudevents/retries.go	84.61% <0.00%> (ø)
pkg/kncloudevents/message_sender.go	86.95% <0.00%> (+4.69%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 393e411...a4cf357. Read the comment docs.

[matzew]

/lgtm
/approve

[knative-prow-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: benmoss, matzew

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [matzew]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment