changefeedccl: change cdc latency hist bucket sizes

[rharding6373]

Increases the changefeed.admit_latency and changefeed.commit_latency max
from 5m to 60m and decreases their min to 5ms.

The metrics changefeed.parallel_io_queue_nanos,
changefeed.parallel_io_result_queue_nanos,
changefeed.sink_batch_hist_nanos,
changefeed.flush_hist_nanos, and
changefeed.kafka_throttling_hist_nanos have the new limits 5ms to 10m
(previously 500ms to 5m).

Epic: none
Fixes: #134593

Release note (general change): In order to improve the granularity of
changefeed pipeline metrics, the changefeed metrics
changefeed.admit_latency and changefeed.commit_latency have histogram
buckets from 5ms to 60m (previously 500ms to 5m). The changefeed
metrics changefeed.parallel_io_queue_nanos,
changefeed.parallel_io_result_queue_nanos,
changefeed.sink_batch_hist_nanos,
changefeed.flush_hist_nanos, and
changefeed.kafka_throttling_hist_nanos have histogram buckets from 5ms
to 10m (previously 500ms to 5m).

[blathers-crl]

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is 

[andyyang890]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @asg0451 and @rharding6373)

---

-- commits line 9 at r1:
nit: release notes should have a category

---

pkg/ccl/changefeedccl/metrics.go line 1062 at r1 (raw file):

			MaxVal:       commitLatencyMaxValue.Nanoseconds(),
			SigFigs:      1,
			BucketConfig: metric.LongRunning60mLatencyBuckets,

I recall we were previously discussing introducing a new bucket config with a smaller min bucket than 500ms. Should we address that in the same PR?

[rharding6373]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @andyyang890 and @asg0451)

---

pkg/ccl/changefeedccl/metrics.go line 1062 at r1 (raw file):

Previously, andyyang890 (Andy Yang) wrote…

I recall we were previously discussing introducing a new bucket config with a smaller min bucket than 500ms. Should we address that in the same PR?

You're referring to the metrics in this issue? #134593

Those are different metrics. I can try to follow up with that one next week.

[asg0451]

since we're making changes to the histogram buckets, can we change them to ones with a min < 500ms?

also may want to call out which metrics are changing by name in the release notes, in case this affects metric users.

[rharding6373]

TFTRs!

bors r=asg0451

[craig]

Build succeeded:

• macos_amd64_build
• local_roachtest_fips
• lint
• linux_amd64_build
• acceptance
• local_roachtest
• linux_amd64_fips_build
• check_generated_code
• macos_arm64_build
• docker_image_amd64
• unit_tests
• linux_arm64_build
• windows_build
• examples_orms

[blathers-crl]

Based on the specified backports for this PR, I applied new labels to the following linked issue(s). Please adjust the labels as needed to match the branches actually affected by the issue(s), including adding any known older branches.

---

Issue #134593: branch-release-23.2, branch-release-24.1, branch-release-24.2, branch-release-24.3.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from 83e8c2a to blathers/backport-release-23.2-136265: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 23.2.x failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}