docs(outputs): Clarify buffer limits behavior and fix spec wording

[Hr0bar]

Summary

As discussed in #15908 , clarifying the behavior of output buffer behavior when it fills up in general, fixing incorrect specs wording and adding related useful note to docs of outpus.azure_monitor.

Hopefully the wording is clear enough?

Also just noticed in the specs: "Telegraf will not make any attempt to limit the size on disk taken by these files beyond cleaning up WAL files for metrics that have successfully been flushed to their output source."

Perhaps the last word should be destination instead of source ?

Checklist

• No AI generated code was used in this PR

Related issues

#15908

[DStrand1]

Thanks for the PR! This should clarify things much better.

Also just noticed in the specs: "Telegraf will not make any attempt to limit the size on disk taken by these files beyond cleaning up WAL files for metrics that have successfully been flushed to their output source."

Perhaps the last word should be destination instead of source ?

I agree changing to "destination" here would be clearer wording, feel free to make that change as well!

[Hr0bar]

There is also "Tracking metrics will be accepted either on a successful write to the output source like currently, or on write to the WAL file."

And Im not so sure anymore, never used tracking metrics, but seems to be write to a buffer WAL file only, perhaps what is meant is that the metric is only handed over to the output plugin (in which case output source would make sense probably), to be written to output destination later (as in external system, file, stdout...).

I would rather leave changing "source" to "destination" in this spec to someone more knowledgeable about Telegraf internals. Perhaps the first occurrence is correct and should be "output source" and only the previously discussed second occurrence needs changing - as "cleaning up WAL files for metrics that have successfully been flushed" seems to indicate write to an external system/file/stdout, rather than only handing the metric over to output plugin.

[DStrand1]

There is also "Tracking metrics will be accepted either on a successful write to the output source like currently, or on write to the WAL file."

And Im not so sure anymore, never used tracking metrics, but seems to be write to a buffer WAL file only, perhaps what is meant is that the metric is only handed over to the output plugin (in which case output source would make sense probably), to be written to output destination later (as in external system, file, stdout...).

This line is actually more incorrect now, it should say something like "Tracking metrics will be accepted on a successful write to the output destination". Because of this I think using destination over source makes more sense, as there is no middle distinction here between "handed to the output plugin" and "written to the output destination," metrics are accepted and cleaned up from the WAL file only if the output plugin successfully delivers the metric

[Hr0bar]

Okay, Ive updated it, makes sense. The previous wording almost got me thinking there is some "in between" state where the metric is "in the output plugin", but not sent to the actual output destination yet. Glad to hear that is not the case.

But we should verify the tracking metrics being "accepted" with the WAL buffer - Im new to the WAL on disk buffer strategy, but I see it is persisted across telegraf instances/restarts, are we sure tracking metrics are "accepted" only when they are flushed from the WAL to the actual output destination ?

For example, I see this comment in the on disk buffer test code:

// Expected to drop the 4th metric, as tracking metrics from
// previous instances  are dropped when the wal file is reopened.

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

[DStrand1]

... I see it is persisted across telegraf instances/restarts, are we sure tracking metrics are "accepted" only when they are flushed from the WAL to the actual output destination ?

Tracking data is only relevant on the current instance of telegraf, the contract is that if a tracking metric is not accepted, the input plugin that generated it will not assume that metric is "done," and will send it again if asked for more metrics. However on a new instance of telegraf, this tracking data has no way to inform the input, so these metrics have to be removed from the buffer and be reacquired again from the input plugin with new tracking data.

[Hr0bar]

Makes sense, the changes should be good then

[DStrand1]

Thanks for updating this!

[srebhan]

Thanks @Hr0bar!