chore(deps): 'cannot find -lsasl' in some environments

[DarkWanderer]

When trying to build vector on Fedora 37, I encounter following error:
/usr/bin/ld: cannot find -lsasl
even though I have cyrus-sasl installed. rdkafka official recommendation is to use cmake to fix such issues

This PR switches rdkafka's build system to cmake to allow for better compatibility

[netlify]

✅ Deploy Preview for vrl-playground ready!

Name	Link
🔨 Latest commit	e21c4cd
🔍 Latest deploy log	https://app.netlify.com/sites/vrl-playground/deploys/63a19a332ffea7000856b810
😎 Deploy Preview	https://deploy-preview-15650--vrl-playground.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[netlify]

✅ Deploy Preview for vector-project canceled.

Name	Link
🔨 Latest commit	e21c4cd
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/63a19a33608ba6000972e142

[github-actions]

Regression Test Results

Run ID: 1d4acd8e-8f87-491d-9ae5-670fd044097f
Baseline: 1c0716f
Comparison: e21c4cd
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
http_text_to_http_json	769.15KiB/CPU-s	3.21	100.00%	23.36MiB/CPU-s	644.05KiB/CPU-s	8.31KiB/CPU-s	0.0	0.026917	24.12MiB/CPU-s	625.3KiB/CPU-s	8.07KiB/CPU-s	0.0	0.02532	False	False
datadog_agent_remap_datadog_logs_acks	900.19KiB/CPU-s	2.68	100.00%	32.8MiB/CPU-s	1.25MiB/CPU-s	16.55KiB/CPU-s	0.0	0.038173	33.68MiB/CPU-s	734.86KiB/CPU-s	9.48KiB/CPU-s	0.0	0.021304	False	False
socket_to_socket_blackhole	212.18KiB/CPU-s	1.56	100.00%	13.24MiB/CPU-s	259.12KiB/CPU-s	3.35KiB/CPU-s	0.0	0.019106	13.45MiB/CPU-s	151.79KiB/CPU-s	1.96KiB/CPU-s	0.0	0.01102	False	False
syslog_log2metric_splunk_hec_metrics	139.68KiB/CPU-s	1.52	100.00%	8.97MiB/CPU-s	288.8KiB/CPU-s	3.73KiB/CPU-s	0.0	0.031431	9.11MiB/CPU-s	99.33KiB/CPU-s	1.28KiB/CPU-s	0.0	0.010648	False	False
splunk_hec_route_s3	164.09KiB/CPU-s	1.37	100.00%	11.66MiB/CPU-s	672.98KiB/CPU-s	8.68KiB/CPU-s	0.0	0.05637	11.82MiB/CPU-s	521.9KiB/CPU-s	6.74KiB/CPU-s	0.0	0.043122	False	False
syslog_loki	113.82KiB/CPU-s	1.34	100.00%	8.29MiB/CPU-s	255.87KiB/CPU-s	3.3KiB/CPU-s	0.0	0.030139	8.4MiB/CPU-s	135.14KiB/CPU-s	1.74KiB/CPU-s	0.0	0.015707	False	False
datadog_agent_remap_datadog_logs	409.29KiB/CPU-s	1.2	100.00%	33.42MiB/CPU-s	1.2MiB/CPU-s	15.81KiB/CPU-s	0.0	0.035815	33.82MiB/CPU-s	1.12MiB/CPU-s	14.82KiB/CPU-s	0.0	0.033181	False	False
otlp_http_to_blackhole	16.38KiB/CPU-s	1.05	100.00%	1.52MiB/CPU-s	116.98KiB/CPU-s	1.51KiB/CPU-s	0.0	0.075236	1.53MiB/CPU-s	121.68KiB/CPU-s	1.57KiB/CPU-s	0.0	0.077449	False	False
http_to_http_acks	33.52KiB/CPU-s	0.62	48.37%	5.24MiB/CPU-s	2.72MiB/CPU-s	35.94KiB/CPU-s	0.0	0.519285	5.27MiB/CPU-s	2.81MiB/CPU-s	37.08KiB/CPU-s	0.0	0.532439	True	False
syslog_regex_logs2metric_ddmetrics	17.48KiB/CPU-s	0.47	97.34%	3.64MiB/CPU-s	463.43KiB/CPU-s	5.98KiB/CPU-s	0.0	0.124311	3.66MiB/CPU-s	397.45KiB/CPU-s	5.13KiB/CPU-s	0.0	0.106116	True	False
otlp_grpc_to_blackhole	2.0KiB/CPU-s	0.19	98.27%	1.02MiB/CPU-s	36.95KiB/CPU-s	488.5B/CPU-s	0.0	0.035378	1.02MiB/CPU-s	53.62KiB/CPU-s	708.31B/CPU-s	0.0	0.051237	False	False
enterprise_http_to_http	6.0KiB/CPU-s	0.04	87.99%	13.62MiB/CPU-s	259.06KiB/CPU-s	3.34KiB/CPU-s	0.0	0.018576	13.62MiB/CPU-s	149.36KiB/CPU-s	1.93KiB/CPU-s	0.0	0.010705	False	False
splunk_hec_to_splunk_hec_logs_noack	3.34KiB/CPU-s	0.02	61.89%	13.62MiB/CPU-s	223.04KiB/CPU-s	2.88KiB/CPU-s	0.0	0.015989	13.63MiB/CPU-s	193.17KiB/CPU-s	2.49KiB/CPU-s	0.0	0.013844	False	False
fluent_elasticsearch	-40.78B/CPU-s	-0.0	5.82%	45.41MiB/CPU-s	30.31KiB/CPU-s	396.29B/CPU-s	0.0	0.000652	45.41MiB/CPU-s	30.14KiB/CPU-s	394.22B/CPU-s	0.0	0.000648	False	False
http_to_http_noack	56.41B/CPU-s	0.0	0.77%	13.61MiB/CPU-s	311.53KiB/CPU-s	4.02KiB/CPU-s	0.0	0.022345	13.61MiB/CPU-s	311.81KiB/CPU-s	4.02KiB/CPU-s	0.0	0.022366	False	False
file_to_blackhole	-4.98KiB/CPU-s	-0.01	17.72%	54.49MiB/CPU-s	1.17MiB/CPU-s	15.4KiB/CPU-s	0.0	0.021386	54.49MiB/CPU-s	1.21MiB/CPU-s	16.01KiB/CPU-s	0.0	0.022264	False	False
splunk_hec_indexer_ack_blackhole	-1.9KiB/CPU-s	-0.01	32.74%	13.62MiB/CPU-s	239.82KiB/CPU-s	3.09KiB/CPU-s	0.0	0.017195	13.62MiB/CPU-s	252.29KiB/CPU-s	3.25KiB/CPU-s	0.0	0.018091	False	False
splunk_hec_to_splunk_hec_logs_acks	-750.75B/CPU-s	-0.01	9.29%	13.62MiB/CPU-s	342.11KiB/CPU-s	4.41KiB/CPU-s	0.0	0.024533	13.62MiB/CPU-s	346.48KiB/CPU-s	4.47KiB/CPU-s	0.0	0.024847	False	False
http_to_http_json	-9.38KiB/CPU-s	-0.07	94.64%	13.6MiB/CPU-s	264.84KiB/CPU-s	3.42KiB/CPU-s	0.0	0.01902	13.59MiB/CPU-s	267.94KiB/CPU-s	3.46KiB/CPU-s	0.0	0.019255	False	False
syslog_humio_logs	-14.72KiB/CPU-s	-0.17	100.00%	8.65MiB/CPU-s	201.62KiB/CPU-s	2.6KiB/CPU-s	0.0	0.022747	8.64MiB/CPU-s	183.23KiB/CPU-s	2.37KiB/CPU-s	0.0	0.020707	False	False
datadog_agent_remap_blackhole	-218.5KiB/CPU-s	-0.69	100.00%	31.11MiB/CPU-s	1.13MiB/CPU-s	14.92KiB/CPU-s	0.0	0.036278	30.9MiB/CPU-s	858.25KiB/CPU-s	11.08KiB/CPU-s	0.0	0.027125	False	False
syslog_splunk_hec_logs	-87.21KiB/CPU-s	-0.97	100.00%	8.74MiB/CPU-s	172.77KiB/CPU-s	2.23KiB/CPU-s	0.0	0.019296	8.66MiB/CPU-s	256.0KiB/CPU-s	3.3KiB/CPU-s	0.0	0.028873	False	False
syslog_log2metric_humio_metrics	-101.95KiB/CPU-s	-1.64	100.00%	6.06MiB/CPU-s	196.38KiB/CPU-s	2.54KiB/CPU-s	0.0	0.031649	5.96MiB/CPU-s	194.45KiB/CPU-s	2.51KiB/CPU-s	0.0	0.031862	False	False
datadog_agent_remap_blackhole_acks	-676.33KiB/CPU-s	-2.08	100.00%	31.7MiB/CPU-s	687.0KiB/CPU-s	8.87KiB/CPU-s	0.0	0.021159	31.04MiB/CPU-s	1.13MiB/CPU-s	15.0KiB/CPU-s	0.0	0.036535	False	False

[bruceg]

This can also be resolved by using cargo build --features=default-cmake. Do we need to make this the normal default?

[bruceg]

Noting for posterity that the separation between default and default-cmake dates back to December 2019 in issue #1331. AFAICT building vector on Linux requires cmake for other dependencies, so it might be pointless to try to build at all without cmake, which would point to eliminating that feature flag entirely.

[bruceg]

Correction: The default build of Vector (i.e. with default features) on Linux does not require cmake to build, so the question about the direction of this request still stands.

[jszwedko]

@DarkWanderer apologies for the delay in this. Could you link to where rdkafka recommends the use of cmake to resolve the issue you are seeing?

[DarkWanderer]

@DarkWanderer apologies for the delay in this. Could you link to where rdkafka recommends the use of cmake to resolve the issue you are seeing?

No problem. Here is a link to the comment in rust-rdkafka repo: fede1024/rust-rdkafka#359 (comment)

[jszwedko]

@DarkWanderer apologies for the delay in this. Could you link to where rdkafka recommends the use of cmake to resolve the issue you are seeing?

No problem. Here is a link to the comment in rust-rdkafka repo: fede1024/rust-rdkafka#359 (comment)

Gotcha, thanks! I see. My one concern is that librdkafka documents cmake support as experimental but in practice it seems to have worked for us as we do use it for the musl-based builds. Given that, I think it's ok to attempt making the default for Vector's cross compilation. I'll see about getting this integrated. Thanks for the discussion!

[jszwedko]

It actually looks like we already include this feature in our release builds:

vector/Cargo.toml

Lines 407 to 416 in cb937fb

	target-aarch64-unknown-linux-gnu = ["api", "api-client", "enrichment-tables", "rdkafka?/cmake_build", "sinks", "sources", "sources-dnstap", "transforms", "unix", "vrl-cli", "enterprise"]
	target-aarch64-unknown-linux-musl = ["api", "api-client", "enrichment-tables", "rdkafka?/cmake_build", "sinks", "sources", "sources-dnstap", "transforms", "unix", "vrl-cli", "enterprise"]
	target-armv7-unknown-linux-gnueabihf = ["api", "api-client", "enrichment-tables", "rdkafka?/cmake_build", "sinks", "sources", "sources-dnstap", "transforms", "unix", "vrl-cli", "enterprise"]
	target-armv7-unknown-linux-musleabihf = ["api", "api-client", "rdkafka?/cmake_build", "enrichment-tables", "sinks", "sources", "sources-dnstap", "transforms", "vrl-cli", "enterprise"]
	target-x86_64-unknown-linux-gnu = ["api", "api-client", "rdkafka?/cmake_build", "enrichment-tables", "sinks", "sources", "sources-dnstap", "transforms", "unix", "rdkafka?/gssapi-vendored", "vrl-cli", "enterprise"]
	target-x86_64-unknown-linux-musl = ["api", "api-client", "rdkafka?/cmake_build", "enrichment-tables", "sinks", "sources", "sources-dnstap", "transforms", "unix", "vrl-cli", "enterprise"]
	# Does not currently build
	target-powerpc64le-unknown-linux-gnu = ["api", "api-client", "enrichment-tables", "rdkafka?/cmake_build", "sinks", "sources", "sources-dnstap", "transforms", "unix", "vrl-cli", "enterprise"]
	# Currently doesn't build due to lack of support for 64-bit atomics
	target-powerpc-unknown-linux-gnu = ["api", "api-client", "enrichment-tables", "rdkafka?/cmake_build", "sinks", "sources", "sources-dnstap", "transforms", "unix", "vrl-cli", "enterprise"]

@DarkWanderer you should be able to use one of those features when building to compile in support (e.g. cargo build --no-default-features --features target-x86_64-unknown-linux-gnu.

I'll close this out since it seems to be resolved but let me know if you have any trouble with it!

[StephenWakely]

For what its worth, the docs build uses the docs feature which doesn't include rdkafka?/gssapi-vendored since libsasl is not available in the docs build environment.

[DarkWanderer]

I'll close this out since it seems to be resolved but let me know if you have any trouble with it!

Wouldn't say I have trouble - it's just having to specify additional options to get a debug binary - and having cargo build fail by default - feels a bit... unconventional.

Not a big deal though and certainly your choice - thanks for looking into it!

[jszwedko]

I'll close this out since it seems to be resolved but let me know if you have any trouble with it!

Wouldn't say I have trouble - it's just having to specify additional options to get a debug binary - and having cargo build fail by default - feels a bit... unconventional.

Not a big deal though and certainly your choice - thanks for looking into it!

Yeah, we would like to do better with the default features but a project like Vector really needs platform-specific features, which cargo doesn't support yet (rust-lang/cargo#1197).