Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

in_premetheus_remote_write: Implement handler of payloads of prometheus remote write protocol #8725

Merged
merged 6 commits into from
May 16, 2024
Merged

in_premetheus_remote_write: Implement handler of payloads of prometheus remote write protocol #8725

merged 6 commits into from
May 16, 2024

Conversation

cosmo0920
Copy link
Contributor

@cosmo0920 cosmo0920 commented Apr 17, 2024 •
edited
Loading

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
$  bin/fluent-bit -i prometheus_remote_write -pport=8080 -phttp2=off -o stdout -v
  • Debug log output from testing the change
Fluent Bit v3.0.3
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/04/19 20:24:33] [ info] Configuration:
[2024/04/19 20:24:33] [ info]  flush time     | 1.000000 seconds
[2024/04/19 20:24:33] [ info]  grace          | 5 seconds
[2024/04/19 20:24:33] [ info]  daemon         | 0
[2024/04/19 20:24:33] [ info] ___________
[2024/04/19 20:24:33] [ info]  inputs:
[2024/04/19 20:24:33] [ info]      prometheus_remote_write
[2024/04/19 20:24:33] [ info] ___________
[2024/04/19 20:24:33] [ info]  filters:
[2024/04/19 20:24:33] [ info] ___________
[2024/04/19 20:24:33] [ info]  outputs:
[2024/04/19 20:24:33] [ info]      stdout.0
[2024/04/19 20:24:33] [ info] ___________
[2024/04/19 20:24:33] [ info]  collectors:
[2024/04/19 20:24:33] [ info] [fluent bit] version=3.0.3, commit=ec1c65f73a, pid=186265
[2024/04/19 20:24:33] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/04/19 20:24:33] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/04/19 20:24:33] [ info] [cmetrics] version=0.7.3
[2024/04/19 20:24:33] [ info] [ctraces ] version=0.4.0
[2024/04/19 20:24:33] [ info] [input:prometheus_remote_write:prometheus_remote_write.0] initializing
[2024/04/19 20:24:33] [ info] [input:prometheus_remote_write:prometheus_remote_write.0] storage_strategy='memory' (memory only)
[2024/04/19 20:24:33] [ info] [output:stdout:stdout.0] worker #0 started
[2024/04/19 20:24:33] [debug] [prometheus_remote_write:prometheus_remote_write.0] created event channels: read=21 write=22
[2024/04/19 20:24:33] [debug] [downstream] listening on 0.0.0.0:8080
[2024/04/19 20:24:33] [ info] [input:prometheus_remote_write:prometheus_remote_write.0] listening on 0.0.0.0:8080
[2024/04/19 20:24:33] [debug] [stdout:stdout.0] created event channels: read=24 write=25
[2024/04/19 20:24:33] [ info] [sp] stream processor started
[2024/04/19 20:24:38] [debug] [task] created task=0x5faf030 id=0 OK
2024-04-19T11:24:36.216000000Z fluentbit_uptime{__name__="fluentbit_uptime",hostname="cosmo-desktop2"} = 1
2024-04-19T11:24:35.170000000Z fluentbit_input_bytes_total{__name__="fluentbit_input_bytes_total",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_records_total{__name__="fluentbit_input_records_total",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:36.216000000Z fluentbit_input_metrics_scrapes_total{__name__="fluentbit_input_metrics_scrapes_total",name="fluentbit_metrics.0"} = 1
2024-04-19T11:24:35.170000000Z fluentbit_output_proc_records_total{__name__="fluentbit_output_proc_records_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_proc_bytes_total{__name__="fluentbit_output_proc_bytes_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_errors_total{__name__="fluentbit_output_errors_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_retries_total{__name__="fluentbit_output_retries_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_retries_failed_total{__name__="fluentbit_output_retries_failed_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_dropped_records_total{__name__="fluentbit_output_dropped_records_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_retried_records_total{__name__="fluentbit_output_retried_records_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:36.216000000Z fluentbit_process_start_time_seconds{__name__="fluentbit_process_start_time_seconds",hostname="cosmo-desktop2"} = 1713525875
2024-04-19T11:24:36.216000000Z fluentbit_build_info{__name__="fluentbit_build_info",hostname="cosmo-desktop2",version="3.0.3",os="linux"} = 1713525875
2024-04-19T11:24:36.216000000Z fluentbit_hot_reloaded_times{__name__="fluentbit_hot_reloaded_times",hostname="cosmo-desktop2"} = 0
2024-04-19T11:24:36.216000000Z fluentbit_storage_chunks{__name__="fluentbit_storage_chunks"} = 0
2024-04-19T11:24:36.216000000Z fluentbit_storage_mem_chunks{__name__="fluentbit_storage_mem_chunks"} = 0
2024-04-19T11:24:36.216000000Z fluentbit_storage_fs_chunks{__name__="fluentbit_storage_fs_chunks"} = 0
2024-04-19T11:24:36.216000000Z fluentbit_storage_fs_chunks_up{__name__="fluentbit_storage_fs_chunks_up"} = 0
2024-04-19T11:24:36.216000000Z fluentbit_storage_fs_chunks_down{__name__="fluentbit_storage_fs_chunks_down"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_ingestion_paused{__name__="fluentbit_input_ingestion_paused",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_overlimit{__name__="fluentbit_input_storage_overlimit",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_memory_bytes{__name__="fluentbit_input_storage_memory_bytes",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks{__name__="fluentbit_input_storage_chunks",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_up{__name__="fluentbit_input_storage_chunks_up",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_down{__name__="fluentbit_input_storage_chunks_down",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_busy{__name__="fluentbit_input_storage_chunks_busy",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_busy_bytes{__name__="fluentbit_input_storage_chunks_busy_bytes",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_upstream_total_connections{__name__="fluentbit_output_upstream_total_connections",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_upstream_busy_connections{__name__="fluentbit_output_upstream_busy_connections",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_chunk_available_capacity_percent{__name__="fluentbit_output_chunk_available_capacity_percent",name="prometheus_remote_write.0"} = 100
[2024/04/19 20:24:38] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2024/04/19 20:24:38] [debug] [out flush] cb_destroy coro_id=0
[2024/04/19 20:24:38] [debug] [task] destroy task=0x5faf030 (task_id=0)
2024-04-19T11:24:38.216000000Z fluentbit_uptime{__name__="fluentbit_uptime",hostname="cosmo-desktop2"} = 3
2024-04-19T11:24:35.170000000Z fluentbit_input_bytes_total{__name__="fluentbit_input_bytes_total",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_records_total{__name__="fluentbit_input_records_total",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:38.216000000Z fluentbit_input_metrics_scrapes_total{__name__="fluentbit_input_metrics_scrapes_total",name="fluentbit_metrics.0"} = 2
2024-04-19T11:24:37.278000000Z fluentbit_output_proc_records_total{__name__="fluentbit_output_proc_records_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:37.278000000Z fluentbit_output_proc_bytes_total{__name__="fluentbit_output_proc_bytes_total",name="prometheus_remote_write.0"} = 6175
2024-04-19T11:24:35.170000000Z fluentbit_output_errors_total{__name__="fluentbit_output_errors_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_retries_tota[2024/04/19 20:24:39] [debug] [task] created task=0x6180f00 id=0 OK
l{__name__="fluentbit_output_retries_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_retries_failed_total{__name__="fluentbit_output_retries_failed_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_dropped_records_total{__name__="fluentbit_output_dropped_records_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_retried_records_total{__name__="fluentbit_output_retried_records_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:38.216000000Z fluentbit_process_start_time_seconds{__name__="fluentbit_process_start_time_seconds",hostname="cosmo-desktop2"} = 1713525875
2024-04-19T11:24:38.216000000Z fluentbit_build_info{__name__="fluentbit_build_info",hostname="cosmo-desktop2",version="3.0.3",os="linux"} = 1713525875
2024-04-19T11:24:38.216000000Z fluentbit_hot_reloaded_times{__name__="fluentbit_hot_reloaded_times",hostname="cosmo-desktop2"} = 0
2024-04-19T11:24:38.216000000Z fluentbit_storage_chunks{__name__="fluentbit_storage_chunks"} = 0
2024-04-19T11:24:38.216000000Z fluentbit_storage_mem_chunks{__name__="fluentbit_storage_mem_chunks"} = 0
2024-04-19T11:24:38.216000000Z fluentbit_storage_fs_chunks{__name__="fluentbit_storage_fs_chunks"} = 0
2024-04-19T11:24:38.216000000Z fluentbit_storage_fs_chunks_up{__name__="fluentbit_storage_fs_chunks_up"} = 0
2024-04-19T11:24:38.216000000Z fluentbit_storage_fs_chunks_down{__name__="fluentbit_storage_fs_chunks_down"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_ingestion_paused{__name__="fluentbit_input_ingestion_paused",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_overlimit{__name__="fluentbit_input_storage_overlimit",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_memory_bytes{__name__="fluentbit_input_storage_memory_bytes",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks{__name__="fluentbit_input_storage_chunks",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_up{__name__="fluentbit_input_storage_chunks_up",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_down{__name__="fluentbit_input_storage_chunks_down",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_busy{__name__="fluentbit_input_storage_chunks_busy",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_busy_bytes{__name__="fluentbit_input_storage_chunks_busy_bytes",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:37.216000000Z fluentbit_output_upstream_total_connections{__name__="fluentbit_output_upstream_total_connections",name="prometheus_remote_write.0"} = 1
2024-04-19T11:24:37.278000000Z fluentbit_output_upstream_busy_connections{__name__="fluentbit_output_upstream_busy_connections",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:37.278000000Z fluentbit_output_chunk_available_capacity_percent{__name__="fluentbit_output_chunk_available_capacity_percent",name="prometheus_remote_write.0"} = 100
[2024/04/19 20:24:39] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2024/04/19 20:24:39] [debug] [task] destroy task=0x6180f00 (task_id=0)
[2024/04/19 20:24:39] [debug] [out flush] cb_destroy coro_id=1
^C[2024/04/19 20:24:40] [engine] caught signal (SIGINT)
[2024/04/19 20:24:40] [ warn] [engine] service will shutdown in max 5 seconds
[2024/04/19 20:24:40] [ info] [engine] service has stopped (0 pending tasks)
[2024/04/19 20:24:40] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/04/19 20:24:40] [ info] [output:stdout:stdout.0] thread worker #0 stopped
  • Attached Valgrind output that shows no leaks or memory corruption was found
==186265== 
==186265== HEAP SUMMARY:
==186265==     in use at exit: 0 bytes in 0 blocks
==186265==   total heap usage: 7,158 allocs, 7,158 frees, 3,974,699 bytes allocated
==186265== 
==186265== All heap blocks were freed -- no leaks are possible
==186265== 
==186265== For lists of detected and suppressed errors, rerun with: -s
==186265== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

fluent/fluent-bit-docs#1363

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@cosmo0920 cosmo0920 changed the title in_premetheus_remote_write: Implement ingestor of payloads of prometheus remote write protocol in_premetheus_remote_write: Implement handler of payloads of prometheus remote write protocol Apr 17, 2024
@edsiper
Copy link
Member

edsiper commented Apr 18, 2024

@cosmo0920 I see the metrics generated have an _ as a prefix, e.g:

2024-04-17T05:13:34.937000000Z _fluentbit_uptime{__name__="fluentbit_uptime",hostname="cosmo-desktop2"} = 1

maybe is something around the definition of the metrics name,subsystem,etc ?

@cosmo0920
Copy link
Contributor Author

@cosmo0920 I see the metrics generated have an _ as a prefix, e.g:

2024-04-17T05:13:34.937000000Z _fluentbit_uptime{__name__="fluentbit_uptime",hostname="cosmo-desktop2"} = 1

maybe is something around the definition of the metrics name,subsystem,etc ?

This could be generated from cmetrics but I'm not sure why the reason. I just following how to decode on the open telemetry payloads.

@cosmo0920 cosmo0920 force-pushed the cosmo0920-in_prometheus-remote-write branch from bae0fa5 to 0b714aa Compare April 19, 2024 11:19
@cosmo0920
Copy link
Contributor Author

I got it.

@cosmo0920 I see the metrics generated have an _ as a prefix, e.g:

2024-04-17T05:13:34.937000000Z _fluentbit_uptime{__name__="fluentbit_uptime",hostname="cosmo-desktop2"} = 1

maybe is something around the definition of the metrics name,subsystem,etc ?

This could be generated from cmetrics but I'm not sure why the reason. I just following how to decode on the open telemetry payloads.

This was generated from cmetrics' msgpack decoder:
fluent/cmetrics@8cb2739

@cosmo0920 cosmo0920 force-pushed the cosmo0920-in_prometheus-remote-write branch from 0b714aa to a2d1869 Compare April 22, 2024 07:54
@cosmo0920 cosmo0920 force-pushed the cosmo0920-in_prometheus-remote-write branch from a2d1869 to 3435eb7 Compare April 25, 2024 06:52
@cosmo0920
in_prometheus_remote_write: Use abbreviated names for promethues remo…
4781bdd 
…te write

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@cosmo0920 cosmo0920 force-pushed the cosmo0920-in_prometheus-remote-write branch from 336cc0b to 4781bdd Compare May 1, 2024 06:05
@cosmo0920
in_prometheus_remote_write: Remove a needless FIXME
04da871 
Because prometheus remote write just needs to decode the protobuf
payloads.

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@cosmo0920
in_prometheus_remote_write: Use one line declarations
1b985cc 
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@cosmo0920
in_prometheus_remote_write: Check return value from flb_io_net_write
95bfcf0 
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@cosmo0920
in_prometheus_remote_write: Add allocation check for mk_server
4dc32b4 
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@edsiper edsiper merged commit 7de2c45 into master May 16, 2024
45 of 46 checks passed
@edsiper edsiper deleted the cosmo0920-in_prometheus-remote-write branch May 16, 2024 02:57
@stevehipwell stevehipwell mentioned this pull request May 16, 2024
Closed
@BrewTestBot BrewTestBot mentioned this pull request May 21, 2024
Merged
markuman pushed a commit to markuman/fluent-bit that referenced this pull request May 29, 2024
@cosmo0920 @markuman
in_premetheus_remote_write: Implement handler of payloads of promethe…
65fc34d 
…us remote write protocol (fluent#8725)

 in_prometheus_remote_write: Implement prometheus remote write input plugin. This plugin is able to handle the following types currently:

- Counter
- Gauge
- Untyped
- Histogram

Summary type of metrics shouldn't be handled and decoded correctly for now.

---------

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
Signed-off-by: Markus Bergholz <git@osuv.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@edsiper edsiper edsiper requested changes

@fujimotos fujimotos Awaiting requested review from fujimotos fujimotos is a code owner

@niedbalski niedbalski Awaiting requested review from niedbalski niedbalski is a code owner

@patrick-stephens patrick-stephens Awaiting requested review from patrick-stephens patrick-stephens is a code owner

@celalettin1286 celalettin1286 Awaiting requested review from celalettin1286 celalettin1286 is a code owner

@leonardo-albertovich leonardo-albertovich Awaiting requested review from leonardo-albertovich leonardo-albertovich is a code owner

@koleini koleini Awaiting requested review from koleini koleini is a code owner

Assignees
No one assigned
Labels
docs-required
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@cosmo0920 @edsiper