processor_tda: Implement Topological Data Analysis (TDA) plugin for metrics by cosmo0920 · Pull Request #11250 · fluent/fluent-bit

cosmo0920 · 2025-12-03T07:23:25Z

This PR introduces a new processor plugin, tda, which performs Topological Data Analysis (TDA) on stream metrics using persistent homology.

The plugin aggregates incoming counters, gauges, and untyped metrics into a unified n-dimensional feature vector, maintains a sliding window, and utilizes a C-wrapped version of Ripser to compute Betti numbers.

Implementation Details:

Metric Aggregation & Normalization:
Multiple metric streams are mapped to a fixed feature dimension. To handle varying magnitudes and bursty traffic:
- Counters are converted to rates (differentiated against the previous snapshot).
- Values are normalized using log1p (natural logarithm of 1 + magnitude) to dampen dynamic range before distance calculation.
Sliding Window & Phase Space Reconstruction:
The plugin keeps a ring buffer of these vectors. Before processing, it optionally applies Delay Embedding (see below) to reconstruct the phase space geometry.
Persistent Homology via Ripser:
A dense Euclidean distance matrix is computed from the window. Ripser determines the persistence intervals, which are summarized into Betti numbers exported as new gauges:
- fluentbit.tda.betti0: Connected components (clusters).
- fluentbit.tda.betti1: Loops/Cycles (recurrence).
- fluentbit.tda.betti2: Voids (higher-order structures).

Delay Embedding (Takens' Theorem):

This plugin supports an optional delay embedding [2] of the aggregated metric vectors. When embed_dim > 1, we reconstruct the state space vectors $x_t$ as:

$$x_t \to (x_t, x_{t-\tau}, \dots, x_{t-(m-1)\tau})$$

Where:

$m =$ embed_dim
$\tau =$ embed_delay

This transformation allows the processor to detect cyclic or quasi-periodic regimes (loops in the trajectory) even from limited metric dimensions. These loops translate into $H_1$ features in the persistent homology. If embed_dim = 1 (default), the behavior falls back to the original "no embedding" mode.

Motivation:

TDA and persistent homology can help reveal hidden order, phase transitions, or subtle cyclic behaviors in complex systems that are not easily visible from raw time series or standard statistical aggregates. Similar approaches have been explored in condensed matter physics [1] for detecting phase transitions.

Configuration Options:

window_size (int, default: 60): Number of samples to keep in the TDA sliding window.
min_points (int, default: 10): Minimum number of samples required before running Ripser.
embed_dim (int, default: 3): Delay embedding dimension ($m$). Set to 1 to disable.
embed_delay (int, default: 1): Lag ($\tau$) in samples between successive delays.
threshold (double, default: 0): Distance scale selector. 0 enables auto multi-quantile scan; (0,1) uses the specific quantile.

References:

Donato, I., Gori, M., & Sarti, A. (2016). Persistent homology analysis of phase transitions. Physical Review E, 93, 052138.
F. Takens, "Detecting strange attractors in turbulence", in D. Rand and L.-S. Young (eds.), Dynamical Systems and Turbulence, Lecture Notes in Mathematics, vol. 898, Springer, 1981, pp. 366-381.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change

service:
  http_server: On
  http_port: 2021
pipeline:
    inputs:
      - name: dummy
        tag: log.raw
        samples: 10000
      - name: fluentbit_metrics
        tag: metrics.raw

        processors:
          metrics:
            - name: metrics_selector
              metric_name: /process_start_time_seconds/
              action: exclude
            - name: metrics_selector
              metric_name: /build_info/
              action: exclude
            - name: tda

    outputs:
      - name: stdout
        match: '*'

Additional Log:

2025-12-03T07:27:16.013990065Z fluentbit_tda_betti0 = 39
2025-12-03T07:27:16.013990065Z fluentbit_tda_betti1 = 7
2025-12-03T07:27:16.013990065Z fluentbit_tda_betti2 = 0
[2025/12/03 16:27:16.930210000] [error] [net] TCP connection failed: localhost:8443 (Connection refused)
[2025/12/03 16:27:16.930442000] [error] [net] TCP connection failed: localhost:8443 (Connection refused)
[2025/12/03 16:27:16.930461000] [error] [output:http:http.0] no upstream connections available to localhost:8443
[2025/12/03 16:27:16.930554000] [ warn] [engine] failed to flush chunk '30288-1764746835.908400000.flb', retry in 9 seconds: task_id=5, input=dummy.0 > output=http.0 (out_id=0)
2025-12-03T07:27:18.017259794Z fluentbit_tda_betti0 = 40
2025-12-03T07:27:18.017259794Z fluentbit_tda_betti1 = 7
2025-12-03T07:27:18.017259794Z fluentbit_tda_betti2 = 0
2025-12-03T07:27:20.024738944Z fluentbit_tda_betti0 = 41
2025-12-03T07:27:20.024738944Z fluentbit_tda_betti1 = 7
2025-12-03T07:27:20.024738944Z fluentbit_tda_betti2 = 1
[2025/12/03 16:27:21.995837000] [ info] [output:http:http.0] localhost:8443, HTTP status=200
{"status":"ok","errors":false}
2025-12-03T07:27:22.033923596Z fluentbit_tda_betti0 = 42
2025-12-03T07:27:22.033923596Z fluentbit_tda_betti1 = 7
2025-12-03T07:27:22.033923596Z fluentbit_tda_betti2 = 0
[2025/12/03 16:27:23.605981000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:23.606029000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:23.606022000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:23.606048000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:23.606089000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:23.606101000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:23.606148000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:23.606162000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:23.606243000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:23.606255000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:23.606328000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:23.606340000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:23.606400000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:23.606412000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:23.606437000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:23.606458000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:23.606472000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:23.606471000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:23.606560000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:23.606578000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:23.606593000] [error] [engine] chunk '30288-1764746830.908242000.flb' cannot be retried: task_id=0, input=dummy.0 > output=http.0
[2025/12/03 16:27:23.606725000] [ warn] [engine] failed to flush chunk '30288-1764746841.908663000.flb', retry in 10 seconds: task_id=8, input=dummy.0 > output=http.0 (out_id=0)
[2025/12/03 16:27:23.606780000] [ warn] [engine] failed to flush chunk '30288-1764746840.908811000.flb', retry in 6 seconds: task_id=13, input=dummy.0 > output=http.0 (out_id=0)
[2025/12/03 16:27:23.606825000] [ warn] [engine] failed to flush chunk '30288-1764746838.907344000.flb', retry in 10 seconds: task_id=11, input=dummy.0 > output=http.0 (out_id=0)
[2025/12/03 16:27:23.606845000] [error] [engine] chunk '30288-1764746827.908136000.flb' cannot be retried: task_id=2, input=dummy.0 > output=http.0
[2025/12/03 16:27:23.606905000] [error] [engine] chunk '30288-1764746831.908588000.flb' cannot be retried: task_id=6, input=dummy.0 > output=http.0
[2025/12/03 16:27:23.606941000] [error] [engine] chunk '30288-1764746828.909077000.flb' cannot be retried: task_id=3, input=dummy.0 > output=http.0
[2025/12/03 16:27:23.606991000] [error] [engine] chunk '30288-1764746832.908679000.flb' cannot be retried: task_id=7, input=dummy.0 > output=http.0
[2025/12/03 16:27:23.607060000] [ warn] [engine] failed to flush chunk '30288-1764746839.908556000.flb', retry in 8 seconds: task_id=12, input=dummy.0 > output=http.0 (out_id=0)
[2025/12/03 16:27:23.607112000] [ warn] [engine] failed to flush chunk '30288-1764746837.908369000.flb', retry in 6 seconds: task_id=10, input=dummy.0 > output=http.0 (out_id=0)
[2025/12/03 16:27:23.932627000] [error] [net] TCP connection failed: localhost:8443 (Connection refused)
[2025/12/03 16:27:23.932773000] [error] [net] TCP connection failed: localhost:8443 (Connection refused)
[2025/12/03 16:27:23.932794000] [error] [output:http:http.0] no upstream connections available to localhost:8443
[2025/12/03 16:27:24.30227000] [ warn] [engine] failed to flush chunk '30288-1764746842.908959000.flb', retry in 10 seconds: task_id=0, input=dummy.0 > output=http.0 (out_id=0)
2025-12-03T07:27:24.030021859Z fluentbit_tda_betti0 = 43
2025-12-03T07:27:24.030021859Z fluentbit_tda_betti1 = 9
2025-12-03T07:27:24.030021859Z fluentbit_tda_betti2 = 0
[2025/12/03 16:27:25.777729000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:25.777762000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:25.777780000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:25.777804000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:25.777824000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:25.777842000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:25.777864000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:25.777883000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:25.777916000] [error] [engine] chunk '30288-1764746826.908579000.flb' cannot be retried: task_id=1, input=dummy.0 > output=http.0
[2025/12/03 16:27:25.777955000] [error] [engine] chunk '30288-1764746833.907394000.flb' cannot be retried: task_id=9, input=dummy.0 > output=http.0
[2025/12/03 16:27:25.777981000] [error] [engine] chunk '30288-1764746829.908135000.flb' cannot be retried: task_id=4, input=dummy.0 > output=http.0
[2025/12/03 16:27:25.778051000] [ warn] [engine] failed to flush chunk '30288-1764746843.907694000.flb', retry in 11 seconds: task_id=2, input=dummy.0 > output=http.0 (out_id=0)
[2025/12/03 16:27:25.918758000] [error] [net] TCP connection failed: localhost:8443 (Connection refused)
[2025/12/03 16:27:25.918857000] [error] [net] TCP connection failed: localhost:8443 (Connection refused)
[2025/12/03 16:27:25.918872000] [error] [output:http:http.0] no upstream connections available to localhost:8443
[2025/12/03 16:27:26.13749000] [ warn] [engine] failed to flush chunk '30288-1764746844.908008000.flb', retry in 8 seconds: task_id=1, input=dummy.0 > output=http.0 (out_id=0)
[2025/12/03 16:27:26.25748000] [error] [net] TCP connection failed: localhost:8443 (Connection refused)
[2025/12/03 16:27:26.25885000] [error] [net] TCP connection failed: localhost:8443 (Connection refused)
[2025/12/03 16:27:26.25899000] [error] [output:http:http.0] no upstream connections available to localhost:8443
[2025/12/03 16:27:26.25923000] [error] [engine] chunk '30288-1764746835.908400000.flb' cannot be retried: task_id=5, input=dummy.0 > output=http.0
2025-12-03T07:27:26.013531523Z fluentbit_tda_betti0 = 44
2025-12-03T07:27:26.013531523Z fluentbit_tda_betti1 = 9
2025-12-03T07:27:26.013531523Z fluentbit_tda_betti2 = 0
[2025/12/03 16:27:27.572929000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:27.572971000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:27.572986000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:27.573010000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:27.573137000] [ warn] [engine] failed to flush chunk '30288-1764746845.906586000.flb', retry in 9 seconds: task_id=3, input=dummy.0 > output=http.0 (out_id=0)
[2025/12/03 16:27:27.573176000] [error] [engine] chunk '30288-1764746834.908597000.flb' cannot be retried: task_id=14, input=dummy.0 > output=http.0
[2025/12/03 16:27:27.935499000] [error] [net] TCP connection failed: localhost:8443 (Connection refused)
[2025/12/03 16:27:27.935638000] [error] [net] TCP connection failed: localhost:8443 (Connection refused)
[2025/12/03 16:27:27.935659000] [error] [output:http:http.0] no upstream connections available to localhost:8443
[2025/12/03 16:27:28.50004000] [ warn] [engine] failed to flush chunk '30288-1764746846.908355000.flb', retry in 10 seconds: task_id=4, input=dummy.0 > output=http.0 (out_id=0)
2025-12-03T07:27:28.049800623Z fluentbit_tda_betti0 = 45
2025-12-03T07:27:28.049800623Z fluentbit_tda_betti1 = 10
2025-12-03T07:27:28.049800623Z fluentbit_tda_betti2 = 1
[2025/12/03 16:27:29.881491000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:29.881542000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:29.881535000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:29.881565000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:29.881633000] [error] [http_client] broken connection to localhost:8443 ?
[2025/12/03 16:27:29.881648000] [error] [output:http:http.0] could not flush records to localhost:8443 (http_do=-1)
[2025/12/03 16:27:29.881736000] [error] [engine] chunk '30288-1764746840.908811000.flb' cannot be retried: task_id=13, input=dummy.0 > output=http.0
[2025/12/03 16:27:29.881782000] [error] [engine] chunk '30288-1764746837.908369000.flb' cannot be retried: task_id=10, input=dummy.0 > output=http.0
[2025/12/03 16:27:29.881873000] [ warn] [engine] failed to flush chunk '30288-1764746847.906725000.flb', retry in 9 seconds: task_id=5, input=dummy.0 > output=http.0 (out_id=0)
[2025/12/03 16:27:29.926923000] [error] [net] TCP connection failed: localhost:8443 (Connection refused)
[2025/12/03 16:27:29.927037000] [error] [net] TCP connection failed: localhost:8443 (Connection refused)
[2025/12/03 16:27:29.927068000] [error] [output:http:http.0] no upstream connections available to localhost:8443
[2025/12/03 16:27:30.37614000] [ warn] [engine] failed to flush chunk '30288-1764746848.907077000.flb', retry in 11 seconds: task_id=6, input=dummy.0 > output=http.0 (out_id=0)
2025-12-03T07:27:30.037452292Z fluentbit_tda_betti0 = 46
2025-12-03T07:27:30.037452292Z fluentbit_tda_betti1 = 12
2025-12-03T07:27:30.037452292Z fluentbit_tda_betti2 = 1
[2025/12/03 16:27:30.937891000] [error] [net] TCP connection failed: localhost:8443 (Connection refused)
[2025/12/03 16:27:30.938089000] [error] [net] TCP connection failed: localhost:8443 (Connection refused)
[2025/12/03 16:27:30.938120000] [error] [output:http:http.0] no upstream connections available to localhost:8443
[2025/12/03 16:27:30.938185000] [ warn] [engine] failed to flush chunk '30288-1764746849.906988000.flb', retry in 11 seconds: task_id=7, input=dummy.0 > output=http.0 (out_id=0)
2025-12-03T07:27:32.060665835Z fluentbit_tda_betti0 = 47
2025-12-03T07:27:32.060665835Z fluentbit_tda_betti1 = 12
2025-12-03T07:27:32.060665835Z fluentbit_tda_betti2 = 2
2025-12-03T07:27:34.073613867Z fluentbit_tda_betti0 = 48
2025-12-03T07:27:34.073613867Z fluentbit_tda_betti1 = 13
2025-12-03T07:27:34.073613867Z fluentbit_tda_betti2 = 2
[2025/12/03 16:27:36.264575000] [ info] [output:http:http.0] localhost:8443, HTTP status=200
{"status":"ok","errors":false}
[2025/12/03 16:27:36.264791000] [ info] [engine] flush chunk '30288-1764746839.908556000.flb' succeeded at retry 1: task_id=12, input=dummy.0 > output=http.0 (out_id=0)
2025-12-03T07:27:36.082592852Z fluentbit_tda_betti0 = 49
2025-12-03T07:27:36.082592852Z fluentbit_tda_betti1 = 13
2025-12-03T07:27:36.082592852Z fluentbit_tda_betti2 = 2
2025-12-03T07:27:38.070223396Z fluentbit_tda_betti0 = 50
2025-12-03T07:27:38.070223396Z fluentbit_tda_betti1 = 13
2025-12-03T07:27:38.070223396Z fluentbit_tda_betti2 = 1
2025-12-03T07:27:40.066529659Z fluentbit_tda_betti0 = 51
2025-12-03T07:27:40.066529659Z fluentbit_tda_betti1 = 13
2025-12-03T07:27:40.066529659Z fluentbit_tda_betti2 = 1
[2025/12/03 16:27:41.265885000] [ info] [output:http:http.0] localhost:8443, HTTP status=200
{"status":"ok","errors":false}
2025-12-03T07:27:42.072408891Z fluentbit_tda_betti0 = 52
2025-12-03T07:27:42.072408891Z fluentbit_tda_betti1 = 13
2025-12-03T07:27:42.072408891Z fluentbit_tda_betti2 = 2
2025-12-03T07:27:44.090558970Z fluentbit_tda_betti0 = 53
2025-12-03T07:27:44.090558970Z fluentbit_tda_betti1 = 13
2025-12-03T07:27:44.090558970Z fluentbit_tda_betti2 = 2
[2025/12/03 16:27:46.264449000] [ info] [output:http:http.0] localhost:8443, HTTP status=200
{"status":"ok","errors":false}
2025-12-03T07:27:46.100740206Z fluentbit_tda_betti0 = 54
2025-12-03T07:27:46.100740206Z fluentbit_tda_betti1 = 13
2025-12-03T07:27:46.100740206Z fluentbit_tda_betti2 = 2
2025-12-03T07:27:48.094721175Z fluentbit_tda_betti0 = 55
2025-12-03T07:27:48.094721175Z fluentbit_tda_betti1 = 13
2025-12-03T07:27:48.094721175Z fluentbit_tda_betti2 = 2
2025-12-03T07:27:50.083926971Z fluentbit_tda_betti0 = 56
2025-12-03T07:27:50.083926971Z fluentbit_tda_betti1 = 13
2025-12-03T07:27:50.083926971Z fluentbit_tda_betti2 = 2
[2025/12/03 16:27:51.265835000] [ info] [output:http:http.0] localhost:8443, HTTP status=200
{"status":"ok","errors":false}
[2025/12/03 16:27:51.266086000] [ info] [engine] flush chunk '30288-1764746841.908663000.flb' succeeded at retry 1: task_id=8, input=dummy.0 > output=http.0 (out_id=0)
2025-12-03T07:27:52.104521303Z fluentbit_tda_betti0 = 56
2025-12-03T07:27:52.104521303Z fluentbit_tda_betti1 = 13
2025-12-03T07:27:52.104521303Z fluentbit_tda_betti2 = 2
2025-12-03T07:27:54.104147236Z fluentbit_tda_betti0 = 56
2025-12-03T07:27:54.104147236Z fluentbit_tda_betti1 = 12
2025-12-03T07:27:54.104147236Z fluentbit_tda_betti2 = 2
[2025/12/03 16:27:56.264185000] [ info] [output:http:http.0] localhost:8443, HTTP status=200
{"status":"ok","errors":false}
[2025/12/03 16:27:56.264291000] [ info] [engine] flush chunk '30288-1764746838.907344000.flb' succeeded at retry 1: task_id=11, input=dummy.0 > output=http.0 (out_id=0)
2025-12-03T07:27:56.107265131Z fluentbit_tda_betti0 = 56
2025-12-03T07:27:56.107265131Z fluentbit_tda_betti1 = 12
2025-12-03T07:27:56.107265131Z fluentbit_tda_betti2 = 2
2025-12-03T07:27:58.093616840Z fluentbit_tda_betti0 = 56
2025-12-03T07:27:58.093616840Z fluentbit_tda_betti1 = 12
2025-12-03T07:27:58.093616840Z fluentbit_tda_betti2 = 2
2025-12-03T07:28:00.088709102Z fluentbit_tda_betti0 = 56
2025-12-03T07:28:00.088709102Z fluentbit_tda_betti1 = 12
2025-12-03T07:28:00.088709102Z fluentbit_tda_betti2 = 2
[2025/12/03 16:28:01.264444000] [ info] [output:http:http.0] localhost:8443, HTTP status=200
{"status":"ok","errors":false}
2025-12-03T07:28:02.097946671Z fluentbit_tda_betti0 = 56
2025-12-03T07:28:02.097946671Z fluentbit_tda_betti1 = 11
2025-12-03T07:28:02.097946671Z fluentbit_tda_betti2 = 2
2025-12-03T07:28:04.093339307Z fluentbit_tda_betti0 = 56
2025-12-03T07:28:04.093339307Z fluentbit_tda_betti1 = 12
2025-12-03T07:28:04.093339307Z fluentbit_tda_betti2 = 2
[2025/12/03 16:28:06.264442000] [ info] [output:http:http.0] localhost:8443, HTTP status=200
{"status":"ok","errors":false}
[2025/12/03 16:28:06.264618000] [ info] [engine] flush chunk '30288-1764746844.908008000.flb' succeeded at retry 1: task_id=1, input=dummy.0 > output=http.0 (out_id=0)
2025-12-03T07:28:06.093379532Z fluentbit_tda_betti0 = 56
2025-12-03T07:28:06.093379532Z fluentbit_tda_betti1 = 12
2025-12-03T07:28:06.093379532Z fluentbit_tda_betti2 = 2

For just one-time failing case, there is no increasing betti1 and betti2 metrics.
But intermittent failing cases just like the above, this higher order of metrics would raise and detected some of the "phase transitions" which means that there's no stable phase.

Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

This log is macOS's memory leak detector:

Process 30709 is not debuggable. Due to security restrictions, leaks can only show or save contents of readonly memory of restricted processes.

Process:         fluent-bit [30709]
Path:            /Users/USER/*/fluent-bit
Load Address:    0x104aa4000
Identifier:      fluent-bit
Version:         0
Code Type:       ARM64
Platform:        macOS
Parent Process:  leaks [30708]
Target Type:     live task

Date/Time:       2025-12-03 16:33:19.616 +0900
Launch Time:     2025-12-03 16:33:06.144 +0900
OS Version:      macOS 26.0.1 (25A362)
Report Version:  7
Analysis Tool:   /usr/bin/leaks

Physical footprint:         13.0M
Physical footprint (peak):  13.1M
Idle exit:                  untracked
----

leaks Report Version: 4.0, multi-line stacks
Process 30709: 2752 nodes malloced for 419 KB
Process 30709: 0 leaks for 0 total leaked bytes.

[2025/12/03 16:33:20] [engine] caught signal (SIGCONT)
[2025/12/03 16:33:20] Fluent Bit Dump

There's no leaks in this plugin.

Plus, there's no rules but the TDA metrics tells there's something happens with betti2 and betti1 metrics with non-zeros:

This metrics' detector is different direction to lighten in the depth of anomaly detections.

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Run local packaging test showing all targets (including any new ones) build.
Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

fluent/fluent-bit-docs#2277

Backporting

Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

New Features
- Optional Ripser integration for persistent homology and a new TDA processor that emits Betti metrics (betti0/betti1/betti2).
API
- Public Ripser C API and wrapper to compute Betti numbers and persistent intervals from dense distance matrices; public headers and installable ripser headers added.
Documentation
- Added Ripser README, license, and contributing guidance.
Tests
- New unit test validating Ripser Betti computations.
Chores
- Build and Docker flags to enable/disable Ripser and wire install/build steps.

coderabbitai · 2025-12-03T07:23:41Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds Ripser v1.2.1 as an optional bundled library, exposes a C wrapper and C++ integration, introduces a new TDA processor plugin that computes Betti numbers from time-series via delay embedding, and wires build, packaging, tests, and header installation to conditionally include Ripser support.

Changes

Cohort / File(s)	Summary
Top-level CMake & options `CMakeLists.txt`, `cmake/libraries.cmake`, `cmake/plugins_options.cmake`, `src/CMakeLists.txt`, `include/CMakeLists.txt`	Add FLB_USE_RIPSER/FLB_RIPSER detection and public option, define `FLB_PATH_LIB_RIPSER`, add `FLB_PROCESSOR_TDA` option, set `FLB_HAVE_RIPSER` when enabled, conditional `add_subdirectory(ripser)`, and enforce C++11 for Ripser paths.
Bundled Ripser library `lib/ripser-1.2.1/*`	Add Ripser 1.2.1 sources, build files (CMakeLists/Makefile), license/CONTRIBUTING, README, .gitignore/.gitmodules, examples, and define `ripser-static` target.
Ripser wrapper & integration `include/fluent-bit/ripser/flb_ripser_wrapper.h`, `src/ripser/flb_ripser_wrapper.cpp`, `src/ripser/CMakeLists.txt`, `lib/ripser-1.2.1/ripser_internal.hpp`	Add public C API header for Betti/intervals, implement C++ wrapper converting dense matrices to Ripser formats, interval filtering/bridging, two exported functions for Betti and intervals, and add `flb-ripser-wrapper-static` target linking to `ripser-static`.
Processor plugin (TDA) `plugins/processor_tda/tda.h`, `plugins/processor_tda/tda.c`, `plugins/processor_tda/CMakeLists.txt`, `plugins/CMakeLists.txt`	Add `tda` processor plugin gated by `FLB_RIPSER` implementing sliding windows, grouping, delay embedding, dense distance construction, Ripser-driven Betti computation, and register `processor_tda_plugin`.
Tests `tests/internal/ripser.c`, `tests/internal/CMakeLists.txt`	Add unit test `test_ripser_betti_circle()` and conditionally include ripser test source when `FLB_RIPSER` is enabled.
Packaging & Docker `dockerfiles/Dockerfile.centos7`, `packaging/distros/centos/Dockerfile`	Add `FLB_RIPSER` build ARG/ENV in CentOS Dockerfiles, propagate through stages, pass `-DFLB_RIPSER="${FLB_RIPSER}"` to CMake; centos7 Dockerfile sets `FLB_RIPSER=Off` in build.
Install & headers `include/CMakeLists.txt`, `include/fluent-bit/ripser/*.h`	Install Ripser wrapper headers to `${FLB_INSTALL_INCLUDEDIR}/fluent-bit/ripser/` when `FLB_RIPSER` is enabled.
Misc / examples `lib/ripser-1.2.1/examples/*`	Add example distance-matrix data files and Ripser README documentation.

Sequence Diagram(s)

sequenceDiagram
    participant Metrics as Metrics Stream
    participant Processor as TDA Processor
    participant Window as Sliding Window
    participant Embed as Delay Embedding
    participant DistMat as Dense→Compressed Builder
    participant Ripser as Ripser Engine
    participant Export as Metrics Export

    Metrics->>Processor: incoming metric points
    Processor->>Window: append / rotate samples
    Window->>Processor: snapshot when window ready
    Processor->>Embed: build embedded vectors (m, τ)
    Embed->>DistMat: compute dense pairwise distances
    DistMat->>Ripser: convert to compressed & run
    Ripser-->>Processor: emit intervals / betti counts (via bridge)
    Processor->>Export: emit betti gauges (betti0, betti1, betti2)

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Suggested reviewers

koleini
patrick-stephens
fujimotos

Poem

🐰
I hopped through windows, stitched time's thread,
Counted loops where metrics tread,
Holes and cycles, small and grand,
Betti bells ring across the land —
A rabbit's joy in data spread.

🚥 Pre-merge checks | ✅ 2 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 8.27% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Merge Conflict Detection	⚠️ Warning	❌ Merge conflicts detected (23 files): ⚔️ `CMakeLists.txt` (content) ⚔️ `cmake/libbacktrace.cmake` (content) ⚔️ `cmake/libraries.cmake` (content) ⚔️ `cmake/plugins_options.cmake` (content) ⚔️ `dockerfiles/Dockerfile.centos7` (content) ⚔️ `include/CMakeLists.txt` (content) ⚔️ `packaging/distros/centos/Dockerfile` (content) ⚔️ `plugins/CMakeLists.txt` (content) ⚔️ `plugins/in_tail/tail_config.c` (content) ⚔️ `plugins/in_tail/tail_config.h` (content) ⚔️ `plugins/in_tail/tail_db.c` (content) ⚔️ `plugins/in_tail/tail_file.c` (content) ⚔️ `plugins/in_winevtlog/in_winevtlog.c` (content) ⚔️ `plugins/in_winevtlog/pack.c` (content) ⚔️ `plugins/in_winevtlog/winevtlog.c` (content) ⚔️ `plugins/in_winevtlog/winevtlog.h` (content) ⚔️ `src/CMakeLists.txt` (content) ⚔️ `src/flb_lock.c` (content) ⚔️ `src/flb_processor.c` (content) ⚔️ `src/tls/flb_tls.c` (content) ⚔️ `src/tls/openssl.c` (content) ⚔️ `tests/internal/CMakeLists.txt` (content) ⚔️ `tests/runtime_shell/processor_conditional.sh` (content) These conflicts must be resolved before merging into `master`.	Resolve conflicts locally and push changes to this branch.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: implementing a Topological Data Analysis (TDA) processor plugin for metrics, which aligns with the primary objective and major changes throughout the pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch cosmo0920-ripser-for-analysis

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>

This processor plugin performs Topological Data Analysis (TDA) on metrics using ripser, which computes persistent homology. The plugin aggregates incoming counters, gauges and untyped metrics into a 1-D time series, keeps a sliding window, builds a dense distance matrix and runs ripser through the new flb_ripser_* wrapper helpers. The resulting Betti numbers (currently betti0 and betti1) are exported as additional gauge metrics. TDA and persistent homology can help reveal hidden order or phase transitions in complex systems that are not easily visible from raw time series. Similar approaches have already been explored in condensed matter physics, for example: Donato, I., Gori, M., & Sarti, A. (2016). Persistent homology analysis of phase transitions. Physical Review E, 93, 052138. https://doi.org/10.1103/PhysRevE.93.052138 The TDA metrics processor now supports an optional delay embedding of the aggregated metric vectors before building the dense distance matrix used by Ripser. When `embed_dim > 1`, we reconstruct a Takens-style delay embedding x_t -> (x_t, x_{t-ﾏм, ..., x_{t-(m-1)ﾏм) over the sliding window, where `m = embed_dim` and `ﾏ= embed_delay`. Each embedded point is a flattened vector of size feature_dim ﾃm and we keep using an Euclidean distance on this reconstructed phase space. This makes the processor more sensitive to occasional cyclic / quasi- periodic regimes in the metric time series: loops in the reconstructed trajectory translate into H1 features in the persistent homology. When `embed_dim = 1`, the behaviour is unchanged and we fall back to the original "no embedding" mode. This change also adds two configuration options: - `embed_dim` (int, default: 3) Delay embedding dimension m. Set to 1 to disable delay embedding. - `embed_delay` (int, default: 1) Lag ﾏin samples between successive delays. The design follows the standard delay embedding approach from Takens' theorem, which shows that (under mild conditions) the attractor of an unknown dynamical system can be reconstructed from a single observed time series via delay coordinates. Reference - F. Takens, "Detecting strange attractors in turbulence", in D. Rand and L.-S. Young (eds.), Dynamical Systems and Turbulence, Lecture Notes in Mathematics, vol. 898, Springer, 1981, pp. 366-381. Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>

…tions Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>

github-actions bot added the docs-required label Dec 3, 2025

cosmo0920 had a problem deploying to pr December 3, 2025 07:23 — with GitHub Actions Failure

cosmo0920 force-pushed the cosmo0920-ripser-for-analysis branch from 47dccf4 to 327ad4a Compare December 3, 2025 07:24

cosmo0920 had a problem deploying to pr December 3, 2025 07:24 — with GitHub Actions Failure

cosmo0920 temporarily deployed to pr December 3, 2025 08:23 — with GitHub Actions Inactive

cosmo0920 force-pushed the cosmo0920-ripser-for-analysis branch from d7c8e49 to 162f01e Compare December 3, 2025 08:23

cosmo0920 temporarily deployed to pr December 3, 2025 08:23 — with GitHub Actions Inactive

cosmo0920 temporarily deployed to pr December 3, 2025 08:29 — with GitHub Actions Inactive

cosmo0920 temporarily deployed to pr December 3, 2025 08:33 — with GitHub Actions Inactive

cosmo0920 temporarily deployed to pr December 3, 2025 08:53 — with GitHub Actions Inactive

cosmo0920 force-pushed the cosmo0920-ripser-for-analysis branch from 7c7cad7 to 9f9d30b Compare December 3, 2025 11:47

cosmo0920 temporarily deployed to pr December 3, 2025 11:47 — with GitHub Actions Inactive

cosmo0920 force-pushed the cosmo0920-ripser-for-analysis branch from 99a04b4 to 78c17a2 Compare February 13, 2026 17:01

cosmo0920 temporarily deployed to pr February 13, 2026 17:01 — with GitHub Actions Inactive

lib: ripser: Make a wrapper friendly interface

c213e94

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>

cosmo0920 force-pushed the cosmo0920-ripser-for-analysis branch from 78c17a2 to 0639950 Compare February 13, 2026 17:11

cosmo0920 temporarily deployed to pr February 13, 2026 17:11 — with GitHub Actions Inactive

cosmo0920 temporarily deployed to pr February 13, 2026 17:33 — with GitHub Actions Inactive

cosmo0920 had a problem deploying to pr February 13, 2026 17:33 — with GitHub Actions Failure

cosmo0920 temporarily deployed to pr February 13, 2026 17:33 — with GitHub Actions Inactive

cosmo0920 added 6 commits February 14, 2026 10:26

ripser: provide ripser C wrapper

0515ad2

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>

tests: internal: ripser: Add internal test case for TDA library

5f76d4a

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>

lib: ripser: Fix MSVC errors in ripser's explicit template specializa…

64ce54d

…tions Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>

dockerfiles: Fix CentOS 7 build for disabling ripser support

98e436a

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>

packaging: centos: Handle ripser support properly

cde0ea1

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>

cosmo0920 force-pushed the cosmo0920-ripser-for-analysis branch from 0639950 to cde0ea1 Compare February 14, 2026 01:27

cosmo0920 temporarily deployed to pr February 14, 2026 01:28 — with GitHub Actions Inactive

edsiper merged commit d142e3f into master Feb 14, 2026
97 checks passed

edsiper deleted the cosmo0920-ripser-for-analysis branch February 14, 2026 01:33

cosmo0920 temporarily deployed to pr February 14, 2026 01:47 — with GitHub Actions Inactive

cosmo0920 had a problem deploying to pr February 14, 2026 01:48 — with GitHub Actions Failure

cosmo0920 temporarily deployed to pr February 14, 2026 01:48 — with GitHub Actions Inactive

mabrarov mentioned this pull request Feb 14, 2026

build: CMake project fixes for MS Visual C++ compiler #11397

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

processor_tda: Implement Topological Data Analysis (TDA) plugin for metrics#11250

processor_tda: Implement Topological Data Analysis (TDA) plugin for metrics#11250
edsiper merged 9 commits intomasterfrom
cosmo0920-ripser-for-analysis

cosmo0920 commented Dec 3, 2025 •

edited by coderabbitai bot

Loading

Uh oh!

coderabbitai bot commented Dec 3, 2025 •

edited

Loading

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Suggested reviewers

Poem

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

cosmo0920 commented Dec 3, 2025 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Suggested reviewers

Poem

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cosmo0920 commented Dec 3, 2025 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Dec 3, 2025 •

edited

Loading