in_tail: Use temp table on overflowing the limit of SQL statement

[cosmo0920]

in_tail plugins will fail to start when exceeded the number of placeholders for deleting inodes on DB.
So, this PR introduces an offloading operation which creates temp table and runs deletions with it.

Closes #11272

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Prepare massive amount of files.

mkdir -p /tmp/tailmass
for i in $(seq 1 55000); do
  echo "log $i" > /tmp/tailmass/file_$i.log
done

• Example configuration file for the change

$ /path/to/fluent-bit \
  -i tail -p path=/tmp/tailmass/*.log -p db=/tmp/tail.db \
  -o null -v

• Debug log output from testing the change

This change can detect this type of overflow and use temp database to drop the definitions beyond the number of limit of SQLite's placeholders.

<snip>
[2026/01/22 15:50:28.415179297] [ warn] [input:tail:tail.0] db: large file set detected (55000 files) exceeds SQLite variable limit (32766); using temp-table cleanup for stale inode deletion
[2026/01/22 15:50:28.507476246] [ info] [input:tail:tail.0] db: delete unmonitored stale inodes from the database: count=0

• Attached Valgrind output that shows no leaks or memory corruption was found

==1535682== 
==1535682== HEAP SUMMARY:
==1535682==     in use at exit: 0 bytes in 0 blocks
==1535682==   total heap usage: 1,325,123 allocs, 1,325,123 frees, 2,591,568,841 bytes allocated
==1535682== 
==1535682== All heap blocks were freed -- no leaks are possible
==1535682== 
==1535682== For lists of detected and suppressed errors, rerun with: -s
==1535682== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• Bug Fixes
  • Enhanced file monitoring performance when tracking large numbers of files through optimized database cleanup operations.
  • Improved stale file entry removal efficiency, enabling better scalability for extensive file tracking scenarios.
  • Better handling of database cleanup when no files are monitored.

[coderabbitai]

📝 Walkthrough

Walkthrough

The change addresses SQLite variable limit failures when monitoring large numbers of files (55,000+) by introducing an alternative stale inode cleanup path using a temporary table instead of a single NOT IN statement with thousands of placeholders.

Changes

Cohort / File(s)	Summary
Stale File Cleanup Refactoring plugins/in_tail/tail_db.c	Adds flb_tail_db_stale_file_delete_temp_table() function to perform database cleanup using a temporary table when file count exceeds SQLite's variable limit (SQLITE_LIMIT_VARIABLE_NUMBER). Integrates this path into existing cleanup flow, adjusts NOT IN SQL generation logic, handles zero-file case by deleting all entries, and includes inttypes.h for PRIu64 formatting in log messages.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested reviewers

• edsiper
• koleini
• fujimotos

Poem

🐰 twitches nose at database woes
When inodes grew large beyond SQLite's rows,
A temp table bounded, transactions quite sound,
Large file monitoring no longer downed!
The rabbit rejoices, cleanup paths flow. 🌟

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: implementing a temp table approach to handle SQL statement overflow when monitoring large numbers of files.
Linked Issues check	✅ Passed	The pull request fully addresses issue #11272 by implementing a temp-table cleanup path to avoid exceeding SQLite's variable limit when handling large file counts.
Out of Scope Changes check	✅ Passed	All changes are directly related to solving the SQLite variable limit issue for large file monitoring, with no out-of-scope modifications detected.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch cosmo0920-use-temp-table-on-overflowing-stmt-limit

---

No actionable comments were generated in the recent review. 🎉

🧹 Recent nitpick comments

plugins/in_tail/tail_db.c (2)

292-307: Consider using DROP TABLE IF EXISTS + CREATE instead of CREATE IF NOT EXISTS + DELETE.

The current approach creates the temp table if it doesn't exist, then unconditionally clears it. A simpler and more idiomatic alternative is to drop and recreate the temp table, which avoids the extra DELETE round-trip and is arguably clearer about intent.

That said, the current approach is functionally correct and the cost is negligible, so this is purely a style suggestion.

---

359-368: sqlite3_changes() called after COMMIT — works but is subtle.

sqlite3_changes() returns the row count from the most recently completed INSERT/UPDATE/DELETE. Since COMMIT is not a DML statement, the value should still reflect the preceding DELETE. This is correct per SQLite semantics, but the ordering dependency is non-obvious.

A safer alternative is to capture sqlite3_changes() immediately after the DELETE query (line 353) and before the COMMIT, which makes the intent explicit and resilient to future code insertions between DELETE and the changes read.

Proposed fix

     if (ret != FLB_OK) {
         flb_plg_error(ctx->ins, "db: cannot delete stale inodes using temp table");
         goto error;
     }
 
+    changes = sqlite3_changes(ctx->db->handler);
+
     ret = flb_sqldb_query(ctx->db, "COMMIT;", NULL, NULL);
     if (ret != FLB_OK) {
         flb_plg_error(ctx->ins, "db: cannot commit transaction for temp inode inserts");
         goto error;
     }
     txn_started = FLB_FALSE;
 
-    changes = sqlite3_changes(ctx->db->handler);
     flb_plg_info(ctx->ins, "db: delete unmonitored stale inodes from the database: count=%d",
                  changes);
     return 0;

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}