Effective (and fast) S09 threading

[gluesmith2021]

• What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

"Improvement" or "bug fix", depending how we see it.

• What is the current behavior? (You can also link to an open issue here)

With THREADED=1, generating strings_bins in S09 is very slow, and get slower and slower as the number of generated files increases (gets slower than single-thread at some point)

Cause: one process is spawned per file to generate under strings_bins, and all their PIDs are kept in a list and not removed until all strings_bins are generated. Once the PIDs lists is longer than ${MAX_MOD_THREADS}*3, the max_pids_protection kicks in to make sure there aren't too many processes at once. Problem is, as the PIDs list gets longer, max_pids_protection takes longer to execute than the last spawned process to extract strings (mostly because of checking so many dead PIDs I guess). Since a new process is not spawned until max_pids_protection returns, what we have is a "main process" doing nothing but checking dead PIDs and short "job processes" running only once in a while.

There is also a potential issue (not observed for sure yet) with PID recycling over time. See other information below.

• What is the new behavior (if this is a feature change)? If possible add a screenshot.

String generation is now "really" multi-threaded and much faster, especially on large firmwares. To do so, the PID list is cleaned up from dead PID at every MAX_MOD_THREADS spawned process (in tests, most often all processes were dead and cleaned up, and sometimes there was 1 still running).

To do so, existing logic in S09 (in the second part of S09) was adapted and reused.

Here's an anecdotical benchmark performed on a firmware that I can't share.

• Ubuntu VM with 32 cores and 96 GB RAM
• EMBA running in a Docker container
• S09 generates about 13,000 files under strings_bins
• variables at S09 entry, if that matters:
  • THREADED: 1
  • QEMULATION: 1
  • SELECTED_MODULES: ("S06" "S08" "S09" "S24" "S25" "S26" "S115" "S116" "S118")
  • SBOM_UNTRACKED_FILES: 1
  • FIRMWARE: 1
  • RTOS: 0
• S09 happened to be running at the same time as S25 and S115, but even though this might blur timings somewhat, the improvement is clear.

Generating strings took 4 hours (almost exactly) before fix, and under 6 minutes after the fix. This is only for the string generation part. The remainder of S09 took about 20 minutes to execute in both cases (did not expect any change).

Also, looking at file access times in strings_bins folder, I can roughly compute the average throughput (how much files created per seconds) at various points of S09 execution:

• After fix: about 36 files/sec from start to end
• Before fix:
  • Around the 50th: 38
  • Around the 500th: 15
  • Around the 2,500th: 2.9
  • Around the 5,000th: 1.3
  • Around the 10,000th: 0.52
  • At the end (13,000 files): 0.34 (more than 100x slower than at the beginning)

Of course, extracted strings in strings_bins were the same before and after the fix.

• Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

No.

• Other information:

Cleaning up the list of running PIDs in S09 not only speeds up things, but helps to prevent an issue with recycled PIDs.

When S09, S25 and S115 are running at once, and even when S09 is completed, I noticed on my system (and the above test firmware) that there is about 500,000 processes created per hour. At this rate, PIDs are recycled about every 8 hours (pid_max in the docker container on my system is 4 million) . After the fix, S09 doesn't run for very long anymore, but it used to before the fix, and others modules such as S25 and S115 (and others that come after) can still run for a very long time.

Keeping lists of spawned (and dead) PIDs for several hours is thus getting in the "danger zone" of monitoring (or even killing) unrelated processes. I haven't observed this yet, but on a very large firmware this could happen. As such, there might be other instances, in other modules, where it might be safer to tidy up PIDs list as it's now done in S09.

[m-1-k-3]

Generating strings took 4 hours (almost exactly) before fix, and under 6 minutes after the fix.

holy shit ... I will test this ASAP
Thank your for fixing

[m-1-k-3]

nice catch ...

old
[*] Generate strings overview for static version analysis of 930 files ... Fri Feb 14 18:13:15 CET 2025
[*] Proceeding with version detection for 930 binary files .. Fri Feb 14 18:14:59 CET 2025

improved threading
[*] Generate strings overview for static version analysis of 930 files ... Fri Feb 14 18:37:44 CET 2025
[*] Proceeding with version detection for 930 binary files .. Fri Feb 14 18:37:53 CET 2025

Probably we have similar issues in other areas ...

[gluesmith2021]

nice catch ...

old
[*] Generate strings overview for static version analysis of 930 files ... Fri Feb 14 18:13:15 CET 2025
[*] Proceeding with version detection for 930 binary files .. Fri Feb 14 18:14:59 CET 2025

improved threading
[*] Generate strings overview for static version analysis of 930 files ... Fri Feb 14 18:37:44 CET 2025
[*] Proceeding with version detection for 930 binary files .. Fri Feb 14 18:37:53 CET 2025

Probably we have similar issues in other areas ...

Indeed, this issue may occur in other modules as well, but it might not be as obvious if the spawned processes take longer to execute than max_pids_protection() (either because they really run for a longer time, as with emulation, or because there isn't that many PID created, as with you test above). I haven't investigated further on other modules.

[m-1-k-3]

Thank you for fixing this beautiful bug. Could you please check my little comment? Afterwards we can merge it

[m-1-k-3]

Thank you for this fix and improvement