Draft: Introduce support for DPDK secondary mode for workers runmode #7283
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lukashino
force-pushed
the
feat/5203-primary-app-v1
branch
from
ee0ffa8 to
e043330
Compare
Up to now, DPDK workers run as pthreads. In DPDK terms, this has meant that they've been running as non-registered non-EAL threads. This limits funcionality of certain DPDK libraries (e.g. not using per-lcore cache for rte_mempool or rte_ring limitations). It is possible to register pthreads so they become registered non-EAL threads. This mitigates the aforementioned limitations. https://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#non-eal-pthread-support However, running Suricata as a secondary process would defy this approach. According to the DPDK documentation, registered non-EAL pthread behaves like a non-registered non-EAL pthread. Support for running Suricata as a secondary process is a planned extension of work. https://doc.dpdk.org/api/rte__lcore_8h.html#a71e6840cf1fe8950cce8fd88d0859bce
Flag `--libs-only-L` of pkg-config should output a path where libdpdk is stored. This usually happens when DPDK is installed manually (from source) into a custom location. On the other hand, there is no output when installing DPDK with a package manager. For that reason, the linker flag `-Wl,-R,` remain incomplete. The flag `-Wl,-R,` then binds incorrectly with the following output of pkg-config. The output of `pkg-config --libs libdpdk` contains linker flag `--as-need` to remove unnecessary libraries during the linking. It is silently skipped and the binary is linked with all of the DPDK libraries.
Implement secondary mode operation in Suricata. Receive packets over DPDK rings. To each worker assign a separate ring. Provide configuration options specific for the secondary mode along with possibility to specify arbitrary ring name. Add relevant documentation.
… as a secondary process
lukashino
force-pushed
the
feat/5203-primary-app-v1
branch
from
e043330 to
03ee36b
Compare
|
Warning: no commits in this PR have specified the following ticket(s):
Please update the commit(s) and submit a new PR. |
3 tasks
3 tasks
|
Closing and moving to #7433 |
Closed
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement secondary mode operation in Suricata. Receive
packets over DPDK rings where each worker has a separate RX/TX ring pair.
Provide configuration options specific for the secondary mode
along with a possibility to specify an arbitrary ring name.
Change threading from threads to DPDK lcores.
Add relevant documentation.
Add an application that passes traffic from NICs to Suricata, where the application runs as a primary process and Suricata as a secondary process.
Link to redmine ticket about Suricata secondary process support.
Link to redmine ticket about prefilter application.
This is a follow-up of #7128 which incorporates two Redmine tickets together, one for Suricata secondary mode support and the other for adding an app that would run as a primary process. The application is intended to be a base for the prefilter concept. There are already some concepts present in the code/settings (such as bypass mempool, bypass table, results ring, task ring) but are not used in any way at the moment. As of now, the prefilter only configures the provided NICs according to the YAML configuration and then passes packets to Suricata through DPDK rings. Both IDS and IPS modes are supported by the prefilter/Suricata application. The prefilter is intended to be architecturally flexible (not depending on any hardcoded module) but as of now, for some use-cases, the prefilter uses Suricata library. The plan is to have a shared DPDK configuration codebase for prefilter and Suricata. As of result, prior to the compilation of the prefilter app, Suricata needs to be first compiled and installed (together with headers and library). Makefile target
dpdk-apps-prereqshould solve the issue. After the Suricata library is installed, it needs to be added to the list of shared libraries (ldconfig) or at least editLD_LIBRARY_PATHvariable. Makefile targetdpdk-appscompiles the prefitler app. The finalprefilterbinary can be found indpdk/prefilter/buildfolder.Changelog from #7128:
uint32_t