From 27e7e76cc87c7df991aacd9ea74e83a772993679 Mon Sep 17 00:00:00 2001 From: Melissa Kilby Date: Mon, 3 Jul 2023 16:10:27 -0700 Subject: [PATCH] cleanup(proposals): clarify rules maturity framework and rules criteria Co-authored-by: Luca Guerra Signed-off-by: Melissa Kilby --- ...-adoption-management-maturity-framework.md | 51 ++++++++----------- 1 file changed, 22 insertions(+), 29 deletions(-) diff --git a/proposals/20230605-rules-adoption-management-maturity-framework.md b/proposals/20230605-rules-adoption-management-maturity-framework.md index 564b2c63a..2b243c36b 100644 --- a/proposals/20230605-rules-adoption-management-maturity-framework.md +++ b/proposals/20230605-rules-adoption-management-maturity-framework.md @@ -40,48 +40,35 @@ Adopters will be presented with clear primary use cases to better understand how ### Rules Maturity Framework -A rules maturity framework will be developed for Falco users to better facilitate the adoption of non-custom rules. This framework ensures a smooth transition for adopters, whether they use rules generically or for specific use cases. +A rules maturity framework will be developed for Falco users to better facilitate the adoption of non-custom rules. This framework ensures a smooth transition for adopters, whether they use rules generically or for specific use cases. A smooth adoption process is defined by making it easy for adopters to understand each rule and also gain an understanding of not just what the rule is doing, but also how beneficial it can be under various circumstances. Additionally, adopters should have a clear idea of which rules can likely be adopted as-is versus which rules may require engineering efforts to evaluate and adopt. The rules maturity framework shall be accompanied by a guide that provides clear and unambiguous criteria for categorizing a rule that both the reviewer and contributor should adhere to. -The rules maturity framework will align with the [status](https://github.com/falcosecurity/evolution/blob/main/REPOSITORIES.md#status) levels used within The Falco Project repositories, namely "Stable", "Incubating", "Sandbox" and "Deprecated" as well as the [scope](https://github.com/falcosecurity/evolution/blob/main/REPOSITORIES.md#scope) levels of "Core" and "Ecosystem" which indicate the role and responsibilities of each repository. +The rules maturity framework will align with the [status](https://github.com/falcosecurity/evolution/blob/main/REPOSITORIES.md#status) levels used within The Falco Project repositories, namely "Stable", "Incubating", "Sandbox" and "Deprecated". -When applied to the rules maturity framework, these levels signify the following, and the "status" level will be rebranded as "maturity". +Not every rule has the potential to evolve and reach the "Stable" level. This is because "Stable" rules should address a broader category of attacks rather than being overly specific and easily bypassed. However, this does not mean that very specific rules do not provide value; on the contrary, they can serve a very specific purpose. The intention of the new framework is to make it clearer for adopters to recognize that they need to evaluate such rules for their own environment. More details about this will be explained in the respective maturity level section. -#### `scope` - -The framework introduces the `scope` key to clearly and unambiguously indicate the relevance of the rules for community use cases. This key enables the categorization and classification of rules based on their applicability and alignment with specific user needs and scenarios. - -Levels: - -- **Core** indicates that the rule has undergone thorough evaluation by domain experts with hands-on production experience. These rules are deemed highly relevant to address broader threats and are recommended for customization to specific environments. They focus on universal system-level detections, such as more generic reverse shells or container escapes, providing a solid baseline for threat detection across diverse industries. Inherently, this bias works against the inclusion of more application-specific detections, as they may not achieve broad relevance or applicability. However, to mitigate this bias, a grey area will be reserved, allowing for case-by-case judgments to be made. -- **Ecosystem** indicates that the rules have been identified by experts as catering to more specific use cases, which may or may not be relevant for each adopter. However, these rules address relevant threats and provide a certain level of robustness guarantee. This category is expected to encompass many more application-specific rules. -- **Empty** indicates that the potential for broader usefulness and relevance of the rules is currently being assessed. These rules can serve as an inspiration for future development and may be further evaluated and refined for wider adoption. - - -#### `maturity` - -Each rule has the potential to evolve and reach the "Stable" level. However, it is important for "Stable" rules to address a broader category of attacks rather than being overly specific and easily bypassed. There is certainly some degree of overlap with scope levels in this regard. The guide for rule contributions will provide clear and unambiguous criteria that both the reviewer and contributor should adhere to. - -The guide will outline the specific requirements and expectations for rule contributions, ensuring that they undergo thorough review and meet the necessary criteria for stability and effectiveness. This will help maintain a high standard of rule quality and ensure that rules provide meaningful detection capabilities for a wide range of relevant security threats. - Levels: -- **Stable** (enabled by default) indicates that the rule has been thoroughly evaluated by experts in the field who have hands-on production experience. These experts have determined that the rules embody the utmost best practices and demonstrate optimal robustness in terms of the potential for attackers to bypass Falco, taking into account Falco's current capabilities. -- **Incubating** (disabled by default) indicates that the rules address relevant threats, provide a certain level of robustness guarantee, and adhere to best practices in rule writing. -- **Sandbox** (disabled by default) indicates that the rule is in an experimental stage. The potential for broader usefulness and relevance of "Sandbox" rules is currently being assessed. These rules can serve as an inspiration and adhere to the minimum acceptance criteria for rules. +- **Stable** (enabled by default) indicates that the rule has undergone thorough evaluation by experts with hands-on production experience. These experts have determined that the rules embody best practices and exhibit optimal robustness, making it more difficult for attackers to bypass Falco. These rules are highly relevant for addressing broader threats and are recommended for customization to specific environments. They primarily focus on universal system-level detections, such as generic reverse shells or container escapes, which establish a solid baseline for threat detection across diverse industries. This inherent bias against including more application-specific detections is due to their potential lack of broad relevance or applicability. However, to mitigate this bias, a grey area will be reserved, enabling case-by-case judgments to be made. +- **Incubating** (disabled by default) indicates that the rules address relevant threats, provide a certain level of robustness guarantee, and adhere to best practices in rule writing. Furthermore, it signifies that the rules have been identified by experts as catering to more specific use cases, which may or may not be relevant for each adopter. This category is expected to include a larger number of application-specific rules. +- **Sandbox** (disabled by default) indicates that the rule is in an experimental stage. The potential for broader usefulness and relevance of "Sandbox" rules is currently being assessed. These rules can serve as inspiration and adhere to the minimum acceptance criteria for rules. - **Deprecated** (disabled by default), indicates that, upon re-assessment, the rule was deemed less applicable to the broader community. Each adopter needs to determine the relevance of these rules on their own. They are kept as examples but are no longer actively supported or tuned by The Falco Project. -> Falco introduces two new keys, `scope` and `maturity` for each rule. These keys, considered in combination, reflect the robustness, relevance, applicability, and stability of each predefined rule in the [falcosecurity/rules](https://github.com/falcosecurity/rules/blob/main/rules/) repository. They serve as general guidance to determine which rules may provide the highest return on investment. As a minimum requirement, each rule must go through the "Incubating" state before advancing to "Stable." Only "Stable" rules will be enabled by default. On the other hand, the scope assignment adheres to predefined criteria, and the reviewer ensures compliance with these criteria during the rule acceptance process. +> Falco introduces a new key, `maturity`, for each rule. This key reflects the robustness, relevance, applicability, and stability of each predefined rule in the [falcosecurity/rules](https://github.com/falcosecurity/rules/blob/main/rules/) repository. It serves as general guidance to determine which rules may provide the highest return on investment. As a minimum requirement, each rule must go through the "Incubating" state before advancing to "Stable". Only "Stable" rules will be enabled by default. We introduce a new yaml key to provide more transparent configurability through the `falco.yaml` file to support the unique use cases of adopters. Therefore, the maturity level should be treated as a top-level feature, similar to the priority key, and not just another tag. The maturity level of the rules, however, does not directly reflect their potential for generating noise in the adopters' environment. This is due to the unique and constantly changing nature of each environment, especially in cloud environments, making it challenging to accurately predict the impact of rules. -Newcomers to Falco will be encouraged to start by configuring their setup with introductory rules labeled as "Falco's default rules" (`scope: "Core"` and `maturity: "Stable"`). These rules, which are currently based on syscall and container events, will live in the established [falco_rules.yaml](https://github.com/falcosecurity/rules/blob/main/rules/falco_rules.yaml) file. +Newcomers to Falco will be encouraged to start by configuring their setup with introductory rules labeled as "Falco's default rules" (`maturity: "Stable"`). These rules, which are currently based on syscall and container events, will live in the established [falco_rules.yaml](https://github.com/falcosecurity/rules/blob/main/rules/falco_rules.yaml) file. As users become more familiar with Falco and better understand their unique environments, they can gradually fine-tune the default rules to meet their specific requirements. Tuning rules goes hand in hand with assessing the performance overhead and adjusting Falco's [configuration](https://github.com/falcosecurity/falco/blob/master/falco.yaml) accordingly. This consideration is crucial to convey to adopters, as it is important to keep in mind that there are usually limitations to the budget allocated for security monitoring. -Once adopters have integrated the stable default rules with low false positives and acceptable performance consistently, they can add a next set of rules. This set may include rules with `scope: "Ecosystem"` and above, and/or `maturity: "Incubating"` and above, providing more specific detections. These efforts will be supported by guidance on rule customization and broader educational initiatives that focus on Linux OS runtime security. +Once adopters have integrated the stable default rules with low false positives and acceptable performance consistently, they can add a next set of rules. This set may include rules with `maturity: "Incubating"`, providing more specific detections. These efforts will be supported by guidance on rule customization and broader educational initiatives that focus on Linux OS runtime security. + +> Experts in the field, with expertise in areas such as offensive security, Linux kernel tracing, production deployment, cyber security, threat detection, compliance, data analysis, and data science, will be responsible for assessing the `maturity` levels of rules. These experts have a deep familiarity with Falco. + + +*Disclaimer*: The Falco maintainers understand that this approach may not be perfect from the beginning and that it requires field experience and community feedback to move towards a desired end solution. We have chosen the outlined rules maturity tagging approach as an initial solution to introduce more clarity and flexibility than we currently have while still aiming to keep things relatively simple in the beginning. -> Experts in the field, with expertise in areas such as offensive security, Linux kernel tracing, production deployment, cyber security, threat detection, compliance, data analysis, and data science, will be responsible for assessing the `scope` and `maturity` levels of rules. These experts have a deep familiarity with Falco. ### Define Rule Contribution Process @@ -95,7 +82,11 @@ The minimum criteria for rules creation shall include: *Correctness* -The rule must be both syntactically and grammatically correct and should evaluate to true during successful end-to-end tests. Furthermore, it needs to accurately detect the intended cyber threats, specifically the Tactics, Techniques, and Procedures (TTPs) aligned with the [Mitre ATT&CK](https://attack.mitre.org/) framework. To ensure this, a new [Mitre Checker Module](https://github.com/falcosecurity/rules/issues/88) will be utilized to audit the correct Mitre tags and references. +The criteria with regard to correctness shall include, at a minimum: + +- Correctness of the expression language, both syntactically and grammatically. +- Consistency with the name/description. +- If any tests are present, they must pass. During the initial review process and subsequent changes, manual testing should also be conducted to verify that the rule is capable of detecting the cyber threat(s) it aims to detect (read more under the testing section). *Robustness* @@ -116,7 +107,7 @@ Possible criteria: *Testing* -How was the rule tested? The desired testing approach includes not only functional end-to-end tests in virtual machines (VMs) but also deployments in real-world infrastructure. The Falco maintainers will provide guidance and support to contributors throughout the testing process, recognizing that it will vary significantly for each rule due to its unique nature. +How was the rule tested? The desired testing approach includes not only functional end-to-end tests in virtual machines (VMs), but also deployments in real-world infrastructure. The Falco maintainers will provide guidance and support to contributors throughout the testing process, recognizing that it will vary significantly for each rule due to its unique nature. Additionally, while we aim to automate tests and expand test coverage, we acknowledge that it is a large engineering effort that will take time. Therefore, manual tests during the review processes and best effort deployment tests will be necessary in the interim. ### Guidance on Rule Customization and Tuning @@ -147,6 +138,8 @@ In summary, the following action items are planned to be completed leading up to As a result of these changes, Falco's principles for rules adoption, maturity, and management will be updated. However, adopters can continue to use Falco as they have been doing before. +*Disclaimer*: If new information becomes available during the implementation of these changes, such as expanded feedback from the community, the Falco maintainers reserve the right to be flexible and adaptive, and re-evaluate choices and designs. + ## Appendix