Skip to content

Conversation

AmineRaouane
Copy link
Member

@AmineRaouane AmineRaouane commented May 19, 2025

Summary

This PR adds a new heuristic that analyzes code to detect suspicious use of excessive spaces and invisible characters. It checks whether the amount of spacing and invisible Unicode characters exceeds a defined threshold.

Description of changes

  • Implemented the WhiteSpaces heuristic in a new Python module.
  • Registered the new heuristic inside the main heuristics.py file.
  • Created unit tests to verify the behavior of the WhiteSpacesAnalyzer heuristic.
  • Updated detect_malicious_metadata_check.py to integrate and execute the new heuristic logic during analysis.
  • The heuristic scans the codebase for abnormal invisible characters and spaces in the code.
  • This heuristic is combined with ForceSetup to justify high confidence in detection, as the presence of extra spaces alone could be due to poor formatting rather than malicious intent.

Related issues

None

Checklist

  • I have reviewed the contribution guide.
  • My PR title and commits follow the Conventional Commits convention.
  • My commits include the "Signed-off-by" line.
  • I have signed my commits following the instructions provided by GitHub. Note that we run GitHub's commit verification tool to check the commit signatures. A green verified label should appear next to all of your commits on GitHub.
  • I have updated the relevant documentation, if applicable.
  • I have tested my changes and verified they work as expected.

@oracle-contributor-agreement oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label May 19, 2025
@behnazh-w behnazh-w changed the title feat(heuristics): add Whitespace Check to detect excessive spacing and invisible characters. feat(malware-check): add whitespace Check to detect excessive spacing and invisible characters. May 20, 2025
@behnazh-w behnazh-w changed the title feat(malware-check): add whitespace Check to detect excessive spacing and invisible characters. feat(malware-check): add whitespace check to detect excessive spacing and invisible characters. May 20, 2025
@behnazh-w behnazh-w changed the title feat(malware-check): add whitespace check to detect excessive spacing and invisible characters. feat(malware-check): add whitespace check to detect excessive spacing and invisible characters May 20, 2025
@behnazh-w behnazh-w requested a review from art1f1c3R May 26, 2025 05:15
…d invisible characters

Signed-off-by: Amine <amine.raouane@enim.ac.ma>
Signed-off-by: Amine <amine.raouane@enim.ac.ma>
@AmineRaouane AmineRaouane force-pushed the white-spaces-heuristic branch from db0e35c to 6978bd7 Compare May 26, 2025 10:58
benmss and others added 12 commits June 17, 2025 15:45
Signed-off-by: Ben Selwyn-Smith <benselwynsmith@googlemail.com>
Signed-off-by: Amine <amine.raouane@enim.ac.ma>
…ackages (oracle#965)

Include support for using Semgrep for analysis of source code to detect malicious code patterns, specified using Semgrep's YAML files.

Signed-off-by: Carl Flottmann <carl.flottmann@oracle.com>
This PR allows Macaron to discover GitHub attestation. To retrieve these attestations, the SHA256 hash of the related artefact is required. Hashes are computed from local artefact files if available, or from downloaded ones otherwise.

Signed-off-by: Ben Selwyn-Smith <benselwynsmith@googlemail.com>
…acle#1096)

This PR replaces the Go shared library previously used via C-bindings in Python with a standalone binary for the cuevalidator component. The binary can now be invoked as a subprocess, simplifying integration and improving portability.

Signed-off-by: behnazh-w <behnaz.hassanshahi@oracle.com>
…e. (oracle#1102)

The detail info containing inspector links now contains links as keys regardless of whether they are reachable, and includes a boolean value for reachability.

Signed-off-by: Carl Flottmann <carl.flottmann@oracle.com>
…torial (oracle#1101)

Signed-off-by: Carl Flottmann <carl.flottmann@oracle.com>
…d invisible characters

Signed-off-by: Amine <amine.raouane@enim.ac.ma>
Signed-off-by: Amine <amine.raouane@enim.ac.ma>
Signed-off-by: Amine <amine.raouane@enim.ac.ma>
Signed-off-by: Amine <amine.raouane@enim.ac.ma>
art1f1c3R
art1f1c3R previously approved these changes Aug 8, 2025
@behnazh-w behnazh-w changed the title feat(malware-check): add whitespace check to detect excessive spacing and invisible characters feat(heuristics): add whitespace check to detect excessive spacing and invisible characters for malware check Aug 8, 2025
@art1f1c3R
Copy link
Member

art1f1c3R commented Aug 8, 2025

The CI test seems to be failing due to detecting excessive whitespace in django@5.0.6 for django/utils/html.py:149 and tests/gis_tests/distapp/tests.py:384.

@art1f1c3R art1f1c3R dismissed their stale review September 3, 2025 23:07

Dismissing approval until integration test failure is resolved.

@art1f1c3R
Copy link
Member

art1f1c3R commented Sep 5, 2025

I have investigated this problem. in django/utils/html.py:149-150 the following text appears in a docstring:

      format_html_join('\n', "<li>{} {}</li>", ((u.first_name, u.last_name)
                                                  for u in users))

In tests/gis_tests/distapp/tests.py:384-386 the following text also appears in a docstring:

================================

                                                | Projected Geometry | Lon/lat Geometry

Both of these examples are triggered by the excessive whitespace Semgrep rule as there are over 50 spaces before some of the indented lines. Both of these examples occur in docstrings, so my proposed solution (which I have tested does not trigger on django@5.0.6) is to alter the rule to this:

- id: obfuscation_excessive-spacing
  metadata:
    description: Detects the use of excessive spacing in code, which may indicate obfuscation or hidden code.
  message: Hidden code after excessive spacing
  languages:
  - python
  severity: WARNING
  pattern-either:
  - pattern-regex: '[\s]{50,}(\S)+' # The 50 here is the threshold for excessive spacing , more than that is considered obfuscation
  - pattern-not-inside: |
        """ ... """

I have used \s for whitespace characters and \S for non-whitespace characters for clarity here. The pattern-not-inside makes the rule avoid docstrings. Please try and see if this resolves the CI test failing issue.

Something we may have to be wary of is benign code blocks that are excessively indented and will cause this to trigger. Many projects will not encounter this, as the indentation level will not reach more than 50 spaces and/or code linters will prevent this from happening, so I don't expect too many false positives with that, but it is a possibility.

behnazh-w and others added 3 commits September 11, 2025 15:01
Resolve CI errors by not considering large whitespace text present in function docstrings.

Signed-off-by: Carl Flottmann <carlflottmann@gmail.com>
The pattern-not-inside was found to not behave as expected. This was resolved, along with formatting.

Signed-off-by: Carl Flottmann <carl.flottmann@oracle.com>
Copy link
Member

@art1f1c3R art1f1c3R left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI fix has been resolved by ignoring excessive spacing present in docstrings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OCA Verified All contributors have signed the Oracle Contributor Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants