Skip to content

Commit 38387e4

Browse files
author
Trong Nhan Mai
authored
feat!: allow specifying the dependency depth resolution through CLI and make dependency resolution off by default (#840)
Signed-off-by: Trong Nhan Mai <trong.nhan.mai@oracle.com>
1 parent c4dbe46 commit 38387e4

File tree

76 files changed

+5502
-153
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

76 files changed

+5502
-153
lines changed

docs/source/pages/cli_usage/command_analyze.rst

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,8 @@ Usage
2222
usage: ./run_macaron.sh analyze
2323
[-h] [-sbom SBOM_PATH] [-purl PURL] [-rp REPO_PATH] [-b BRANCH]
2424
[-d DIGEST] [-pe PROVENANCE_EXPECTATION]
25-
[--skip-deps] [-g TEMPLATE_PATH]
25+
[--skip-deps] [--deps-depth DEPS_DEPTH] [-g TEMPLATE_PATH]
26+
[--python-venv PYTHON_VENV]
2627
2728
-------
2829
Options
@@ -64,12 +65,20 @@ Options
6465

6566
.. option:: --skip-deps
6667

67-
Skip automatic dependency analysis.
68+
DEPRECATED. Dependency resolution is off by default. This flag does nothing and will be removed in the next release.
69+
70+
.. option:: --deps-depth DEPS_DEPTH
71+
72+
The depth of the dependency resolution. 0: disable, 1: direct dependencies, inf: all transitive dependencies. (Default: 0)
6873

6974
.. option:: -g TEMPLATE_PATH, --template-path TEMPLATE_PATH
7075

7176
The path to the Jinja2 html template (please make sure to use .html or .j2 extensions).
7277

78+
.. option:: --python-venv PYTHON_VENV
79+
80+
The path to the Python virtual environment of the target software component.
81+
7382
-----------
7483
Environment
7584
-----------

docs/source/pages/supported_technologies/index.rst

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,18 @@ Provenances
107107
* The provenance should be published on JFrog Artifactory
108108
- :doc:`page </pages/supported_technologies/jfrog>`
109109

110+
.. _supported_automatic_deps_resolution:
111+
112+
-------------------------------
113+
Automatic dependency resolution
114+
-------------------------------
115+
116+
Currently, we support the following type of project for automatic dependency resolution.
117+
118+
* Java Maven
119+
* Java Gradle
120+
* Python (with a Python virtual environment created and packages installed using Python3.11, see :ref:`providing Python virtual environment <python-venv-deps>`.)
121+
110122
--------
111123
See also
112124
--------

docs/source/pages/tutorials/commit_finder.rst

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ To perform an analysis on Arrow, Macaron can be run with the following command:
5050

5151
.. code-block:: shell
5252
53-
./run_macaron.sh analyze -rp https://github.com/arrow-py/arrow --skip-deps
53+
./run_macaron.sh analyze -rp https://github.com/arrow-py/arrow
5454
5555
However, this will return results based only on the current state of the repository, which as described above, is not what we want to achieve in this tutorial. To perform analyses on other repository states, we need to provide Macaron with the target artifact versions in the form of `PURLs <https://github.com/package-url/purl-spec>`_, or Package URLs, which is a convenient way to encode packages from different ecosystems into the same format.
5656

@@ -67,7 +67,7 @@ We will start by running the analysis on the latest version, ``1.3.0``, with the
6767

6868
.. code-block:: shell
6969
70-
./run_macaron.sh analyze -purl pkg:pypi/arrow@1.3.0 --skip-deps
70+
./run_macaron.sh analyze -purl pkg:pypi/arrow@1.3.0
7171
7272
The analysis involves Macaron downloading the contents of the target repository to the configured, or default, ``output`` folder. Results from the analysis, including checks, are stored in the database found at ``output/macaron.db`` (See :ref:`Output Files Guide <output_files_guide>`). Once the analysis is complete, Macaron will also produce a report in the form of a HTML file.
7373

@@ -101,7 +101,7 @@ Now we should run the next analysis, and then open the new report.
101101

102102
.. code-block:: shell
103103
104-
./run_macaron.sh analyze -purl pkg:pypi/arrow@0.15.0 --skip-deps
104+
./run_macaron.sh analyze -purl pkg:pypi/arrow@0.15.0
105105
open output/reports/pypi/arrow/arrow.html
106106
107107
.. _fig_arrow_0.15.0_top:

docs/source/pages/tutorials/detect_malicious_package.rst

Lines changed: 5 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ First, we need to run the ``analyze`` command of Macaron to run a number of :ref
6565

6666
.. code-block:: shell
6767
68-
./run_macaron.sh analyze -purl pkg:pypi/django@5.0.6 --skip-deps
68+
./run_macaron.sh analyze -purl pkg:pypi/django@5.0.6
6969
7070
.. note:: By default, Macaron clones the repositories and creates output files under the ``output`` directory. To understand the structure of this directory please see :ref:`Output Files Guide <output_files_guide>`.
7171

@@ -172,27 +172,17 @@ Let's assume ``/tmp/.django_venv`` is the virtual environment where ``django@5.0
172172

173173
.. note:: If you want Macaron to analyze the virtual environment directly to identify the dependencies, we require Python 3.11 to be used to install the package. Alternatively, you can generate the SBOM as instructed :ref:`here <python-sbom>` and pass it to Macaron as input.
174174

175-
Run Macaron as follows to analyze ``django`` and its dependencies.
175+
Run Macaron as follows to analyze ``django`` and its direct dependencies.
176176

177177
.. code-block:: shell
178178
179-
./run_macaron.sh analyze -purl pkg:pypi/django@5.0.6 --python-venv "/tmp/.django_venv"
179+
./run_macaron.sh analyze -purl pkg:pypi/django@5.0.6 --python-venv "/tmp/.django_venv" --deps-depth=1
180180
181-
182-
By default Macaron only checks the direct dependencies. To turn on recursive dependency analysis, add the following to the ``configurations.ini`` file:
183-
184-
.. code-block:: ini
185-
186-
[dependency.resolver]
187-
recursive = True
188-
189-
And pass that to the ``analyze`` command:
181+
Or alternatively, run Macaron as follows to analyze ``django`` and all its transitive dependencies.
190182

191183
.. code-block:: shell
192184
193-
./run_macaron.sh --defaults-path configurations.ini analyze -purl pkg:pypi/django@5.0.6 --python-venv "/tmp/.django_venv"
194-
195-
To learn more about changing configurations see :ref:`here <change-config>`.
185+
./run_macaron.sh analyze -purl pkg:pypi/django@5.0.6 --python-venv "/tmp/.django_venv" --deps-depth=inf
196186
197187
Now we can enforce the policy below to ensure that the ``mcn_detect_malicious_metadata_1`` check always passes on ``django`` and its dependencies, indicating that none of the dependencies have malicious behavior.
198188

docs/source/pages/tutorials/exclude_include_checks.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ Normally, this is how you would run Macaron:
3737

3838
.. code-block:: shell
3939
40-
./run_macaron.sh analyze --package-url pkg:maven/io.micronaut/micronaut-core@4.3.10 --skip-deps
40+
./run_macaron.sh analyze --package-url pkg:maven/io.micronaut/micronaut-core@4.3.10
4141
4242
However, there can be checks in Macaron that are not relevant for the ``io.micronaut/micronaut-core`` artifact.
4343
For example, the ``mcn_provenance_witness_level_one_1`` check (defined in :class:`ProvenanceWitnessL1Check <macaron.slsa_analyzer.checks.provenance_witness_l1_check.ProvenanceWitnessL1Check>`) is not relevant because ``micronaut-projects/micronaut-core`` generates and publishes :term:`SLSA` provenances and no :term:`Witness` provenances. Therefore, we could exclude this check from running by performing the following steps.
@@ -61,7 +61,7 @@ With these two configuration options, all checks except for the excluded ``mcn_p
6161

6262
.. code-block:: shell
6363
64-
./run_macaron.sh --defaults-path ./defaults.ini analyze --package-url pkg:maven/io.micronaut/micronaut-core@4.3.10 --skip-deps
64+
./run_macaron.sh --defaults-path ./defaults.ini analyze --package-url pkg:maven/io.micronaut/micronaut-core@4.3.10
6565
6666
This time, the check ``mcn_provenance_witness_level_one_1`` doesn't run. After the ``analyze`` command finishes, we can view the data that Macaron has gathered about the ``micronaut-projects/micronaut-core`` repository at ``v4.3.10`` in an HTML report. Note that the result of the excluded check is not recorded in the Macaron HTML reports, JSON reports, or the database).
6767

docs/source/pages/tutorials/generate_verification_summary_attestation.rst

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,8 +47,7 @@ In order to verify the artifact with Macaron, you can follow the following steps
4747
./run_macaron.sh analyze \
4848
--package-url pkg:maven/io.micronaut.openapi/micronaut-openapi@6.8.0?type=jar \
4949
--provenance-file multiple.intoto.jsonl \
50-
--provenance-expectation expectation.cue \
51-
--skip-deps
50+
--provenance-expectation expectation.cue
5251
5352
.. note::
5453

docs/source/pages/tutorials/npm_provenance.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ To perform an analysis on the latest version of semver (when this tutorial was w
4242

4343
.. code-block:: shell
4444
45-
./run_macaron.sh analyze -purl pkg:npm/semver@7.6.2 --skip-deps
45+
./run_macaron.sh analyze -purl pkg:npm/semver@7.6.2
4646
4747
The analysis involves Macaron downloading the contents of the target repository to the configured, or default, ``output`` folder. Results from the analysis, including checks, are stored in the database found at ``output/macaron.db`` (See :ref:`Output Files Guide <output_files_guide>`). Once the analysis is complete, Macaron will also produce a report in the form of a HTML file.
4848

docs/source/pages/using.rst

Lines changed: 41 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ Using Macaron
1717
Analyzing an artifact with a PURL string
1818
----------------------------------------
1919

20-
Macaron can analyze an artifact (and its dependencies) to determine its supply chain security posture. To analyze an artifact, you need to provide the PURL identifier of the artifact:
20+
Macaron can analyze an artifact to determine its supply chain security posture. To analyze an artifact, you need to provide the PURL identifier of the artifact:
2121

2222
.. code-block::
2323
@@ -51,6 +51,7 @@ To run Macaron on an artifact, we use the following command:
5151
5252
./run_macaron.sh analyze -purl <artifact-purl>
5353
54+
Macaron can also analyze the package's dependencies. Please see :ref:`automate-deps-resolution`.
5455

5556
''''''''''''''''''''''''''''''''''''''
5657
Automated repository and commit finder
@@ -79,6 +80,7 @@ Within the configuration file under the ``repofinder.java`` header, three option
7980
- ``repo_pom_paths`` (Values: List of POM tags) - Determines where to search for repository information in the POM files. E.g. scm.url.
8081
- ``find_parents`` (Values: True or False) - When enabled, the Repository Finding feature will also search for repository URLs in parents POM files of the current dependency.
8182

83+
.. note:: Dependency related configurations like ``artifact_repositories`` or ``find_parents`` can affect :ref:`Macaron automatic dependency resolution <automate-deps-resolution>`.
8284

8385
.. note:: Finding repositories requires at least one remote call, adding some additional overhead to an analysis run.
8486

@@ -113,7 +115,7 @@ Analyzing a source code repository
113115
Analyzing a public GitHub repository
114116
''''''''''''''''''''''''''''''''''''
115117

116-
Macaron can also analyze a public GitHub repository (and potentially the repositories of its dependencies).
118+
Macaron can also analyze a public GitHub repository.
117119

118120
To run Macaron on a GitHub public repository, we use the following command:
119121

@@ -135,14 +137,6 @@ For example, to analyze the SLSA posture of `micronaut-core <https://github.com/
135137
136138
./run_macaron.sh analyze -rp https://github.com/micronaut-projects/micronaut-core -b 4.0.x -d 82d115b4901d10226552ac67b0a10978cd5bc603
137139
138-
.. note:: Macaron automatically detects and analyzes **direct** dependencies for Java Maven and Gradle projects. This process might take a while and can be skipped by using the ``--skip-deps`` option.
139-
140-
Take the same example as above, to disable analyzing `micronaut-core <https://github.com/micronaut-projects/micronaut-core>`_ direct dependencies, we could use the following command:
141-
142-
.. code-block:: shell
143-
144-
./run_macaron.sh analyze -rp https://github.com/micronaut-projects/micronaut-core -b 4.0.x -d 82d115b4901d10226552ac67b0a10978cd5bc603 --skip-deps
145-
146140
.. note:: By default, Macaron would generate report files into the ``output`` directory in the current working directory. To understand the structure of this directory please see :ref:`Output Files Guide <output_files_guide>`.
147141

148142
With the example above, the generated output reports can be seen here:
@@ -242,7 +236,7 @@ workflows.
242236

243237
.. code-block:: shell
244238
245-
./run_macaron.sh analyze -pe micronaut-core.cue -rp https://github.com/micronaut-projects/micronaut-core -b 4.0.x -d 82d115b4901d10226552ac67b0a10978cd5bc603 --skip-deps
239+
./run_macaron.sh analyze -pe micronaut-core.cue -rp https://github.com/micronaut-projects/micronaut-core -b 4.0.x -d 82d115b4901d10226552ac67b0a10978cd5bc603
246240
247241
where ``micronaut-core.cue`` file can contain:
248242

@@ -263,6 +257,32 @@ where ``micronaut-core.cue`` file can contain:
263257
.. note::
264258
The provenance expectation is verified via the ``provenance_expectation`` check in Macaron. You can see the result of this check in the HTML or JSON report and see if the provenance found by Macaron meets the expectation CUE file.
265259

260+
.. _automate-deps-resolution:
261+
262+
------------------------------------
263+
Analyzing dependencies automatically
264+
------------------------------------
265+
266+
Macaron supports automatically detecting and analyzing dependencies for certain types of projects (:ref:`supported_automatic_deps_resolution`). This feature is disabled by default and can be enabled with the CLI flag ``--deps-depth``.
267+
268+
The ``--deps-depth`` flag currently accepts these values:
269+
270+
* ``0``: Disable dependency resolution (Default).
271+
* ``1``: Resolve and analyze direct dependencies.
272+
* ``inf``: Resolve and analyze all transitive dependencies.
273+
274+
For example, to analyze `micronaut-core <https://github.com/micronaut-projects/micronaut-core>`_ and its **direct** dependencies, we could use the following command:
275+
276+
.. code-block:: shell
277+
278+
./run_macaron.sh analyze \
279+
-rp https://github.com/micronaut-projects/micronaut-core \
280+
-b 4.0.x \
281+
-d 82d115b4901d10226552ac67b0a10978cd5bc603 \
282+
--deps-depth=1
283+
284+
.. note:: This process might take a while. Alternatively, you can help Macaron by providing the dependencies information through : :ref:`an sbom <with-sbom>` or :ref:`a Python virtual environment <python-venv-deps>` (for Python packages only).
285+
266286
.. _with-sbom:
267287

268288
----------------------
@@ -283,10 +303,12 @@ To run the analysis against that SBOM, run this command:
283303

284304
.. code-block:: shell
285305
286-
./run_macaron.sh analyze -purl pkg:maven/org.apache.maven/maven@3.9.7?type=pom -sbom <path_to_sbom>
306+
./run_macaron.sh analyze -purl pkg:maven/org.apache.maven/maven@3.9.7?type=pom -sbom <path_to_sbom> --deps-depth=inf
287307
288308
Where ``path_to_sbom`` is the path to the SBOM you want to use.
289309

310+
.. note:: Make sure to enable dependency resolution with ``--deps-depth``.
311+
290312
.. _python-sbom:
291313

292314
''''''''''''''''''''''''
@@ -305,7 +327,7 @@ Then run Macaron and pass the SBOM file as input:
305327

306328
.. code-block:: shell
307329
308-
./run_macaron.sh analyze -purl pkg:pypi/django@5.0.6 -sbom <path_to_django_sbom.json>
330+
./run_macaron.sh analyze -purl pkg:pypi/django@5.0.6 -sbom <path_to_django_sbom.json> --deps-depth=inf
309331
310332
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
311333
Analyzing dependencies in the SBOM without the main software component
@@ -320,7 +342,7 @@ Then the analysis can be run as follows:
320342

321343
.. code-block:: shell
322344
323-
./run_macaron.sh analyze -purl pkg:maven/private.apache.maven/maven@4.0.0-alpha-1-SNAPSHOT?type=pom -sbom <path_to_sbom>
345+
./run_macaron.sh analyze -purl pkg:maven/private.apache.maven/maven@4.0.0-alpha-1-SNAPSHOT?type=pom -sbom <path_to_sbom> --deps-depth=inf
324346
325347
Where ``path_to_sbom`` is the path to the SBOM you want to use.
326348

@@ -344,10 +366,12 @@ Then run Macaron as follows:
344366

345367
.. code-block:: shell
346368
347-
./run_macaron.sh analyze -purl pkg:pypi/django@5.0.6 --python-venv "/tmp/.django_venv"
369+
./run_macaron.sh analyze -purl pkg:pypi/django@5.0.6 --python-venv "/tmp/.django_venv" --deps-depth=1
348370
349371
Where ``--python-venv`` is the path to virtual environment.
350372

373+
.. note:: Make sure to enable dependency resolution with ``--deps-depth``.
374+
351375
Alternatively, you can create an SBOM for the python package and provide it to Macaron as input as explained :ref:`here <with-sbom>`.
352376

353377
.. note:: We only support Python 3.11 for this feature of Macaron. Please make sure to install the package using this version of Python.
@@ -396,7 +420,7 @@ We can run Macaron against the local repository at ``target`` by using this comm
396420
397421
./run_macaron.sh --local-repos-path ./boo/foo --defaults-path ./defaults.ini analyze --repo-path target <rest_of_args>
398422
399-
With ``rest_of_args`` being the arguments to the ``analyze`` command (e.g. ``--branch/-b``, ``--digest/-d`` or ``--skip-deps`` similar to two previous examples).
423+
With ``rest_of_args`` being the arguments to the ``analyze`` command (e.g. ``--branch/-b``, ``--digest/-d`` similar to two previous examples).
400424

401425
The ``--local-repos-path/-lr`` flag tells Macaron to look into ``./boo/foo`` for local repositories. For more information, please see :ref:`Command Line Usage <cli-usage>`.
402426

@@ -422,7 +446,7 @@ We can run Macaron against the local repository at ``target`` by using this comm
422446
423447
./run_macaron.sh --local-repos-path ./boo/foo analyze --repo-path target <rest_of_args>
424448
425-
With ``rest_of_args`` being the arguments to the ``analyze`` command (e.g. ``--branch/-b``, ``--digest/-d`` or ``--skip-deps`` similar to two previous examples).
449+
With ``rest_of_args`` being the arguments to the ``analyze`` command (e.g. ``--branch/-b``, ``--digest/-d`` similar to two previous examples).
426450

427451
The ``--local-repos-path/-lr`` flag tells Macaron to look into ``./boo/foo`` for local repositories. For more information, please see :ref:`Command Line Usage <cli-usage>`.
428452

src/macaron/__main__.py

Lines changed: 44 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,23 @@
3131

3232
def analyze_slsa_levels_single(analyzer_single_args: argparse.Namespace) -> None:
3333
"""Run the SLSA checks against a single target repository."""
34+
deps_depth = None
35+
if analyzer_single_args.deps_depth == "inf":
36+
deps_depth = -1
37+
else:
38+
try:
39+
deps_depth = int(analyzer_single_args.deps_depth)
40+
except ValueError:
41+
logger.error("Please provide '1', '0' or 'inf' to `--deps-depth`")
42+
sys.exit(os.EX_USAGE)
43+
44+
if deps_depth not in [-1, 0, 1]:
45+
logger.error("Please provide '1', '0' or 'inf' to `--deps-depth`")
46+
sys.exit(os.EX_USAGE)
47+
3448
if not (analyzer_single_args.repo_path or analyzer_single_args.package_url):
49+
# We don't mention --config-path as a possible option in this log message as it going to be move soon.
50+
# See: https://github.com/oracle/macaron/issues/417
3551
logger.error(
3652
"""Analysis target missing. Please provide a package url (PURL) and/or repo path.
3753
Examples of a PURL can be seen at https://github.com/package-url/purl-spec:
@@ -128,10 +144,16 @@ def analyze_slsa_levels_single(analyzer_single_args: argparse.Namespace) -> None
128144
logger.error("Error while loading the input provenance file: %s", error)
129145
sys.exit(os.EX_DATAERR)
130146

147+
if analyzer_single_args.skip_deps:
148+
logger.warning(
149+
"The --skip-deps flag has been deprecated and WILL NOT do anything. "
150+
+ "Dependency resolution is off by default. This flag does nothing and will be removed in the next release."
151+
)
152+
131153
status_code = analyzer.run(
132154
run_config,
133155
analyzer_single_args.sbom_path,
134-
analyzer_single_args.skip_deps,
156+
deps_depth,
135157
provenance_payload=prov_payload,
136158
)
137159
sys.exit(status_code)
@@ -312,7 +334,10 @@ def main(argv: list[str] | None = None) -> None:
312334
required=False,
313335
type=str,
314336
default="",
315-
help=("The path to the SBOM of the analysis target."),
337+
help=(
338+
"The path to the SBOM of the analysis target. If this is set, "
339+
+ "dependency resolution must be enabled with '--deps-depth'."
340+
),
316341
)
317342

318343
single_analyze_parser.add_argument(
@@ -375,7 +400,19 @@ def main(argv: list[str] | None = None) -> None:
375400
required=False,
376401
action="store_true",
377402
default=False,
378-
help=("Skip automatic dependency analysis."),
403+
help=(
404+
"DEPRECATED. Dependency resolution is off by default. This flag does nothing and will be removed in the next release."
405+
),
406+
)
407+
408+
single_analyze_parser.add_argument(
409+
"--deps-depth",
410+
required=False,
411+
default="0",
412+
help=(
413+
"The depth of the dependency resolution. 0: disable, 1: direct dependencies, "
414+
+ "inf: all transitive dependencies. (Default: 0)"
415+
),
379416
)
380417

381418
single_analyze_parser.add_argument(
@@ -390,7 +427,10 @@ def main(argv: list[str] | None = None) -> None:
390427
single_analyze_parser.add_argument(
391428
"--python-venv",
392429
required=False,
393-
help=("The path to the Python virtual environment of the target software component."),
430+
help=(
431+
"The path to the Python virtual environment of the target software component. "
432+
+ "If this is set, dependency resolution must be enabled with '--deps-depth'."
433+
),
394434
)
395435

396436
# Dump the default values.

0 commit comments

Comments
 (0)