feat: Add Julia language analyzer support

[Octogonapus]

Description

This PR adds Julia language support for generating SBOMs.

Related issues

• Close Add Julia language support for SBOM #5659

Checklist

• I've read the guidelines for contributing to this repository.
• I've followed the conventions in the PR title.
• I've added tests that prove my fix is effective or that my feature works.
• I've updated the documentation with the relevant information (if needed).
• I've added usage information (if the PR introduces new options)
• I've included a "before" and "after" example to the description (if the PR is a user interface change).

[DmitriyLewen]

Hello @Octogonapus
Thanks for your work!

I left few comments.

Can you also add tests for SPDX, CycloneDX to see how we use UUID for these formats?

[Octogonapus]

Can you also add tests for SPDX, CycloneDX to see how we use UUID for these formats?

Yes I would like to do this. Can you help me by pointing out where there is an example of how to do this? I looked around the code for a while but I can't figure out how to export an SBOM in code (except of course by invoking the trivy command).

[DmitriyLewen]

take a look on marshal_test.go and unmarshal_test.go files:
CycloneDX - https://github.com/aquasecurity/trivy/tree/main/pkg/sbom/cyclonedx
SPDX - https://github.com/aquasecurity/trivy/tree/main/pkg/sbom/spdx

Integration tests for SBOM - https://github.com/aquasecurity/trivy/blob/main/integration/sbom_test.go

[Octogonapus]

I pushed an SPDX test with Julia, though it currently does not pass because the SPDX marshaller is doing some sort of deduplication based on the package name. i.e. if I changed one of the B packages to C, then the test could pass. Do you know what is causing that?

[DmitriyLewen]

Do you mean that the packages are in random order?

[Octogonapus]

No, I mean that in the SPDX marshal test I added, there are two packages named B, one of which (the last one) gets entirely removed during marshaling. The final SBOM contains 4 package entries instead of 5, and 4 relationships instead of 5.

[DmitriyLewen]

Now i understand you.
This happens because we are overwriting hasher function for the SPDX test:

trivy/pkg/sbom/spdx/marshal_test.go

Lines 968 to 988 in b5874e3

	hasher := func(v interface{}, format hashstructure.Format, opts *hashstructure.HashOptions) (uint64, error) {
	h.Reset()
	var str string
	switch v.(type) {
	case ftypes.Package:
	str = v.(ftypes.Package).Name + v.(ftypes.Package).FilePath
	case string:
	str = v.(string)
	case *ftypes.OS:
	str = v.(*ftypes.OS).Name
	default:
	require.Failf(t, "unknown type", "%T", v)
	}
	if _, err := h.Write([]byte(str)); err != nil {
	return 0, err
	}
	return h.Sum64(), nil
	}

Some packages (like go binary packages) don't have ID, so use that instead of Name here:

trivy/pkg/sbom/spdx/marshal_test.go

Line 974 in b5874e3

str = v.(ftypes.Package).Name + v.(ftypes.Package).FilePath

Perphaps we can just add Indirect field.
wdyt?

[Octogonapus]

Thanks, good idea, that worked.

[Octogonapus]

I pushed a CycloneDX test as well. There are a few problems with it that will require changing its marshaller:

1. There is a problem with the CycloneDX dependencies format. Only package URLs are used, so in the case of a shadowed dep, you don't know which B package is being referenced since there are two pkg:julia/B@1.9.0. Are we required to use PURLs or can we use another format? SPDX uses an arbitrary hash here and that works well.
2. I noticed this comment: https://vscode.dev/github/Octogonapus/trivy/blob/analyzer_language_julia/pkg/sbom/cyclonedx/core/cyclonedx.go#L87. Indeed this is the case as you can see in my (failing) CycloneDX test. Can this functionality be changed, or will that not conform to the CycloneDX specification? If we can't change it, Julia users should simply not use CycloneDX.
3. (applicable to SPDX as well) I have a concern about the hasher used in production, given that we use a custom hasher in tests. Should we update the default hasher rather than providing a custom one in our tests? If not, how do we know a proper hasher implementation (i.e. one which does not discard shadowed packages) will be used in production?

I guess if we really needed to, we could also change the Julia PURL spec as package-url/purl-spec#237 has not been accepted yet. The PURLs could become pkg:UUID@version e.g. pkg:edca9bc6-334e-11e9-3554-9595dbb4349c@1.9.0.

[DmitriyLewen]

Only package URLs are used, so in the case of a shadowed dep, you don't know which B package is being referenced since there are two pkg:julia/B@1.9.0

This is a strange case. Why does Julia create duplicate dependencies?
Why can't we use existing B for A?

About PURL and BomRef - i think we can use next logic:

• create PURL as you offered
• use pkgID (ID from julia file) for BomRef

Can this functionality be changed

What if we update BomRef logic? if we use pkgID, we will not get a conflict, because BomRef will be unique for each julia library.

The PURLs could become pkg:UUID@version e.g. pkg:edca9bc6-334e-11e9-3554-9595dbb4349c@1.9.0.

I don't think it's convenient. You need to read the lock file to find out the dependency name. But we can use tag_id for UUID (as for https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst#swid)
wdyt?

Should we update the default hasher rather than providing a custom one in our tests

If I remember correctly, we added this because SPDX uses different hashes for each run(take a look this test -

trivy/integration/repo_test.go

Lines 360 to 368 in aedbd85

	{
	name: "conda generating SPDX SBOM",
	args: args{
	command: "rootfs",
	format: "spdx-json",
	input: "testdata/fixtures/repo/conda",
	},
	golden: "testdata/conda-spdx.json.golden",
	},

).

[Octogonapus]

This is a strange case. Why does Julia create duplicate dependencies?
Why can't we use existing B for A?

Julia allows you to add two dependencies with the same name but different UUIDs. This is an edge case, but given that it is allowed, I think it is important to test.

Your suggestions for the PURL sounds good to me. After reviewing the spec, I can simply add a qualifier for the uuid. I will work on adding this to the spec and to https://github.com/package-url/packageurl-go.

[Octogonapus]

I added two integration tests for Julia SBOMs (SPDX and CycloneDX). I can't manage to run the tests, though:

cd integration
go test
package github.com/aquasecurity/trivy/integration: build constraints exclude all Go files in /home/salmon/Documents/code/trivy/integration

or

mage test:integration
No .go files marked with the mage build tag in this directory.

So the tests might be wrong. If admin could trigger CI, that would be helpful so I can fix the tests.

[DmitriyLewen]

You need to run mage test:integration from the root directory of the trivy repository:

➜ ls -dhl */
drwxr-xr-x  11 work  staff   352B Jul 31 09:25 brand/
drwxr-xr-x   4 work  staff   128B Aug 30 08:51 ci/
drwxr-xr-x   3 work  staff    96B Jul 31 09:25 cmd/
drwxr-xr-x  10 work  staff   320B Oct 31 12:24 contrib/
drwxr-xr-x  11 work  staff   352B Dec  7 10:47 docs/
drwxr-xr-x   4 work  staff   128B Jul 31 09:25 examples/
drwxr-xr-x   3 work  staff    96B Jul 31 09:25 helm/
drwxr-xr-x  15 work  staff   480B Dec  7 11:40 integration/
drwxr-xr-x   3 work  staff    96B Nov 30 10:08 internal/
drwxr-xr-x   5 work  staff   160B Dec  7 12:07 magefiles/
drwxr-xr-x   5 work  staff   160B Oct 31 12:24 misc/
drwxr-xr-x  43 work  staff   1.3K Nov 30 10:08 pkg/
drwxr-xr-x   5 work  staff   160B Jul 31 09:25 rpc/

➜ mage test:integration
?       github.com/aquasecurity/trivy/pkg/fanal/test/integration/docker [no test files]
...

I will work on adding this to the spec and to https://github.com/package-url/packageurl-go.

Let us know about this work. It may make sense to wait for these changes before merging this PR.

[Octogonapus]

You need to run mage test:integration from the root directory of the trivy repository

That is what I did.

Let us know about this work. It may make sense to wait for these changes before merging this PR.

I've updated the spec PR. Luckily there are no changes required in packageurl-go.

[DmitriyLewen]

That is what I did.

hm.. this is strange.
How did you install mage?

Also you can copy command from mage file (e.g.

trivy/magefiles/magefile.go

Lines 239 to 243 in 01edbda

	// Integration runs integration tests
	func (t Test) Integration() error {
	mg.Deps(t.FixtureContainerImages)
	return sh.RunWithV(ENV, "go", "test", "-timeout", "15m", "-v", "-tags=integration", "./integration/...", "./pkg/fanal/test/integration/...")
	}

)
and use it in bash.

Anyway i ran tests in CI/CD.

[Octogonapus]

I reinstalled mage from source and it is working now 🤷

I fixed the integration tests. Everything is looking good to me, now that the PURL includes the UUID.

[DmitriyLewen]

We recently added Dev flag to Package (Dev dependencies are excluded from the report by default, but users can use --include-dev-deps to include them).

But I'm not sure that Julia's users need this feature.
What do you think? Do users need dev dependencies in reports?

[Octogonapus]

Yes I think this will be helpful! I have tried to implement this from looking at other code and I've updated the test with some dev deps. LMK if it looks correct.

[DmitriyLewen]

LGTM.
Left small comments. Take a look, please

[DmitriyLewen]

LGTM now!
@Octogonapus Thanks for your work!

@knqyf263 i approved this PR. Take a look, when you have time, please.

[DmitriyLewen]

@Octogonapus
Can you fix linter errors, please?

[Octogonapus]

Will be fixed by aquasecurity/go-dep-parser#292

[github-actions]

This PR is stale because it has been labeled with inactivity.

[Octogonapus]

I'm still trying to keep this PR up to date... though it can be hard with how many refactors happen on main

[knqyf263]

Thanks for your patient. LGTM. I left a small comment.

[knqyf263]

We introduced the intermediate representation for SBOM. We don't need tests for SPDX and CycloneDX unless Julia needs any specific handling for SBOM. Either of them is fine.

[Octogonapus]

No specific handling is needed. I have removed the CycloneDX test.