Add an initial BitBake analyzer implementation #8642

sschuberth · 2024-05-13T07:13:07Z

Please have a look at the individual commit messages for the details.

plugins/package-managers/bitbake/src/main/kotlin/BitBake.kt

codecov · 2024-05-13T07:32:14Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.92%. Comparing base (45bb867) to head (bf83c5b).

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #8642   +/-   ##
=========================================
  Coverage     67.92%   67.92%           
  Complexity     1005     1005           
=========================================
  Files           244      244           
  Lines          7772     7772           
  Branches        876      876           
=========================================
  Hits           5279     5279           
  Misses         2110     2110           
  Partials        383      383

Flag	Coverage Δ
funTest-docker	`66.04% <ø> (ø)`
test	`37.84% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Dockerfile

fviernau · 2024-05-15T07:03:35Z

plugins/package-managers/bitbake/README.md

+[do-convert]: https://github.com/doubleopen-project/do-convert
+[SBOM]: https://docs.yoctoproject.org/dev/dev-manual/sbom.html
+[SPDX]: https://spdx.dev/
+[SPDX document file analyzer]: https://oss-review-toolkit.org/ort/docs/tools/analyzer


commit: Why did you choose to use the analyzer for mapping the SPDX file?

(I don't know that analyzer inside out, but I recall that it was implemented with some specific use case in mind which is beyond the SDPX spec)

Why did you choose to use the analyzer for mapping the SPDX file?

That was the easiest way to make use of SpdxDocumentCache / SpdxResolvedDocument and related resolution logic right now.

I recall that it was implemented with some specific use case in mind which is beyond the SDPX spec

Yes, indeed we might run into issues here when fully implementing document resolution. Ideally, any such issues could be fixed in SpdxDocumentFile itself (e.g. by generalizing code or making some behavior configurable). But if that does not work out, we should consider making direct use of the "low-level" spdx-utils in BitBake instead.

fviernau · 2024-05-15T07:07:32Z

plugins/package-managers/bitbake/src/funTest/kotlin/BitBakeToolFunTest.kt

+            }
+
+            result.analyzer?.result shouldNotBeNull {
+                projects shouldHaveSize 90


I think it would be valuable to make the fun test consistent with the ones for other package managers,
to use an expected output file which contains the expected ORT model. This would make it more obvious what the package manager outputs and protect from unintended breaking changes when changing the SPDX analyzer.

plugins/package-managers/bitbake/build.gradle.kts

fviernau · 2024-05-15T07:25:16Z

plugins/package-managers/bitbake/src/main/kotlin/BitBake.kt

+            }
+        }
+
+        if (!scriptFile.delete()) logger.warn { "Unable to delete the temporary '$scriptFile' file." }


Consider putting the '$scriptFile' to the very end of the sentence.

fviernau · 2024-05-15T07:26:42Z

plugins/package-managers/bitbake/src/main/kotlin/BitBake.kt

+    }
+
+    override fun resolveDependencies(definitionFile: File, labels: Map<String, String>): List<ProjectAnalyzerResult> =
+        throw NotImplementedError("This function is not supported for $managerName.")


Please add a code comment explaining why.

fviernau · 2024-05-15T07:28:56Z

plugins/package-managers/bitbake/src/main/kotlin/BitBake.kt

+        throw NotImplementedError("This function is not supported for $managerName.")
+
+    private fun getDeployDir(workingDir: File, target: String): File {
+        val bitbakeEnv = runBitBake(workingDir, "-e", target)


Should this be bitBakeEnv ?

Alternative:

val lines = runBitBake(workingDir, "-e", target).stdout.lineSequence() return lines.mapNotNull { it.withoutPrefix("DEPLOY_DIR=") }.first().let { File(it.removeSurrounding("\"")) }

fviernau · 2024-05-15T07:29:37Z

plugins/package-managers/bitbake/src/main/kotlin/BitBake.kt

+    private fun createSpdx(workingDir: File, target: String) =
+        runBitBake(workingDir, "-r", spdxConfFile.absolutePath, "-c", "create_spdx", target)
+
+    private fun File.findSpdxFiles() = resolve("spdx").walk().filter { it.isFile && it.name.endsWith(".spdx.json") }


Could be moved to file level.

fviernau · 2024-05-15T07:34:46Z

website/docs/tools/analyzer.md

@@ -14,6 +14,7 @@ Currently, the following package managers (grouped by the programming language t

 * C / C++
  * [Bazel](https://bazel.build/) (**experimental**) (limitations: see [open tasks](https://github.com/oss-review-toolkit/ort/issues/264))
+  * [BitBake](https://docs.yoctoproject.org/bitbake/) (**experimental**) (limitations: slow performance, resolution of SPDX external document refs)


Do the SPDX files exported by BitBake use external document refs?

fviernau · 2024-05-15T08:04:00Z

plugins/package-managers/bitbake/src/funTest/kotlin/BitBakeToolFunTest.kt

+
+    "Analyzing recipes from Poky" should {
+        val projectDir = tempdir()
+        val pokyVcsInfo = VcsInfo(VcsType.GIT, "https://git.yoctoproject.org/poky", "kirkstone-4.0.17")


Not sure how long the clone takes, but it seems the repository is rather large.

Not just for speed, but also for minimizing the test, what do you think about crafting a more minimal project under the assets directory? This would enable adding further (corner) cases easily, on future increments of this implementation.

fviernau · 2024-05-15T08:08:50Z

plugins/package-managers/bitbake/src/main/kotlin/BitBake.kt

+
+        logger.info { "Found ${spdxFiles.size} SPDX file(s) in '$commonDeployDir'." }
+
+        return spdxManager.resolveDependencies(spdxFiles, labels)


Without mapping the result from the manager, I suspect that Identifiers (e.g. type), purl, cpe might not match what is desired. Maybe also the concluded licenses should explicitly be nullified.

See [1]. This prepares for BitBake support in the analyzer. [1]: https://docs.yoctoproject.org/bitbake/ Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.org>

Add the beginnings of an analyzer for BitBake [1], see [2] for context. It basically works by making the build inherit from the "create-spdx" class [3] to create SPDX 2.2 files and post-processing them via ORT's SPDX document file analyzer. This initial implementations has several known limitations still. First of all, as the "create-spdx" class cannot be used without building, and builds are not cached, so the analyzer is very slow. Secondly, SPDX external documents refs cannot be resolved yet. This requires some post-processing of the SPDX document files before passing them on, notably by adjusting the `SPDX_NAMESPACE_PREFIX` variable [4]. [1]: https://docs.yoctoproject.org/bitbake/ [2]: oss-review-toolkit#722 [3]: https://docs.yoctoproject.org/dev/dev-manual/sbom.html [4]: https://docs.yoctoproject.org/ref-manual/variables.html#term-SPDX_NAMESPACE_PREFIX Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.org>

Dockerfile

@@ -421,6 +421,17 @@
 COPY --from=bazelbuild /opt/bazel /opt/bazel
 COPY --from=bazelbuild /opt/go/bin/buildozer /opt/go/bin/buildozer

+#------------------------------------------------------------------------
+# BITBAKE
+FROM base as bitbakebuild


sschuberth requested review from a team as code owners May 13, 2024 07:13

github-advanced-security bot found potential problems May 13, 2024

View reviewed changes

plugins/package-managers/bitbake/src/main/kotlin/BitBake.kt Fixed Show fixed Hide fixed

sschuberth force-pushed the bitbake-start branch 2 times, most recently from 066b93e to 1bc9cdd Compare May 13, 2024 07:24

sschuberth force-pushed the bitbake-start branch 3 times, most recently from 8fc4c26 to bd94fe3 Compare May 13, 2024 09:24

sschuberth changed the title ~~Add an initial Bitbake analyzer implementation~~ Add an initial BitBake analyzer implementation May 13, 2024

fviernau requested changes May 15, 2024

View reviewed changes

sschuberth force-pushed the bitbake-start branch 2 times, most recently from a8372f8 to bf83c5b Compare May 16, 2024 10:24

sschuberth added 2 commits September 25, 2024 15:01

feat(docker): Add BitBake, the build tool for the Yocto project

fe667d2

See [1]. This prepares for BitBake support in the analyzer. [1]: https://docs.yoctoproject.org/bitbake/ Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.org>

sschuberth force-pushed the bitbake-start branch from bf83c5b to 7771f2e Compare September 25, 2024 13:03

github-advanced-security bot found potential problems Sep 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an initial BitBake analyzer implementation #8642

Add an initial BitBake analyzer implementation #8642

sschuberth commented May 13, 2024

codecov bot commented May 13, 2024 •

edited

Loading

fviernau May 15, 2024

sschuberth May 15, 2024

fviernau May 15, 2024

fviernau May 15, 2024

fviernau May 15, 2024

fviernau May 15, 2024

fviernau May 15, 2024

fviernau May 15, 2024

fviernau May 15, 2024

fviernau May 15, 2024


		logger.info { "Found ${spdxFiles.size} SPDX file(s) in '$commonDeployDir'." }

		return spdxManager.resolveDependencies(spdxFiles, labels)

Add an initial BitBake analyzer implementation #8642

Are you sure you want to change the base?

Add an initial BitBake analyzer implementation #8642

Conversation

sschuberth commented May 13, 2024

codecov bot commented May 13, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 13, 2024 •

edited

Loading