Fundamental Windows incompatibility when accessing the file system

[Joerki]

What happened:

Syft is at least not working properly on Windows analyzing a project in a directory (syft scan dir:...).
I used variations of SYFT_GOLANG... env variables to enable module remote access.
License information is not present in the Syft JSON SBOM output.

I saw the same symptoms like described in #1933.
I debugged syft . The problems go deeper and do not just affect licenses.go, so I create a dedicated bug for it.

In general, file (path) operations of syft do not support Windows properly.
Path operations for the local file system are done with functions coming from "path" module instead of "path/filepath".
The Go documention states clear that "path" is only for Unix/Linux, but not for Windows.
Instead, "path/filepath" has to be used, because this module support both Unix-like and Windows OS.

I also recognized that syft is hanging (at the end?) of processing on Windows, if -v is NOT set.

Another funny thing I saw is that you try to access paths that are concatenated taking the local path and typical Unix/Linux locations

[0000] TRACE searching filetree by path path=D:\Projekte\Go\Hello_world\etc\os-release
[0000] TRACE searching filetree by path path=D:\Projekte\Go\Hello_world\usr\lib\os-release
[0000] TRACE searching filetree by path path=D:\Projekte\Go\Hello_world\etc\system-release-cpe
[0000] TRACE searching filetree by path path=D:\Projekte\Go\Hello_world\etc\redhat-release
[0000] TRACE searching filetree by path path=D:\Projekte\Go\Hello_world\bin\busybox

I did not investigate what would need to be considered if the scanned object is not a directory tree.

What you expected to happen:

The readme file should have a clear note that there are issues on Windows until these are resolved.
Use "path/filepath" instead of "path" functions for file system accesses to make it compatible with Windows and Unixes.
Syft should terminate properly at the end, even if -v is not used.
Finally #1933 is resolved also.

Steps to reproduce the issue:

Execute syft on a Go project with a colorized "Hello world" (one external dependency) on WIndows.

Anything else we need to know?:

Environment:

• Output of syft version:
  Application: syft
  Version: 0.104.0
  BuildDate: 2024-02-07T20:51:36Z
  GitCommit: bd0cb91
  GitDescription: v0.104.0
  Platform: windows/amd64
  GoVersion: go1.21.6
  Compiler: gc
• OS (e.g: cat /etc/os-release or similar):
  Windows 11

[wagoodman]

Thanks for reporting, we'll look into this.

Use "path/filepath" instead of "path" functions for file system accesses to make it compatible with Windows and Unixes.

Looking at using path vs filepath packages might be a bit of a red herring since our file.Resolver layer normalizes windows paths to unix paths so that much of the business logic early on does not have to worry about these concerns (thus we can use path).

[Joerki]

Hi guys,

I noticed the issues at the places and command in first instance where you are NOT using you file resolver operations, but "path" functions instead of "filepath" directly! So this is absolutely not a red herring.
What I did to resolve go cataloger issues (tested on windows only):
config.go: changed "path" to "filepath" operations (goPath, g.LocalModCacheDir)
licenses.go: a glob definition must also match to the current OS. Also here Base and Join with "filepath" instead of "path"
package.go: also here you have hardcoded "/" and path.Join where file resolver is not used.

I saw that you have implemented the file resolver unindexed_directory.go, but you mix "path" and afero, and this is not platform independent and not working on Windows (so you don't normalize it properly). I did not work with afero yet, but since Windows FS is supported (I took this from their Readme), you must consider platform-independant path operations either solely with pure afero or a working mix with other platform-independent libraries like "filepath".

I hope you understood now why I wrote "fundamental" :)

BR

[willmurphyscode]

Hi @Joerki,

Thanks for the detailed issue report. The specific issue you reported, where Syft was incorrectly computing the path on the host to golang proxy cache, was fixed by #2654.

I'm leaving this issue open because I'd like to improve the CI testing of Syft on Windows in general, and am using this issue to track that work. If you encounter any related bugs, please let us know.

[Joerki]

Thank you very much @willmurphyscode for taking care on the issues.
I will check our build step soon and replace our temporary workaround, stay tuned for updates and keep an eye on the app :)