Determinism in image dependencies

[bdovaz]

Right now, there are many linters that do not have the version of the dependency to install and instead install the latest version available at that time.

This goes against the determinism, security, traceability, etc... Because if you generate the image right now or in 5 minutes you can get completely different versions of dependencies.

In this series of PRs I am trying to partly solve this problem:

#4528 #4529 #4530 #4531 #4532 #4533 #4534 #4535 #4536 #4537 #4538 #4539 #4540 #4541 #4542

I have focused on:

• sh scripts that used the master/main branch.
• Docker images using “latest”.
• Dotnet dependencies with no version set.

Still a lot of npm, pip, etc.... This can be done in a next phase after merge all those PRs.

cc @nvuillam @echoix

[nvuillam]

This will bring us closer and closer to the "My Own MegaLinter Flavor" project :)

It will even allow "My Own MegaLinter Flavor with the linter versions I want" thing 🥳

[bdovaz]

@nvuillam the remaining PRs related to this issue are ready for review and merge.

[bdovaz]

Once this is done, we should go for the apk/npm/pip dependencies.

[bdovaz]

I'm with the remaining packages (a lot of them): apk, cargo, gem, pip and npm.

@nvuillam it occurs to me that when we have all this we can on the page of each linter show with shields.io if the packages that each linter depends on are actively maintained, etc .... Just as we do with the linter itself.

I have encountered @echoix the following problem:

megalinter/megalinter/descriptors/swift.megalinter-descriptor.yml

Line 26 in 8091505

cli_docker_image: ghcr.io/realm/swiftlint

https://github.com/realm/SwiftLint/pkgs/container/swiftlint

Can you think how to make renovate update the tag? It would be necessary to create a custom manager or similar I guess, no? Right now by not putting the tag, it implicitly pulls latest...

[echoix]

There's a cli_docker_image_version field available

megalinter/megalinter/Linter.py

Line 100 in c047fd7

self.cli_docker_image_version = "latest"

So now we only need to get that field updated. I searched on our GitHub repo for swiftlint, but I didn't see the image appear somewhere in the generated files.
If a variable could be substituted, we could have an "ENV " var in the Dockerfile, that could be kept updated from the descriptor and Dockerfile. Otherwise, it would be to create a regex/adjust a regex in the renovate config file to match what we need.

[bdovaz]

PR created but I'm still working on it: #4588

I had to change in build.py how the “core” dependencies are added so that they are also versioned.

[nvuillam]

@bdovaz i have a question and i'm afraid of the answer ^^

When we'll upgrade base image alpine version, we'll have all apk pinned to versions that do not exist in , so in my worst case scenario imagination:

• renovate PR to upgrade alpine version will crash because core apk pinned versions won't be available in the alpine packages versions
• it will generate dozens of PRs to upgrade ccore apk versions

Please can you reassure me ? ^^

I wonder it we shouldn't stick to linter versions in the first step, then see for the rest later, to make sure we don't have too much trouble

cc @echoix

[bdovaz]

@nvuillam if you look at the releases history, they release 2 versions at most per year, I do not think that it's a problem or if for whatever reason we have to do some manual change in that PR we can.

https://alpinelinux.org/releases/

It would make me very unhappy after all the hours you've put in on this PR to have to undo something honestly.... I'd rather once or twice a year have to work on those Alpine version change PRs.

Actually whenever we upgrade Alpine or Python it is never easy and it always takes some time to make changes anyway.

[bdovaz]

Also note that in renovate you can group packages by any criteria you want and one of them can be that the ecosystem is “repology”, that is, that all Alpine packages go in a single PR.

[nvuillam]

If you can group alpine version and alpine packages in the same renovate-generated PR, ok for me :)

[bdovaz]

@nvuillam @echoix to close this issue I think it would only remain (if we want) to review what is in this folder: https://github.com/oxsecurity/megalinter/tree/main/.config

All of this I understand is only infrastructure, none of it goes to end users but it would be nice to also put it in renovate, no? In particular pip dependencias:

https://github.com/oxsecurity/megalinter/blob/main/.config/python/dev/requirements.txt

Because of the Dockerfile only the apt ones are left and I suppose that if we do it would generate a problem like the Alpine apk that we already commented and discarded:

https://github.com/oxsecurity/megalinter/blob/main/.config/gitpod/Dockerfile

[echoix]

Gitpod as we're using it is stopping somewhere this spring, you'd have to self-host on your cloud. So, we could limit the efforts a bit.

[bdovaz]

Ok so then it would just be looking at the requirements.txt file.

Looking for references to those files I see that there is another one here: https://github.com/oxsecurity/megalinter/tree/main/server

What is the server folder @nvuillam?

[nvuillam]

Ok to remove gitpod stuff if it's not used anymore

Maybe we should upgrade to make it work with GitHub Codespaces ?

[bdovaz]

What is the server folder @nvuillam?

??

cc @nvuillam

[nvuillam]

@bdovaz it is to use MegaLinter as a server :)

Used in POC project CodeTotal -> https://github.com/oxsecurity/codetotal

Basically, it can be coupled with a UI, using websockets and Redis exchanges

[bdovaz]

I am closing this issue because what is left is no longer related to this issue itself.