Avoid storing copies of files in layer for kubeval, shellcheck, and misspell

[echoix]

I logged into an interactive shell of the container to be able to see where the disk space was taken.

$ docker run -it --entrypoint /bin/bash megalinter/megalinter:v5

We can see that at the root, there are extra files

$ ls -lah --color=always
total 11M
drwxr-xr-x    1 root root    4.0K Jan 31 08:50 .
drwxr-xr-x    1 root root    4.0K Jan 31 08:50 ..
-rwxr-xr-x    1 root root       0 Jan 31 08:50 .dockerenv
-rw-r--r--    1 root root      19 Jan 30 16:35 .npmrc
-rw-r--r--    1  503 dialout  646 Oct  5  2020 LICENSE
-rw-r--r--    1  503 dialout  917 Oct  5  2020 README.md
drwxr-xr-x    3 root root    4.0K Jan 30 17:04 action
drwxr-xr-x    1 root root    4.0K Jan 30 16:58 bin
-rwxr-xr-x    1 root root     42K Jan 30 16:46 coursier
drwxr-xr-x    5 root root     360 Jan 31 08:50 dev
drwxr-xr-x    2 root root    4.0K Jan 30 17:03 dist
-rwxr-xr-x    1 root root     55K Jan 30 16:39 dotnet-install.sh
-rwxr-xr-x    1 root root    1.3K Jan 30 16:31 entrypoint.sh
drwxr-xr-x    1 root root    4.0K Jan 31 08:50 etc
drwxr-xr-x    4 root root    4.0K Jan 30 16:38 false
drwxr-xr-x    1 root root    4.0K Jan 30 16:46 go
drwxr-xr-x    1 root root    4.0K Jan 30 16:53 home
-rw-r--r--    1 root root    8.7K Jan 30 16:58 install-misspell.sh
-rwxr-xr-x    1  503 dialout 7.1M Mar 30  2021 kubeval
-rw-r--r--    1 root root    2.8M Jan 30 16:49 kubeval-linux-amd64.tar.gz
drwxr-xr-x    1 root root    4.0K Jan 30 16:48 lib
drwxr-xr-x    2 root root    4.0K Jan 30 16:48 lib64
drwxr-xr-x    5 root root    4.0K Aug 31 13:34 media
drwxr-xr-x    6 root root    4.0K Jan 30 17:03 megalinter
drwxr-xr-x    3 root root    4.0K Jan 30 17:03 megalinter-descriptors
drwxr-xr-x    2 root root    4.0K Jan 30 17:03 megalinter.egg-info
drwxr-xr-x    2 root root    4.0K Aug 31 13:34 mnt
drwxr-xr-x 1104 root root     36K Jan 30 16:38 node_modules
drwxr-xr-x    1 root root    4.0K Jan 30 17:01 opt
dr-xr-xr-x  308 root root       0 Jan 31 08:50 proc
drwx------    1 root root    4.0K Jan 30 16:59 root
drwxr-xr-x    2 root root    4.0K Aug 31 13:34 run
drwxr-xr-x    1 root root    4.0K Jan 30 16:33 sbin
drwxr-xr-x    2 root root    4.0K Jan 30 16:47 shellcheck-stable
drwxr-xr-x    2 root root    4.0K Aug 31 13:34 srv
dr-xr-xr-x   11 root root       0 Jan 31 08:50 sys
drwxrwxrwt    1 root root    4.0K Jan 30 16:59 tmp
drwxr-xr-x    1 root root    4.0K Jan 30 16:48 usr
drwxr-xr-x    1 root root    4.0K Jan 30 16:33 var

I made my PR by testing with the terraform flavor, since it was smaller than all, and contained the kubeval tool.
But I continued to explore what else could take space.

To drill down where the space is taken, we can use ncdu

$ apk add ncdu
$ ncdu -e --color dark -2 -r

Expand screenshots of disk usage

---

---

Here is something problematic
There seems to be a python 3.8 install, but we are already in an python:3.9.7-alpine3.13 image.

Expand screenshots of disk usage - python3.8 and next

__pycache__ can be safely deleted since it is where the .pyc files (compiled versions) for the interpreter are stored (they are made when running a .py file).

---

---

---

---

---

Here, are we sure we need FSharp? We could also take a look at the dotnet images to make sure we really need the full SDK https://hub.docker.com/_/microsoft-dotnet or we can make a published version of the tools we use, and thus only need a runtime base to run a .dll https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/docker/building-net-docker-images?view=aspnetcore-6.0

---

What is this false folder? It seems like a boolean value was used as a string and made a folder instead of ignoring it.

---

Go cache to clear

---

/tmp might be to clear at each layer:

---

---

R:

Proposed Changes

1. It is only a start with these three tools. When files are dowloaded, extracted, and copied at the final folder, the downloaded archive and the extracted archive are kept in the layer. Since the all flavor is 7.6 GB extracted, there is some optimisation to do.

Readiness Checklist

Author/Contributor

• If documentation is needed for this change, has that been included in this pull request
  • Not breaking

Reviewing Maintainer

• Label as breaking if this is a large fundamental change
• Label as either automation, bug, documentation, enhancement, infrastructure, or performance

[nvuillam]

@echoix wow, what an analysis, thanks ! :)

Some installation instructions already have "cleanup" script lines, but having them on all linters could dramatically enhance performances ^^

But to apply changes in descriptors to Dockerfiles, you need to run bash build.sh at the root of the repo :) (descriptors are read to dynamically generate Dockerfile content)

https://megalinter.github.io/latest/contributing/

[echoix]

Oh, I missed that, but I figured out that these were the source of the changes. I thought that the Dockerfiles were generated by themselves since it says to not edit them everywhere.

I'm not sure I have a working Linux installation that is ready to run that build script though, I'm on Windows. Docker I'm fine, but the closest I have is WSL2 or msys2, both which I don't think they're constant enough.

It'll be for mother day.

[nvuillam]

@echoix i'm on windows too, just use Git bash (installed with git for windows) and it will work fine :)

[codecov-commenter]

Codecov Report

Merging #1229 (bd4776e) into main (1e261f6) will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1229      +/-   ##
==========================================
+ Coverage   84.49%   84.52%   +0.02%     
==========================================
  Files         142      142              
  Lines        3470     3470              
==========================================
+ Hits         2932     2933       +1     
+ Misses        538      537       -1     

Impacted Files	Coverage Δ
megalinter/reporters/UpdatedSourcesReporter.py	89.18% <0.00%> (+2.70%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1e261f6...bd4776e. Read the comment docs.

[nvuillam]

That's great, thanks a lot for your contribution :)

• plz add "MLDL" in .cspell.json to cspell check passes :)

[echoix]

Did you have a better idea of folder structure for the downloads? And following this, a better idea for the Megalinter download variable? It was just a draft. For the build procedure, it makes senses, but afterwards, it's where the "license" files are kept to try to be compliant with the majority of licenses, and its not that obvious..

[echoix]

Oh, and instead of changing the folder name each time, is there a way to reuse the linter_name of the yaml in the commands edited, or theres no way out of manually specifying it?

[nvuillam]

Did you have a better idea of folder structure for the downloads? And following this, a better idea for the Megalinter download variable? It was just a draft. For the build procedure, it makes senses, but afterwards, it's where the "license" files are kept to try to be compliant with the majority of licenses, and its not that obvious..

I think you can also remove the license files, as all of them are replicated in MegaLinter documentation :)

[nvuillam]

Oh, and instead of changing the folder name each time, is there a way to reuse the linter_name of the yaml in the commands edited, or theres no way out of manually specifying it?

I'm not shocked if we have to specify manually in descriptor files, but maybe there is a repeating pattern of find/replace within RUN instructions that could be applied in build.py ?

Something like when we detect {{ML_DOWNLOAD_DIR}} within RUN instructions, we build a python variable /download/linter_name (ex: /download/shellcheck) , then we add mkdir -p ${MLDL}

Example before:

RUN  wget -qO- "https://github.com/koalaman/shellcheck/releases/download/stable/shellcheck-stable.linux.x86_64.tar.xz" | tar -xJv -C {{ML_DOWNLOAD_DIR}} \
    && mv "{{ML_DOWNLOAD_DIR}}/shellcheck-stable/shellcheck" /usr/bin/ \
    && shellcheck --version

Example after:

RUN mkdir -p /download/shellcheck \
    && wget -qO- "https://github.com/koalaman/shellcheck/releases/download/stable/shellcheck-stable.linux.x86_64.tar.xz" | tar -xJv -C /download/shellcheck \
    && mv "/download/shellcheck/shellcheck-stable/shellcheck" /usr/bin/ \
    && shellcheck --version \
    && find /download/shellcheck -type f -not -name 'LICENSE*' -delete -o -type d -empty -delete \

[echoix]

In that case,

Something like when we detect {{ML_DOWNLOAD_DIR}} within RUN instructions, we build a python variable /download/linter_name (ex: /download/shellcheck) , then we add mkdir -p ${MLDL}

Hmm... I'm not sure adding another layer of abstraction is really advantageous here. At some place we need a build recipe, and none of the software are the same neither. Maybe something that we can understand and debug when changes need to be made can worth more than trying to uniformize too much. All these commands come from an adaptation of the author's original instructions, so changing less (and not hiding what is done) is good.

And for the licences, if we don't need, then the commands could be simplified to removing the folder completely (on a case by case verification that everything is copied elsewhere). But I'm reading if to fully respect some of the licenses that say that when we distribute, we must include the present file, if that means in the container.
https://www.linuxfoundation.org/tools/docker-containers-what-are-the-open-source-licensing-considerations/
https://fossa.com/blog/containers-open-source-license-compliance/
https://jolts.world/index.php/jolts/article/view/147/268

[nvuillam]

You're probably right about licenses :)

So if no additional level of abstraction, let's keep your way by defining things manually :)

[echoix]

You're probably right about licenses :)

Oh, I think this should be usefull https://developers.redhat.com/articles/2021/11/11/best-practices-building-images-pass-red-hat-container-certification#best_practice__7__embed_license_information_inside_your_image
Do we have a folder with the licences contained that can be copied?

On second thought, only leaving the licence files should do it.

[echoix]

After the CI passes, it's going to be ok to merge ;)

[nvuillam]

@echoix so it passed :)
If you have ideas to save space with other linters installation instructions, they'll be very welcome ^^

[echoix]

Of course, it's a start!
And I didn't have time to write a note in an issue, but go take a look at issue 1550 of flake8. It's already fixed in their master. It will make the automated pull request less noisy