monorepo support

[wesley-dean-flexion]

A colleague of mine has a project that's stored in a monorepo and he tells me that it's difficult for him to integrate Megalinter into his workflow as a result. I did a quick search and found #1110 and saw that it didn't go anywhere, so I'm working from the assumption that implementing something is more challenging than is immediately apparent.

I started thinking down the path of how chroot works in terms of generalizations (that is, not the chroot system call) but that didn't get me very far.

Next, I looked through the environment variables documentation and I got to thinking about having multiple GitHub Actions (for example), each using MEGALINTER_CONFIG environment variable pointing to a different .mega-linter.yml file, each of which be specific to each portion of the monorepo, each of which would have paths to different (tool, monorepo)-specific configuration files.

Do you think something like that could work? Do you anticipate any problems? Would logging / reports be problematic? PR comments?

If it looks like something like this might be feasible, I would be happy to write down how it was setup and submit a PR to incorporate it into the documentation, if that'll be helpful.

[nvuillam]

@wesley-dean-flexion MegaLinter own repo has 2 configurations:

• one for whole repo except mega-linter-runner megalinter (python)
• one only for folder mega-linter-runner (javascript)

I followed exactly what you proposed: one Github action + one .mega-linter.yml by folder:

• whole repo except mega-linter-runner:

  • Config: https://github.com/oxsecurity/megalinter/blob/main/.mega-linter.yml
  • Workflow: https://github.com/oxsecurity/megalinter/blob/main/.github/workflows/mega-linter.yml

• mega-linter-runner :

  • Config: https://github.com/oxsecurity/megalinter/blob/main/mega-linter-runner/.mega-linter.yml
  • Workflow: https://github.com/oxsecurity/megalinter/blob/main/.github/workflows/mega-linter-for-runner.yml
    • contains DEFAULT_WORKSPACE: mega-linter-runner

I think that for a monorepo, you can do what has been done for mega-linter-runner :)

Please let me know if it works, if so I think it deserves a page in the documentation :)

[pfiaux]

We've been able to run mega-linter on a monorepo with a single config using a different approach.

• We have our own change detection so we use MEGALINTER_FILES_TO_LINT to pass in the files
• Our config makes use of _FILTER_REGEX_INCLUDE/_FILTER_REGEX_EXCLUDE on the different linters so we can define where we want which linters to run.

That means each linters are limited to a single configuration tho and same for megalinter (which keeps it simpler and less confusing I think)

(note this worked with v6 because we used ^(folder1|folder2) style regex which seem broken in v7, #2744)

[nvuillam]

@pfiaux I'd be very interested to know more about this use of MegaLinter with MEGALINTER_FILES_TO_LINT , would you like to share your configuration ? :)
Probably we may disable the file browsing stuff when MEGALINTER_FILES_TO_LINT is sent

[pfiaux]

Of course. We generate our own list of files with changes (CHANGESET_JSON) in our pipelines which includes all the files added,modified,removed relative to the change. For the linter only added and modified are relevant.

Our image runs a CI agent, when it gets a job it will call a custom start script:

  CHANGESET_JSON="$CHANGESET_JSON" \
      DEFAULT_WORKSPACE="$REPO_ROOT" \
      MEGALINTER_CONFIG=mega-linter.yaml \
      /megalinter.sh "$@"

The script will take the list of files added & modified from CHANGESET_JSON and updates the configuration appendin MEGALINTER_FILES_TO_LINT and the files. We had to do that to avoid size limits with ENV variables/arguments. Then it runs megalinter entrypoint.sh

The base config (with MEGALINTER_FILES_TO_LINT not yet added) looks like this:

VALIDATE_ALL_CODEBASE: false
APPLY_FIXES: none
FAIL_IF_MISSING_LINTER_IN_FLAVOR: true
FILEIO_REPORTER: false
FLAVOR_SUGGESTIONS: false
LOG_LEVEL: INFO
PRINT_ALPACA: false
SHOW_ELAPSED_TIME: true
SHOW_SKIPPED_LINTERS: false
EXCLUDED_DIRECTORIES: []
SECURED_ENV_VARIABLES: []

# Other things we have:
PLUGINS: []
ENABLE_LINTERS: []
# ...blocks of _FILTER_REGEX_INCLUDE/_FILTER_REGEX_EXCLUDE for the various linters
# typically enabling/disabling per team paths
GO_REVIVE_FILTER_REGEX_INCLUDE: ^(tools|team1|team2)
GO_REVIVE_FILTER_REGEX_EXCLUDE: ^(tools/xyz|team1/xyz)
JAVA_CHECKSTYLE_FILTER_REGEX_INCLUDE: ^(team3)

Regarding the use of our own list of files with changes, the main reason is we have that early on in the pipeline and then reuse that list for multiple steps. By using our list rather than letting megalinter generate one we insure consistency for which files we decide to handle in the pipeline or not.
Also early on when we tested megalinter we found that for pull requests it behaved differently than our setup linting against head on trunk. Our setup uses git merge-base $trunk_branch HEAD to determine the base commit. This prevents lint issues that aren't relevant from creeping into pull requests.

Let me know if you have more questions or need more details.

[nvuillam]

@pfiaux thanks a lot for the very interesting details :)

I'm currently overloaded but don't hesitate to mention me later so I can have a closer look about how we can make MegaLinter fit 100% of your requirements, and maybe later that can lead to an article or documentation explaining how it's supposed to work :)

[pfiaux]

I think we're very close to having it work out of the box at least in CI:

• with v7 we have the issue from Megalinter 7.x _REGEX path handling seems not fully compatible with 6.x #2744
• If we could point to a file containing MEGALINTER_FILES_TO_LINT we wouldn't need to adjust the config on the fly (it could be another variable like MEGALINTER_FILES_TO_LINT_PATH which would take either a plaintext, yaml or json list)

We're currently not able to run it locally, most of us have macOS machines and with file virtualization bottlenecks in Docker the git operations that megalinter runs take minutes to complete (as you mentioned could be optimized or skipped when files to lint are given). But in most cases engineers can run the specific linters relevant to them from their IDE.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed in 14 days if no further activity occurs.
Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.