Feature requests around CRD API WRT file inclusion/exclusion

[erikh]

No rush on this, but the ability to include the git repo and all files (no automatic ignore lists) would be first stage goals I would really love to do for ya'll and could probably deliver in a week or two.

I do have use cases for all these but if they are poorly explained, let's discuss please. I would really like these features in the controller functionality, and am happy to make effort towards building them.

• It would be nice to have the ability to include the bare git repository as a part of the working copy
  • It would also be nice to be able to prune remote content and configuration, either optionally or by default
• It would be nice to set the mask for file exclusion
• It would also be nice to set a mask for file inclusion, the inverse where everything is excluded when this option is set.
  • Merging the two results would be applying a exclusion first to all files, then inclusion of the list, then exclusion of the specified files, so that both options hold POLS properties.

I could pass this upwards to my CI users who could then ensure no content gets into CI runs that is not intended to be shared with, say, anonymous parties.

[stefanprodan]

Hi @erikh would embedding the sourceignore in the custom resource be enough?

For example:

apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
  name: repo
spec:
  interval: 1m
  url: https://github.com/org/repo
  ref:
    branch: master
    commit: 26a630c0b4b3452833d96c511d93f6f2d2e90a99
  # .gitignore format: https://git-scm.com/docs/gitignore#_pattern_format
  ignorePatterns: |
    # exclude all
    /*
    # include git meta
    !/.git
    # include source code
    !/src
    # exclude file extensions from source code
    /src/**/*.test
    /src/**/*.out

When ignorePatterns is specified, the controller uses it exclusively, without taking into account the hardcoded defaults or .sourceignore.

[nab-gha]

I like the ignorePatterns approach. My client has a use case where they want to use the source controller to watch and retrieve the contents of a single file from the git repository.
Also, they are only interested in changes to that file, not changes to other files in the repository so it would be good if the custom resource status (transition timestamp and url) only changed when a change occurred to a file that is included by the ignorePatterns specification.
Finally, would it be possible to update the custom resource definition to specify that you want the tar file unpack the url to point to the base directory of the repo. This would enable the client to do a get on /directory/file-I-want rather than fetching the tar file and unpacking it to obtain the files required.
This would seem to me to be a good option for other use cases too because the client of the Source Controller can use it to fetch files from the repository at will. It would address another use case my client has. They have a second repository they want to retrieve files from iterating through sub directories processing each file.
Maybe these two requests should be a separate enhancement request?
Happy to contribute to this functionality if the enhancement proposal is accepted.

[stefanprodan]

Also, they are only interested in changes to that file, not changes to other files in the repository so it would be good if the custom resource status (transition timestamp and url) only changed when a change occurred to a file that is included by the ignorePatterns specification.

This could be a separate proposal after we implement the inclusion/exclusion feature.

Finally, would it be possible to update the custom resource definition to specify that you want the tar file unpack the url to point to the base directory of the repo.

I don't think this is in the source controller scope. @hiddeco @squaremo what are you thoughts on this?

[hiddeco]

The design of the source-controller is to expose one artifact per object. Sticking to this is in the best interest of the project, as I can imagine we will otherwise end up with a Harbor like solution, which is a level of complexity we are not looking for imo.

[nab-gha]

@hiddeco This makes sense although there is something to be said for presenting information in a way that is convenient to your clients. Would a GitOps Toolkit SDK repository be a good idea? We could populate this with code that clients can use to make use of GitOps Toolkit components. This could include, among other things, a function to watch a gitRepository, retrieve the tar and optionally unpack it locally?

[stefanprodan]

We can make a dev guide for developing a controller that reacts to source events and fetches & untars the artifacts. All this code is in kustomize controller, we just need to extract some code snippets and explain how it works.

[stefanprodan]

@paulcarlton-ww can you please create an issue for the above dev guide? Thanks

[nab-gha]

Done, see issue #76

[erikh]

@stefanprodan yes, that would work beautifully!

I can start on these patches later this week, if you'd like.

[stefanprodan]

@erikh sounds great, thank you!

[erikh]

OH! to those who were asking about an untarring solution, this is (for better or worse, lol) working well for me:

https://github.com/erikh/curltar -- it does exactly what you think it does, just stuff it in an init container with a hostpath or pv volume and boot a normal container afterwards with the same volume.