prowgen: operate on sub-directories

[bbguimaraes]

I decided to give this a time limit of 30m and… it was much easier to implement than I had initially thought. If people are happy with the overall design and implementation, I will work on making it prettier (documentation, comments, etc.).

As discussed in #13 and #14, --from-file is incompatible with our new pruning method. It does have advantages, though, that I think we should maintain. Namely:

• Speed: it is ~70x faster on my machine, although the difference is seconds vs. centiseconds.
• Scope: it allows someone working on jobs in a single repository (what I believe is our most common non-expert usage) to focus on their jobs. See prowgen: make --from-file imply --prune=false #14 (comment). Correctness is still enforced by the pre-submit check in openshift/release, which remains unchanged.

The problem with --from-file, AFAICT, is that variants broke the direct mapping we had from ci-operator configuration files to job files. We can no longer safely consider each input file in isolation, which is the basis of our current iteration algorithm.

We can, however, consider a sub-directory in isolation: all variants come from the same leaf directory. This change makes the iteration on both generation and pruning look only at the directories passed as non-flag arguments (previously ignored), similar to git commands, e.g. git diff -- dir0 dir1.

Backward compatibility is maintained by iterating over the entire tree when no argument is given. All tests from #15 pass. I've based this PR on them, the original changes are:

3 files changed, 26 insertions(+), 18 deletions(-)

[petr-muller]

We can, however, consider a sub-directory in isolation: all variants come from the same leaf directory.

Very nice! I like the approach.

[bbguimaraes]

So… I decided to read our documentation. What a concept, I know. Actually, I opened the file to remove the references to --from-file. But it turns out we've always supported passing a sub-directory to --from-dir: https://github.com/openshift/ci-tools/blob/6718aa2f9936aa2a55ef2d555c23ba602fbbfef3/CI_OPERATOR_PROWGEN.md#generate-prow-jobs-for-new-ci-operator-config.

I tested it and it works the same as my implementation (unsurprisingly, in retrospect), except for the pruning stage. I will, in that order, fix pruning, remove the extra code for handling non-flag arguments, and never again tell anyone to RTFM.

[petr-muller]

This unfortunately does not cover (uncommon but real) corner case when some changes make us to stop visiting a directory entirely, which happens when we remove all config for a given component. This happens when repositories are renamed or archived (I remember both happening). That's why I did not implement pruning like this.

[bbguimaraes]

Heh, I realized that five seconds after pushing, but didn't think people would be monitoring the PR =)

I'll add an integration test for that. I had a hard time yesterday trying to reconcile --from-dir with --prune, in the end I think I'll have to revert to my previous implementation with separate arguments.

I'll see if I can find some time in the next few days, I'm trying not to spend too much time on this.

[bbguimaraes]

• rebased
• added a configuration removal test
• reverted to using separate arguments
• added an integration test
• removed --from-file because it cannot work with pruning
• removed --prune becase pruning should now be safe for all operations
• removed references to --from-dir path/to/subdir from the documentation

The last two items can be reverted if there are objections.

The reason for reverting to separate arguments is that I could not find a way, with just --from-dir and --to-dir, to determine how to limit pruning without using heuristics while scanning the input directory. That becomes straightforward when the arguments are names of directories under both the input and output directories.

[bbguimaraes]

Commit order because Github has one job and fails at it:

$ git log --oneline --reverse master..
b4ee936 prowgen: test complete removal
a5f3f02 prowgen: remove --from-file
99d497c prowgen: remove --prune
605ba96 prowgen: operate on sub-directories

[petr-muller]

/lgtm
/approve

Nice job.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bbguimaraes, petr-muller

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [bbguimaraes,petr-muller]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[bbguimaraes]

/woof

[openshift-ci-robot]

@bbguimaraes:

In response to this:

/woof

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.