Generate all CML repositories from a single repository

[iesahin]

This one is to add a generate.bash script that creates multiple CML example repositories from a single seed repository.

Closes #100

[iesahin]

This is mostly done. It may have duplicate files that are better to be removed but I didn't check them before your approval.

generate.bash

• Each $repo directory in github is considered a repository to be placed in https://github.com/iterative/$repo
• Each directory in $repo directory, (e.g. github/repo/experiment) is considered a branch in this repository
• The script clones iterative/example_cml first, copies the files in github/$repo/$branch/ on top of this as a new branch
• It creates a push script to push all generated repositories at once.

I didn't add gitlab and bitbucket yet, need to make sure that this is the right approach first.

• A new $hub is (gitlab, bitbucket) is added by adding a directory to example-cml/$hub
• Each $hub can have its own URL scheme for git remote.
• A new repository to github is added by example-cml/github/$repo
• A new branch to a repository is added by example-cml/github/$repo/$branch
• If you put a modify-branch.bash script to example-cml/github/$repo/$branch directory, it's run before checking in the files. This file is deleted before commit. So if there are modifications that are better be done in a script rather than copying files, you can put them here.

There was a branch/with/slashes in one of the repositories. The above structure is not compatible with branch names with slashes, so I converted it to a branch-with-dashes.

I didn't use any dvc specific code, only the push script runs dvc push and I didn't test it yet. (The push script actually pushes all the repositories and please make sure the repositories are ok before using it.)

I didn't add support for tags, but if you need them I can add another level of directories for tags.

This doesn't use any modifications to the committer name and commit date, yet. These can be modified per-hub basis.

Let me know what you think and would like to see here. Thanks.

@casperdcl @iterative/cml @shcheklein

[iesahin]

I know you're busy with the release but maybe you have time for a quick review to this one: @casperdcl

[iesahin]

ping! @casperdcl :)

[iesahin]

ping :)

[0x2b3bfa0]

🙏🏼 Thank you very much, @iesahin! This is an awesome stride in the right direction, and we should review it rather sooner than later.

@casperdcl, like the rest of the team, is a bit busy with releases, but we should track this issue somewhere so it doesn't get lost in the meanwhile.

[0x2b3bfa0]

Please ping @iterative/cml and @casperdcl as much as you want if you don't hear back from us.

[iesahin]

When do you think you could have more time to review this? @iterative/cml

[0x2b3bfa0]

I guess that the answer is going to be “after the release” 🙃

[shcheklein]

@iesahin I think it's almost impossible to review. Let's split into some baby steps please. One project at a time. 100-500 line PRs if possible ...?

[0x2b3bfa0]

example-cml/github/cml-runner-base-case/expeiment/README.md

s/\(expe\)\(iment\)/\1r\2/g

[casperdcl]

I think the approach leads to a lot of code (14k lines) which could be avoided...

1. Many of the branches and even repos are based on each other, I feel applying patches is a better approach?
2. If patch files are too clunky, maybe we can apply git commits instead. Keep things in a single generator repo and a script will push individual branches to different target repos. Whenever something needs to be updated in the root commit, all branches need to be rebased.

See also this and this

[0x2b3bfa0]

As per iterative/cml#1000 (comment), iterative/cml_dvc_case should be updated to replace vl2png ... | cml publish by vl2png ... > plot.png and cml publish plot.png

[iesahin]

I think with my time constraints until July, I'll close this as is to (maybe) reopen sometime later. Thank you for your reviews.