Support adding modules from git repositories

[vpodzime]

Aside from using the defaul/global index, it's handy to have an
ability to add modules from just a git repository. With a
'cfbs.json' file with the repository-specific index it can be
used as a source of modules.

Ticket: CFE-3771
Changelog: 'cfbs add' now supports adding modules from git repositories with index

TODO:

• interactive "Do you want to add all modules from the repo?"
• proper support for ssh://git@example.com/some/git/repo/here
• support for cfbs add https://some/repo@some_commit_sha
• add index info to cfbs info

[cf-bottom]

Please use a company e-mail instead of v***@mykolab.com

[nickanderson]

Simple case worked for me.

❯ cd cfengine/src/cfbs
❯ ls
cfbs  test  LICENSE  README.md  requirements.txt  setup.py
❯ PYTHONPATH=. cfbs add https://github.com/nickanderson/cfengine-sysctl
Error: This is not a cfbs repo, to get started, type: cfbs init
❯ PYTHONPATH=
❯ cfbs init
Initialized - edit name and description cfbs.json
To add your first module, type: cfbs add masterfiles
❯ PYTHONPATH=. cfbs add https://github.com/nickanderson/cfengine-sysctl
Do you want to add all 3 modules from the repository? [y/N] y
Added module: inventory-kernel-settings-sysctl-current
Added module: inventory-kernel-settings-sysctl-conf
Added module: kernel-settings-sysctl-conf
❯ cat cfbs.json
{
  "name": "Example",
  "description": "Example description",
  "build": [
    {
      "name": "inventory-kernel-settings-sysctl-current",
      "description": "Inventory sysctl settings current state.",
      "tags": ["inventory", "sysctl"],
      "by": "https://github.com/nickanderson",
      "version": "1.0.0",
      "commit": "60fe0eee9b5b44ff078b12989d5057b2660c1eda",
      "subdirectory": "policy/inventory-kernel-settings-sysctl-current",
      "steps": [
        "copy ./main.cf services/inventory-kernel-settings-sysctl-current/",
        "copy ./README.org services/inventory-kernel-settings-sysctl-current/",
        "json ./cfbs/def.json def.json"
      ],
      "index": "https://github.com/nickanderson/cfengine-sysctl",
      "repo": "https://github.com/nickanderson/cfengine-sysctl",
      "added_by": "cfbs add"
    },
    {
      "name": "inventory-kernel-settings-sysctl-conf",
      "description": "Inventory settings from /etc/sysctl.conf.",
      "tags": ["inventory", "sysctl"],
      "by": "https://github.com/nickanderson",
      "version": "1.0.0",
      "commit": "60fe0eee9b5b44ff078b12989d5057b2660c1eda",
      "subdirectory": "policy/inventory-kernel-settings-sysctl-conf",
      "steps": [
        "copy ./main.cf services/inventory-kernel-settings-sysctl-conf/",
        "copy ./README.org services/inventory-kernel-settings-sysctl-conf/",
        "json ./cfbs/def.json def.json"
      ],
      "index": "https://github.com/nickanderson/cfengine-sysctl",
      "repo": "https://github.com/nickanderson/cfengine-sysctl",
      "added_by": "cfbs add"
    },
    {
      "name": "kernel-settings-sysctl-conf",
      "description": "Manage settings in /etc/sysctl.conf.",
      "tags": ["sysctl"],
      "by": "https://github.com/nickanderson",
      "version": "1.0.0",
      "commit": "60fe0eee9b5b44ff078b12989d5057b2660c1eda",
      "subdirectory": "policy/kernel-settings-sysctl-conf",
      "steps": [
        "copy ./main.cf services/kernel-settings-sysctl-conf/",
        "copy ./README.org services/kernel-settings-sysctl-conf/",
        "json ./cfbs/def.json def.json"
      ],
      "index": "https://github.com/nickanderson/cfengine-sysctl",
      "repo": "https://github.com/nickanderson/cfengine-sysctl",
      "added_by": "cfbs add"
    }
  ]
}%                                                                                                                                                                                                                 

[olehermanse]

LGTM. I have some smaller nitpicks but feel free to merge it as is, and address things in follow-ups etc.

[olehermanse]

I see you are calling it an index file. That's not what I intended, up until now we've had 2 types of JSON files cfbs.json (config file for 1 project), index.json (index of all available modules).

BUT, what you did is smart and I think we should probably change this, standardize the format so there is only 1 format, cfbs.json and it can have an index inline or external path to index.

No need to do anything here, but let's discuss this some more :)