tenant_parser.py: add support for extra-config-paths

[swiekr]

When creating/updating repo_map, 'extra-config-paths' list is extracted from tenant configuration and saved in repo's 'tenants' dictionary (next to 'jobs' and 'roles')

Extra-config-paths values are used for Scraper class initialization. The 'scrape_job_files' method extends the whitelist with it

'test_integration' tests were extended to verify this new functionality

[felixedel]

If I understand this correctly, this whole block is about creating a default empty list and adding the tenant to it. In that case we could simplify this to:

if extra_config_paths:
    repo_tenant_extra_config_paths = repo_tenant_entry["tenants"].setdefault("extra_config_paths", defaultdict(lambda: []))
    repo_tenant_extra_config_paths[extra_config_path].append(tenant)

[swiekr]

yes.
Nice optimization :)

[swiekr]

applied

[swiekr]

I have tested the behavior using following test:

from pprint import pprint as pprint
from collections import defaultdict

repo_map = {}
connection_name = "github"

# project_name, tenant , extra_config_paths
test_cases = [
    ("org1/repo1", "foo", []),
    ("org1/repo2", "foo", ["zuul-extra.d"]),
    ("org1/repo2", "bar", ["zuul-extra.d", ".zuul-extra.yaml"]),
    ("org1/repo2", "baz", [".zuul-extra.yaml"]),
    ("org1/repo2", "bud", []),
]

for case in test_cases:
    project_name = case[0]
    tenant = case[1]
    extra_config_paths = case[2]

    repo_tenant_entry = repo_map.setdefault(
        project_name,
        {"tenants": {"jobs": [], "roles": []}, "connection_name": connection_name},
    )

    repo_tenant_entry["tenants"]["jobs"].append(tenant)
    repo_tenant_entry["tenants"]["roles"].append(tenant)

    if extra_config_paths:
        repo_tenant_extra_config_paths = repo_tenant_entry["tenants"].setdefault("extra_config_paths", defaultdict(lambda: []))
        for extra_config_path in extra_config_paths:
            repo_tenant_extra_config_paths[extra_config_path].append(tenant)

    pprint(repo_map)
    print("------------------------------------------------")

for item in repo_map["org1/repo2"]["tenants"]["extra_config_paths"].items():
    print(item)

results are OK (repoMap structure is as we would expect it: no 'extra_config_paths' dict in repos without it defined):

[...]
{'org1/repo1': {'connection_name': 'github',
                'tenants': {'jobs': ['foo'], 'roles': ['foo']}},
 'org1/repo2': {'connection_name': 'github',
                'tenants': {'extra_config_paths': defaultdict(<function <lambda> at 0x7fd5836901f0>,
                                                              {'.zuul-extra.yaml': ['bar',
                                                                                    'baz'],
                                                               'zuul-extra.d': ['foo',
                                                                                'bar']}),
                            'jobs': ['foo', 'bar', 'baz', 'bud'],
                            'roles': ['foo', 'bar', 'baz', 'bud']}}}
------------------------------------------------
('zuul-extra.d', ['foo', 'bar'])
('.zuul-extra.yaml', ['bar', 'baz'])

[felixedel]

Thanks for fixing that 😉