Dynaconf integration - phase 1 (SatelliteQE#8003)

* Change settings to Dynaconf [WIP]

* Use dynaconf from PyPI

* Add simple validators

* Add complex validators

* Introduce ConfigProxy facade

Facade contains references to both Dynaconf settings and legacy
Robottelo settings, and acts as single point for all configuration
values retrieval.

Facade looks into Dynaconf first, and legacy Robottelo settings second.

Facade returns ConfigNodeWrapper objects wrapping actual configuration
value. Different type should be transparent for majority of uses, but
some external modules are picky about type of value they receive on
input.

* Allow test to pass when robottelo.properties exists

* Remove conf/ directory content, so it doesn't shadow legacy settings

* Extend exceptions list

Added all settings referred to in api_fixtures and tests/foreman/api/
They usually are used in Nailgun, where they need to be serialized to
JSON, and wrapper will inevitably fail serialization. Some settings are
probably not serialized, but better to be safe than sorry here.

* Changes after review

* Rebase - update according to recent changes

* Drop virt-who from validation - they moved to separate .properties file

* Avoid json on CLIFactoryError - use pprint instead

* Fix server.get_url implementation in facade

* Rename ConfigNodeWrapper to SettingsNodeWrapper

* Add settings introduced since last rebase to exceptions

Co-authored-by: Bruno Rocha <rochacbruno@gmail.com>

.env

@@ -0,0 +1 @@
+# This file is here for dynaconf to set this as root level directory

.gitignore

@@ -60,3 +60,14 @@ tests/foreman/pytest.ini
 
 # For `coverage xml` generated file.
 **/coverage.xml
+
+# Dynaconf local files
+/.secrets.yaml
+/.secrets_*.yaml
+/settings.local.yaml
+/settings*.local.yaml
+
+# I don't know where those 2 files come from
+# but they are always there.
+full_upgrade
+upgrade_highlights

requirements.txt

@@ -4,6 +4,7 @@
 attrdict==2.0.1
 broker>=0.1.3
 cryptography==3.2.1
+dynaconf==3.1.2
 fauxfactory==3.0.6
 Inflector==3.0.1
 navmazing==1.1.6

robottelo/cli/factory.py

@@ -2,9 +2,9 @@
 Factory object creation for all CLI methods
 """
 import datetime
-import json
 import logging
 import os
+import pprint
 import random
 import time
 from os import chmod
@@ -135,7 +135,7 @@ def create_object(cli_object, options, values):
         # If the object is not created, raise exception, stop the show.
         raise CLIFactoryError(
             'Failed to create {} with data:\n{}\n{}'.format(
-                cli_object.__name__, json.dumps(options, indent=2, sort_keys=True), err.msg
+                cli_object.__name__, pprint.pformat(options, indent=2), err.msg
             )
         )
 

robottelo/config/__init__.py

@@ -1,4 +1,48 @@
-from robottelo.config.base import Settings
+import logging
 
-#: A :class:`Settings` object.
-settings = Settings()
+from dynaconf import LazySettings
+from dynaconf.validator import ValidationError
+
+from .validators import validators
+from robottelo.config.base import ImproperlyConfigured
+from robottelo.config.base import Settings as LegacySettings
+from robottelo.config.facade import SettingsFacade
+from robottelo.config.facade import SettingsNodeWrapper
+
+logger = logging.getLogger('robottelo.config')
+
+legacy_settings = LegacySettings()
+
+dynaconf_settings = LazySettings(
+    envvar_prefix="ROBOTTELO",
+    core_loaders=["YAML"],
+    settings_file="settings.yaml",
+    preload=["conf/*.yaml"],
+    includes=["settings.local.yaml", ".secrets.yaml", ".secrets_*.yaml"],
+    envless_mode=True,
+    lowercase_read=True,
+)
+dynaconf_settings.validators.register(**validators)
+
+try:
+    legacy_settings.configure()
+except ImproperlyConfigured:
+    logger.error(
+        (
+            "Legacy Robottelo settings configure() failed, most likely required "
+            "configuration option is not provided. Continuing for the sake of unit tests"
+        ),
+        exc_info=True,
+    )
+
+try:
+    dynaconf_settings.validators.validate()
+except ValidationError:
+    logger.error(
+        "Dynaconf validation failed, continuing for the sake of unit tests", exc_info=True
+    )
+
+settings_proxy = SettingsFacade()
+settings_proxy.set_configs(dynaconf_settings, legacy_settings)
+
+settings = SettingsNodeWrapper(settings_proxy)