feat!: A Better API for Derived Settings

[kdmccormick]

Description

The Python API for declaring derived settings was confusing to the uninitiated
reader, and also prone to spelling mistakes. This replaces the API with one
that is more readable and more concise, and updates the implementation of
derive_settings to properly derive settings declared using the new API.

BREAKING CHANGE: The derived and derived_collection_entry function are
replaced with the Derived class. We do not expect those functions to have
been used outside of edx-platform, but if they are, this commit will cause them
to loudly ImportError.

Note that there should be NO change in behavior to the derive_settings
function, which we DO know to be used by some external edx-platform plugins.

Before

from openedx.core.lib.derived import derived, derived_collection_entry, derive_settings

# Define a derived default
MY_DERIVED_SETTING = lambda settings: "compute something based on" + settings.ANOTHER_SETTING

# Register that derived default. If this is misspelled, it will silently fail.
derived("MY_DERIVED_SETTING")

# Define a derived default within a collection
MY_DICT_SETTING = {
    "my_list_with_a_derived_value": [
        "value0",
        "value1",
        lambda settings: "compute value2 based on" settings.YET_ANOTHER_SETTING
    ],
   "my_static_list": ["value0", "value1", "value2"],
}

# Register that derived default within a collection. If any of these are misspelled, it will silently fail.
derived_collection_entry("MY_DICT_SETTING", "my_list_with_a_derived_value", 2)
...

derive_settings(__name__)  # This actually renders the derived settings

After

from openedx.core.lib.derived import Derived, derive_settings

# Define a derived default (no registration required)
MY_DERIVED_SETTING = Derived(lambda settings: "compute something based on" + settings.ANOTHER_SETTING)

# Define a derived default within a collection (no registration required)
MY_DICT_SETTING = {
    "my_list_with_a_derived_value": [
        "value0",
        "value1",
        Derived(lambda settings: "compute value2 based on" settings.YET_ANOTHER_SETTING),
    ],
   "my_static_list": ["value0", "value1", "value2"],
}
...

derive_settings(__name__)  # This actually renders the derived settings (same as before)

Supporting info

Part of:

• Simplify edx-platform settings #36215

Builds on a previous settings refactoring:

• Based on [Attempt 2] refactor: Clean up lms/envs/production.py cruft #36131

Next up:

• refactor: Use derive_settings to lazy load settings. #36205

Testing instructions

First, download prod-like YML files to ensure that this works in a non-Tutor environment...

• prodlike-lms.yml: https://gist.github.com/kdmccormick/73d8a2545a9c8e110439b2f521313baf
• prodlike-cms.yml: https://gist.github.com/kdmccormick/2708995e41c6b08a48f199215ab514b0

Then, download and save this to diff_settings.sh in the root of edx-platform:

#!/usr/bin/env bash
# Usage: ./diff_settings.sh GIT_REF_A GIT_REF_B
# Example: ./diff_settings.sh upstream/master myusername/myrefactorbranch
# Will fail if git state is not clean.

set -xeuo pipefail  # Be verbose and strict

REF_A=$1
REF_B=$2
DIR_A=dump_settings_a
DIR_B=dump_settings_b

SETTINGS_AND_CFG=( \
"envs.tutor.production,$LMS_CFG,$CMS_CFG" \
"envs.tutor.development,$LMS_CFG,$CMS_CFG" \
"envs.production,prodlike-lms.yml,prodlike-cms.yml" \
)

rm -rf "$DIR_A" "$DIR_B"
mkdir "$DIR_A" "$DIR_B"

for settings_and_cfg in "${SETTINGS_AND_CFG[@]}" ; do
    settings="${settings_and_cfg%%,*}"
    cfg="${settings_and_cfg#*,}"

    lms_cfg="${cfg%%,*}"
    lms_module="lms.${settings}"
    lms_outfile="${lms_module}__${lms_cfg//\//_}.json"
    git checkout "$REF_A"
    DJANGO_SETTINGS_MODULE="$lms_module" LMS_CFG="$lms_cfg" ./manage.py lms dump_settings > "$DIR_A/$lms_outfile"
    git checkout "$REF_B"
    DJANGO_SETTINGS_MODULE="$lms_module" LMS_CFG="$lms_cfg" ./manage.py lms dump_settings > "$DIR_B/$lms_outfile"

    cms_cfg="${cfg#*,}"
    if [[ -n "$cms_cfg" ]]; then
        cms_module="cms.${settings}"
        cms_outfile="${cms_module}__${cms_cfg//\//_}.json"
        git checkout "$REF_A"
        DJANGO_SETTINGS_MODULE="$cms_module" CMS_CFG="$cms_cfg" ./manage.py lms dump_settings > "$DIR_A/$cms_outfile"
        git checkout "$REF_B"
        DJANGO_SETTINGS_MODULE="$cms_module" CMS_CFG="$cms_cfg" ./manage.py lms dump_settings > "$DIR_B/$cms_outfile"
    fi
done

diff "$DIR_A" "$DIR_B" && echo "No difference!"

Finally, from within a Tutor LMS or CMS shell, run ./diff_settings.sh master kdmccormick/use-derived-tooling

Result should be "No difference!"

[kdmccormick]

@feanil , this is ready for review.

@dianakhuang , I have confirmed this PR using the redacted LMS yaml file. If you're able to provide the redacted CMS yaml, I can confirm with that as well.

[dianakhuang]

@kdmccormick working on a redacted studio file right now, so I'll try to get you one by the end of today.

[kdmccormick]

Tyty @dianakhuang -- I will keep this open until I hear from you.

[feanil]

Looks good, much easier to manage.

[kdmccormick]

Confirmed with CMS settings. Will rebase and merge.

[edx-pipeline-bot]

2U Release Notice: This PR has been deployed to the edX staging environment in preparation for a release to production.

[edx-pipeline-bot]

2U Release Notice: This PR has been deployed to the edX production environment.

[edx-pipeline-bot]

2U Release Notice: This PR has been deployed to the edX production environment.