There are failing tests when CERTIFICATE_LINKEDIN setting is configured at common.py

[haftamuk]

In order to be able to share course certificate to LinkedIn, an operator has to configure SOCIAL_SHARING_SETTINGS with a value True for CERTIFICATE_LINKEDIN. The setting is not available by default.

The following configuration will enable Sharing to LinkedIn on lms.

############################# SOCIAL MEDIA SHARING #############################
SOCIAL_SHARING_SETTINGS = {
    # Note: Ensure 'CUSTOM_COURSE_URLS' has a matching value in lms/envs/common.py
    'CUSTOM_COURSE_URLS': False,
    'DASHBOARD_FACEBOOK': False,
    'CERTIFICATE_FACEBOOK': False,
    'CERTIFICATE_TWITTER': False,
    'DASHBOARD_TWITTER': False,
    'CERTIFICATE_LINKEDIN': True
}

The problem

Once the setting is configured as above(set either True/False), you see there are failing tests.

To Reproduce

Steps to reproduce the behavior:

1. Configure common.py as bellow

############################# SOCIAL MEDIA SHARING #############################
SOCIAL_SHARING_SETTINGS = {
    # Note: Ensure 'CUSTOM_COURSE_URLS' has a matching value in lms/envs/common.py
    'CUSTOM_COURSE_URLS': False,
    'DASHBOARD_FACEBOOK': False,
    'CERTIFICATE_FACEBOOK': False,
    'CERTIFICATE_TWITTER': False,
    'DASHBOARD_TWITTER': False,
    'CERTIFICATE_LINKEDIN': False
}

2. Run the following tests

app@6ab2e0cce482:~/edx-platform$ python -Wd -m pytest -p no:randomly --ds=lms.envs.test ./common/djangoapps/student/tests/

3. See failing tests
   When set to False, you will see the following failing tests:

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================================================================================================== slowest 20 durations ====================================================================================================================
6.72s setup    common/djangoapps/student/tests/test_activate_account.py::TestActivateAccount::test_account_activation_invalid_next_url_redirects_dashboard
2.74s call     common/djangoapps/student/tests/test_views.py::StudentDashboardTests::test_dashboard_with_resume_buttons_and_view_buttons
1.81s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_in_proctored_course_1___honor___False_
1.70s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_1___________audit__
1.69s call     common/djangoapps/student/tests/test_views.py::StudentDashboardTests::test_resume_course_appears_on_dashboard
1.63s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_with_invalid_course_id
1.59s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_upgrade_proctoring_enrollment_3_professional
1.50s call     common/djangoapps/student/tests/test_email.py::EmailChangeConfirmationTests::test_new_email_fails
1.44s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_score_recalculation_on_enrollment_update
1.43s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_in_proctored_course_7___credit___False_
1.43s call     common/djangoapps/student/tests/test_admin_views.py::TestUserProfileAutocompleteAdmin::test_admin_inline_autocomplete_urls_render
1.41s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_in_proctored_course_3___verified___True_
1.40s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_3____honor____verified____audit_____course_modes_choose____honor__
1.39s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_unenroll_when_unenrollment_disabled
1.36s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_with_email_opt_in_10____no_id_professional_____true__
1.35s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_with_email_opt_in_02_______false__
1.34s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_with_email_opt_in_05____honor____verified_____false__
1.33s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_in_proctored_course_5___professional___True_
1.33s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_with_email_opt_in_07____professional_____true__
1.31s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_invalid_enrollment_action
================================================================================================================== short test summary info ===================================================================================================================
FAILED common/djangoapps/student/tests/test_certificates.py::CertificateDisplayTest::test_post_to_linkedin_visibility - AssertionError: False is not true : Couldn't find 'Add Certificate to LinkedIn Profile' in response
FAILED common/djangoapps/student/tests/tests.py::DashboardTest::test_linked_in_add_to_profile_btn_with_certificate - AssertionError: False is not true : Couldn't find 'Add Certificate to LinkedIn' in response
============================================================================================ 2 failed, 583 passed, 6 skipped, 79651 warnings in 154.15s (0:02:34) ============================================================================================
app@6ab2e0cce482:~/edx-platform$

4. Change configuration as bellow

############################# SOCIAL MEDIA SHARING #############################
SOCIAL_SHARING_SETTINGS = {
    # Note: Ensure 'CUSTOM_COURSE_URLS' has a matching value in lms/envs/common.py
    'CUSTOM_COURSE_URLS': False,
    'DASHBOARD_FACEBOOK': False,
    'CERTIFICATE_FACEBOOK': False,
    'CERTIFICATE_TWITTER': False,
    'DASHBOARD_TWITTER': False,
    'CERTIFICATE_LINKEDIN': True
}

5. Run the following tests

app@6ab2e0cce482:~/edx-platform$ python -Wd -m pytest -p no:randomly --ds=lms.envs.test ./common/djangoapps/student/tests/

6. See failing tests
   When set to True, you will see the following failing tests:

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================================================================================================== slowest 20 durations ====================================================================================================================
6.45s setup    common/djangoapps/student/tests/test_activate_account.py::TestActivateAccount::test_account_activation_invalid_next_url_redirects_dashboard
2.56s call     common/djangoapps/student/tests/test_views.py::StudentDashboardTests::test_dashboard_with_resume_buttons_and_view_buttons
1.56s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_upgrade_proctoring_enrollment_4_executive_education
1.50s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_1___________audit__
1.48s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_cea_enrolls_only_one_user
1.43s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_upgrade_proctoring_enrollment_2_masters
1.41s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_in_proctored_course_7___credit___False_
1.41s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_unenroll_when_unenrollment_disabled
1.41s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_in_proctored_course_5___professional___True_
1.40s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_with_email_opt_in_12____no_id_professional____None_
1.39s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_score_recalculation_on_enrollment_update
1.39s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_user_not_authenticated
1.37s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_with_email_opt_in_04____honor____verified_____true__
1.37s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_3____honor____verified____audit_____course_modes_choose____honor__
1.35s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_external_course_updates_signal_1__1
1.35s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_external_course_updates_signal_3_1
1.32s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_with_email_opt_in_07____professional_____true__
1.32s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_with_email_opt_in_10____no_id_professional_____true__
1.31s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_in_proctored_course_3___verified___True_
1.30s call     common/djangoapps/student/tests/test_enrollment.py::EnrollmentTest::test_enroll_with_email_opt_in_02_______false__
================================================================================================================== short test summary info ===================================================================================================================
FAILED common/djangoapps/student/tests/test_certificates.py::CertificateDisplayTest::test_post_to_linkedin_visibility - AssertionError: 1 != 0 : Response should not contain 'Add Certificate to LinkedIn Profile'
FAILED common/djangoapps/student/tests/tests.py::CourseEndingTest::test_cert_info - AssertionError: assert {'can_unenrol...C3%A9dX', ...} == {'can_unenrol...l': None, ...}
FAILED common/djangoapps/student/tests/tests.py::CourseEndingTest::test_cert_info_beta_tester - AssertionError: assert {'can_unenrol...C3%A9dX', ...} == {'can_unenrol...l': None, ...}
FAILED common/djangoapps/student/tests/tests.py::DashboardTest::test_linked_in_add_to_profile_btn_not_appearing_without_config - AssertionError: 1 != 0 : Response should not contain 'Add Certificate to LinkedIn'
============================================================================================ 4 failed, 581 passed, 6 skipped, 79651 warnings in 139.06s (0:02:19) ============================================================================================
app@6ab2e0cce482:~/edx-platform$ 

Expected Behavior

Tests, regardless of configuration settings should be able to test functionalities by mocking inputs or overriding setting information independently.

[feanil]

Unfortunately not all of our tests correctly configure the settings to test their expected behavior. ie. They expect the defaults to be a specific set.

However, this does not mean the tests are broken, they test functionality that is not the default behavior by overriding settings for specific test cases. You can grep for override_settings to see this in action in various tests. In the case of the LinkedIn tests, it's a bit more confusing since there is a django setting but also a config_model and you can see that the test tests both the version where the model is disabled and when it's enabled.

If you want to make PRs to improve the tests to be more resilient to changes to the default config, I'd be happy to review them but I don't consider this a high priority issue that we need to fix today. We do have decent coverage and you could look at coverage data to confirm that specific code-paths are being tested.

Let me know if you disagree, otherwise I'll close this as "Won't Do" in a week or so.

[haftamuk]

If you want to make PRs to improve the tests to be more resilient to changes to the default config, I'd be happy to review them

Using test util imported as with_site_configuration_context, we can associate configuration context to our tests that should make the tests independent and more autonomous from config settings at common.py.

[haftamuk]

Unfortunately not all of our tests correctly configure the settings to test their expected behavior. ie. They expect the defaults to be a specific set.

The case on linkedin is not an issue of a default value! There are failing tests regardless if the value for CERTIFICATE_LINKEDIN is True or False.

[haftamuk]

@feanil Thank you for going through the Issue. If you may kindly have a look at the proposed changes and let me know what you think.

Thank you.

[Cup0fCoffee]

@haftamuk I tend to agree with Feanil. The tests are not wrong. The tests were written assuming there is no default value defined in the common.py and they didn't mock any values. If you look at these lines, you will see that in tests the model is assumed to be enabled. When you introduce a value to the settings file, this behavior is interrupted, and tests start failing. The tests are still correctly testing the behavior, they are just not well isolated. In your PR you don't have to add a default value for the settings. However, if you do want to add a default value, you can, just fix the isolation of the failing tests as well. But otherwise, Feanil is right, this is low priority.