feat: Enhance course optimizer to detect previous run links and expand scanning scope #37104

djoseph-apphelix · 2025-08-01T08:01:50Z

Description

This PR extends the functionality of the Course Optimizer page with the following enhancements:

Previous Run Link Detection:

Adds support to detect links pointing to previous course runs, controlled by a feature flagcontentstore.enable_course_optimizer_check_prev_run_links. If the flag is disabled, the scan covers only broken, external forbidden, and locked links.

Plain Text Link Scanning:

Updates link detection to also search for URLs in plain text, not just in href and src attributes.

Expanded Scanning Scope:

Includes Course Updates, Handouts, and Custom Pages in the link scan process, in addition to the existing course content.

Supporting information

Waffle Flag: contentstore.enable_course_optimizer_check_prev_run_links controls previous run detection.
Enhanced API Response: New course_updates and custom_pages fields with link categorization.
Improved URL Detection: Advanced regex patterns detect URLs in HTML attributes and plain text content.
Previous Run Link Detection: Identifies and categorizes links pointing to previous course runs using CourseRerunState

Jira

TNL2-138

Testing Instructions

With the feature flag disabled, verify that existing behavior remains unchanged and no previous-run links are returned.
Create a course rerun and ensure a CourseRerunState record is created.
Add links pointing to the original (old) course in content, updates, handouts, and custom pages of the new course rerun.
Run the Course Optimizer and verify that these links are correctly categorized as previous-run.
Confirm the API response includes course_updates and custom_pages arrays with the expected data.
Verify that plain text URLs (not just HTML anchor tags) are also detected and categorized correctly.

cms/djangoapps/contentstore/rest_api/v0/serializers/course_optimizer.py

cms/djangoapps/contentstore/tasks.py

papphelix · 2025-08-07T08:10:53Z

cms/djangoapps/contentstore/core/course_optimizer_provider.py

 from user_tasks.models import UserTaskArtifact, UserTaskStatus

-from cms.djangoapps.contentstore.tasks import CourseLinkCheckTask, LinkState
+from cms.djangoapps.contentstore.tasks import CourseLinkCheckTask, LinkState, _get_urls


_ variables should be avoided to import under PEP8 coding guidelines

let's rename _get_urls to extract_content_URLs_from_course may be to avoid PEP8 violiation.

It's updated as per PEP-8

papphelix · 2025-08-07T08:13:30Z

cms/djangoapps/contentstore/toggles.py

+# .. toggle_default: False
+# .. toggle_description: When enabled, allows the Course Optimizer to detect and update links pointing to previous course runs.
+#   This feature enables instructors to fix internal course links that still point to old course runs
+#   after creating a course rerun.


Description says its still pointing to old run, whereas its a feature to show previous run links on new run. Correct me if i am wrong @Faraz32123

Descriptions seems right to me, but little changes can be made here.

# .. toggle_description: When enabled, allows the Course Optimizer to detect and update links pointing to previous course runs. # This feature allows instructors to see & update internal course links that are still pointing to previous course run # in a new course run.

Updated description as per above.

papphelix · 2025-08-07T08:19:44Z

cms/lib/spectacular.py

@@ -10,7 +10,8 @@ def cms_api_filter(endpoints):
    """
    filtered = []
    CMS_PATH_PATTERN = re.compile(


cms_paths = [ "xblock", "videos", "video_transcripts", "file_assets", "youtube_transcripts", "link_check", "link_check_status" ] CMS_PATH_PATTERN = re.compile(r"^/api/contentstore/v0/(" + "|".join(cms_paths) + r")")

is it possible to write this way to avoid multiline regexp

Reverted the file, as these APIs are not intended to be exposed in the Swagger UI

Faraz32123

LGTM! 👌

Faraz32123 · 2025-08-08T05:39:38Z

cms/djangoapps/contentstore/core/course_optimizer_provider.py

+        main_content = None

    if main_content is None:
        main_content = {"sections": []}


can we directly add main_content = {"sections": []} in except block.

It is updated as per above.

…d scanning scope

edx-pipeline-bot · 2025-08-08T10:15:22Z

2U Release Notice: This PR has been deployed to the edX staging environment in preparation for a release to production.

edx-pipeline-bot · 2025-08-08T10:33:14Z

2U Release Notice: This PR has been deployed to the edX production environment.

edx-pipeline-bot · 2025-08-08T10:33:14Z

2U Release Notice: This PR has been deployed to the edX production environment.

edx-pipeline-bot · 2025-08-08T10:59:43Z

2U Release Notice: This PR has been deployed to the edX staging environment in preparation for a release to production.

kdmccormick · 2025-08-11T14:40:41Z

cms/djangoapps/contentstore/toggles.py

+#   after creating a course rerun.
+# .. toggle_use_cases: temporary
+# .. toggle_creation_date: 2025-07-21
+# .. toggle_target_removal_date: None


@djoseph-apphelix @Faraz32123 If this is a temporary toggle, why is there no removal date?

We'll handle this in upcoming PR!

kdmccormick · 2025-08-11T14:44:27Z

@djoseph-apphelix @Faraz32123 Could you provide me the broader context of this pull request? For example, is there a public document outlining the plan, or are there related PRs you could link to? The TNL ticket is internal to 2U so it is not accessible to me.

Faraz32123 · 2025-08-12T05:32:56Z

@djoseph-apphelix @Faraz32123 Could you provide me the broader context of this pull request? For example, is there a public document outlining the plan, or are there related PRs you could link to? The TNL ticket is internal to 2U so it is not accessible to me.

Problem: When we create a rerun of a course through publisher, internal links of a course inside course updates, handouts, custom pages or even course content still points to older run instead of newly created run. Due to which learners are redirected to previous run/course instead of staying inside the new run/course.

So, basically the purpose of this PR is to enhance the functionality of the course optimizer APIs to also detect prevRunLinks as mentioned in the PR description within a course with expanded scope and it also detects broken links. There will be more PRs to update these links & we'll link those PRs with this.

@jristau1984 Do we have any public doc for this?

djoseph-apphelix force-pushed the djoseph/TNL2-138-Update-API-endpoints-for-course-re-run-link-updation-tool branch 2 times, most recently from 1e03b7c to 3d680b5 Compare August 5, 2025 09:55

djoseph-apphelix requested review from Faraz32123 and jristau1984 and removed request for Faraz32123 and jristau1984 August 5, 2025 10:35

djoseph-apphelix force-pushed the djoseph/TNL2-138-Update-API-endpoints-for-course-re-run-link-updation-tool branch from 3d680b5 to cf48844 Compare August 5, 2025 11:39

djoseph-apphelix requested review from Faraz32123 and jristau1984 August 5, 2025 12:43

djoseph-apphelix self-assigned this Aug 5, 2025

Faraz32123 requested review from papphelix and pganesh-apphelix August 6, 2025 11:59

djoseph-apphelix force-pushed the djoseph/TNL2-138-Update-API-endpoints-for-course-re-run-link-updation-tool branch 2 times, most recently from e9a1b00 to 760d224 Compare August 6, 2025 13:11

Faraz32123 reviewed Aug 7, 2025

View reviewed changes

cms/djangoapps/contentstore/rest_api/v0/serializers/course_optimizer.py Outdated Show resolved Hide resolved

papphelix approved these changes Aug 7, 2025

View reviewed changes

cms/djangoapps/contentstore/tasks.py Outdated Show resolved Hide resolved

papphelix self-requested a review August 7, 2025 08:09

papphelix reviewed Aug 7, 2025

View reviewed changes

djoseph-apphelix force-pushed the djoseph/TNL2-138-Update-API-endpoints-for-course-re-run-link-updation-tool branch from 760d224 to d34c891 Compare August 7, 2025 09:49

papphelix approved these changes Aug 7, 2025

View reviewed changes

Faraz32123 approved these changes Aug 7, 2025

View reviewed changes

djoseph-apphelix force-pushed the djoseph/TNL2-138-Update-API-endpoints-for-course-re-run-link-updation-tool branch from 0253811 to 585bb9d Compare August 8, 2025 03:14

Faraz32123 reviewed Aug 8, 2025

View reviewed changes

djoseph-apphelix added 5 commits August 8, 2025 12:59

feat: Enhance course optimizer to detect previous run links and expan…

5d26e21

…d scanning scope

fix: apply linting rules and refactor code

6d0bb37

fix: refactor code based on pep-8 guideline

457b204

fix: added exception handling for links descriptor generator function

9449280

fix: refactor API response to handle empty data

b8bd541

djoseph-apphelix force-pushed the djoseph/TNL2-138-Update-API-endpoints-for-course-re-run-link-updation-tool branch from 585bb9d to b8bd541 Compare August 8, 2025 08:04

Faraz32123 merged commit d3eba7a into openedx:master Aug 8, 2025
49 checks passed

kdmccormick reviewed Aug 11, 2025

View reviewed changes

djoseph-apphelix mentioned this pull request Aug 25, 2025

feat: enhance Course Optimizer to update previous course links via API #37206

Merged

feat: Enhance course optimizer to detect previous run links and expand scanning scope #37104

feat: Enhance course optimizer to detect previous run links and expand scanning scope #37104

Uh oh!

Conversation

djoseph-apphelix commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Supporting information

Jira

Testing Instructions

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Faraz32123 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

edx-pipeline-bot commented Aug 8, 2025

Uh oh!

edx-pipeline-bot commented Aug 8, 2025

Uh oh!

edx-pipeline-bot commented Aug 8, 2025

Uh oh!

edx-pipeline-bot commented Aug 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kdmccormick commented Aug 11, 2025

Uh oh!

Faraz32123 commented Aug 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

djoseph-apphelix commented Aug 1, 2025 •

edited

Loading