Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nightly: building the openedx Docker image triggers RuntimeError because of content_libraries app #726

Closed
regisb opened this issue Sep 30, 2022 · 3 comments
Closed

nightly: building the openedx Docker image triggers RuntimeError because of content_libraries app #726

regisb opened this issue Sep 30, 2022 · 3 comments
Labels
bug Bugs will be investigated and fixed as quickly as possible.

Comments

@regisb
Copy link
Contributor

regisb commented Sep 30, 2022

Bug description

On the nightly branch, building the openedx Docker image fails with the following error:

Step 75/100 : RUN ./manage.py lms --settings=tutor.i18n compilejsi18n
 ---> Running in f63aa6ff6f95
Traceback (most recent call last):
  File "./manage.py", line 103, in <module>
    startup.run()
  File "/openedx/edx-platform/lms/startup.py", line 20, in run
    django.setup()
  File "/openedx/venv/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/openedx/venv/lib/python3.8/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/openedx/venv/lib/python3.8/site-packages/django/apps/config.py", line 301, in import_models
    self.models_module = import_module(models_module_name)
  File "/opt/pyenv/versions/3.8.12/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/openedx/edx-platform/lms/djangoapps/bulk_email/models.py", line 22, in <module>
    from openedx.core.djangoapps.course_groups.cohorts import get_cohort_by_name
  File "/openedx/edx-platform/openedx/core/djangoapps/course_groups/cohorts.py", line 21, in <module>
    from lms.djangoapps.courseware import courses
  File "/openedx/edx-platform/lms/djangoapps/courseware/courses.py", line 52, in <module>
    from lms.djangoapps.courseware.module_render import get_module
  File "/openedx/edx-platform/lms/djangoapps/courseware/module_render.py", line 50, in <module>
    from xmodule.library_tools import LibraryToolsService
  File "/openedx/edx-platform/xmodule/library_tools.py", line 13, in <module>
    from openedx.core.djangoapps.content_libraries import api as library_api
  File "/openedx/edx-platform/openedx/core/djangoapps/content_libraries/api.py", line 78, in <module>
    from openedx.core.djangoapps.content_libraries import permissions
  File "/openedx/edx-platform/openedx/core/djangoapps/content_libraries/permissions.py", line 7, in <module>
    from openedx.core.djangoapps.content_libraries.models import ContentLibraryPermission
  File "/openedx/edx-platform/openedx/core/djangoapps/content_libraries/models.py", line 81, in <module>
    class ContentLibrary(models.Model):
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 113, in __new__
    raise RuntimeError(
RuntimeError: Model class openedx.core.djangoapps.content_libraries.models.ContentLibrary doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.

How to reproduce

Checkout the tutor nightly branch. Then run:

tutor images build --no-cache openedx

I believe this issue was caused by the following commit: openedx/edx-platform@1afb32c

With this change, the from xmodule.library_tools import LibraryToolsService import was moved to the lms/djangoapps/courseware/module_render.py module, which is imported during i18n asset compilation.
@regisb regisb mentioned this issue Sep 30, 2022
Merged
@regisb regisb moved this to Backlog in Tutor project management Sep 30, 2022
@regisb regisb added the bug Bugs will be investigated and fixed as quickly as possible. label Sep 30, 2022
@ormsbee ormsbee mentioned this issue Sep 30, 2022
Closed
@Agrendalath
Copy link

@regisb, reverting the commit you mentioned doesn't fix this issue, so it might have a different origin.

@regisb
Copy link
Contributor Author

regisb commented Oct 3, 2022

Thanks for taking the time to troubleshoot this issue @ormsbee @Agrendalath, I appreciate it.

<rambling>

One thing that is unclear to me is how the content_libraries app models are supposed to be loaded? Neither lms/envs/common.py, lms/envs/production.py, cms/envs/common.py or cms/envs/production.py adds openedx.core.djangoapps.content_libraries to the list of INSTALLED_APPS.

Oh, I see that "content_libraries = openedx.core.djangoapps.content_libraries.apps:ContentLibrariesConfig" is included in edx-platform's setup.py as an entrypoint of "lms.djangoapps", so I'm guessing that it should be automatically added to INSTALLED_APPS? That is, as long as we pip install -e . in the openedx Docker image. We recently removed a pip install -r requirements/local.in statement, and I'm wondering if it could have caused this issue?

Right! This was the issue. The reason why we added this pip install -r requirements/local.in statement in the first place was to recreate the egg-info folders in the "python-requirements" Docker layer. Similar issues were reported elsewhere:
https://stackoverflow.com/questions/54260307/change-egg-info-directory-with-pip-install-editable
https://jbhannah.net/articles/python-docker-disappearing-egg-info

To re-create these folders, we should run pip install -e . again in a later Docker stage, as we used to. I'll open a PR.

Sorry about the rambling, I just wanted to write down my thoughts as I was discovering the issue root cause.

</rambling>

tl;dr: a change I made caused the issue, a fix is on the way. Thanks a lot for your input!

regisb added a commit that referenced this issue Oct 3, 2022
@regisb
fix: installation of local requirements
ddbb396 
The `compilejsi18n` command was failing during image building because the
Open-edX package was not installed properly. The reason for that was an earlier
change where we got rid of the `pip install -r requirements/edx/local.in`
command. Installing the Open-edX package was part of this requirement file.

The local.in requirements file no longer exists, but we still need to `pip
install -e .` the edx-platform repo. To run this command we need both the
edx-platform repo and the virtualenv.

The good news is that there are no more local requirements in the base.txt
requirements file. This means that we no longer have to COPY the edx-platform
repo in the requirements installation step. Thus, changes in edx-platform will
no longer trigger a rebuild of the pip requirements; this means that re-builds
will be much faster when making changes to edx-platform.

Note that plugins that implemented the
"openedx-dockerfile-post-python-requirements" patch and that needed access to
the edx-platform repo will no longer work. Instead, these plugins should
implement the "openedx-dockerfile-pre-assets" patch. This scenario should be
very rare, though.

Close #726
regisb added a commit that referenced this issue Oct 3, 2022
@regisb
fix: installation of local requirements
3add196 
The `compilejsi18n` command was failing during image building because the
Open-edX package was not installed properly. The reason for that was an earlier
change where we got rid of the `pip install -r requirements/edx/local.in`
command. Installing the Open-edX package was part of this requirement file.

The local.in requirements file no longer exists, but we still need to `pip
install -e .` the edx-platform repo. To run this command we need both the
edx-platform repo and the virtualenv.

The good news is that there are no more local requirements in the base.txt
requirements file. This means that we no longer have to COPY the edx-platform
repo in the requirements installation step. Thus, changes in edx-platform will
no longer trigger a rebuild of the pip requirements; this means that re-builds
will be much faster when making changes to edx-platform.

Note that plugins that implemented the
"openedx-dockerfile-post-python-requirements" patch and that needed access to
the edx-platform repo will no longer work. Instead, these plugins should
implement the "openedx-dockerfile-pre-assets" patch. This scenario should be
very rare, though.

Close #726
@regisb regisb mentioned this issue Oct 3, 2022
Merged
regisb added a commit that referenced this issue Oct 3, 2022
@regisb
fix: installation of local requirements
49ce8b8 
The `compilejsi18n` command was failing during image building because the
Open-edX package was not installed properly. The reason for that was an earlier
change where we got rid of the `pip install -r requirements/edx/local.in`
command. Installing the Open-edX package was part of this requirement file.

The local.in requirements file no longer exists, but we still need to `pip
install -e .` the edx-platform repo. To run this command we need both the
edx-platform repo and the virtualenv.

The good news is that there are no more local requirements in the base.txt
requirements file. This means that we no longer have to COPY the edx-platform
repo in the requirements installation step. Thus, changes in edx-platform will
no longer trigger a rebuild of the pip requirements; this means that re-builds
will be much faster when making changes to edx-platform.

Note that plugins that implemented the
"openedx-dockerfile-post-python-requirements" patch and that needed access to
the edx-platform repo will no longer work. Instead, these plugins should
implement the "openedx-dockerfile-pre-assets" patch. This scenario should be
very rare, though.

Close #726
@regisb regisb moved this from Backlog to In Progress in Tutor project management Oct 3, 2022
regisb added a commit that referenced this issue Oct 3, 2022
@regisb
fix: installation of local requirements
1c4339f 
The `compilejsi18n` command was failing during image building because the
Open-edX package was not installed properly. The reason for that was an earlier
change where we got rid of the `pip install -r requirements/edx/local.in`
command. Installing the Open-edX package was part of this requirement file.

The local.in requirements file no longer exists, but we still need to `pip
install -e .` the edx-platform repo. To run this command we need both the
edx-platform repo and the virtualenv.

The good news is that there are no more local requirements in the base.txt
requirements file. This means that we no longer have to COPY the edx-platform
repo in the requirements installation step. Thus, changes in edx-platform will
no longer trigger a rebuild of the pip requirements; this means that re-builds
will be much faster when making changes to edx-platform.

Note that plugins that implemented the
"openedx-dockerfile-post-python-requirements" patch and that needed access to
the edx-platform repo will no longer work. Instead, these plugins should
implement the "openedx-dockerfile-pre-assets" patch. This scenario should be
very rare, though.

Close #726
@kdmccormick kdmccormick assigned kdmccormick and regisb and unassigned kdmccormick and regisb Oct 3, 2022
@regisb
Copy link
Contributor Author

regisb commented Oct 3, 2022

Closed by 1c4339f

@regisb regisb closed this as completed Oct 3, 2022
Repository owner moved this from In Progress to Done in Tutor project management Oct 3, 2022
regisb added a commit that referenced this issue Oct 4, 2022
@regisb
fix: installation of local requirements
a6c6fde 
The `compilejsi18n` command was failing during image building because the
Open-edX package was not installed properly. The reason for that was an earlier
change where we got rid of the `pip install -r requirements/edx/local.in`
command. Installing the Open-edX package was part of this requirement file.

The local.in requirements file no longer exists, but we still need to `pip
install -e .` the edx-platform repo. To run this command we need both the
edx-platform repo and the virtualenv.

The good news is that there are no more local requirements in the base.txt
requirements file. This means that we no longer have to COPY the edx-platform
repo in the requirements installation step. Thus, changes in edx-platform will
no longer trigger a rebuild of the pip requirements; this means that re-builds
will be much faster when making changes to edx-platform.

Note that plugins that implemented the
"openedx-dockerfile-post-python-requirements" patch and that needed access to
the edx-platform repo will no longer work. Instead, these plugins should
implement the "openedx-dockerfile-pre-assets" patch. This scenario should be
very rare, though.

Close #726
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Bugs will be investigated and fixed as quickly as possible.
Projects
Tutor project management
Status: Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@regisb @Agrendalath @kdmccormick