diff --git a/Makefile b/Makefile index cb538d0..7dec6bd 100644 --- a/Makefile +++ b/Makefile @@ -39,6 +39,9 @@ ci_requirements: validation_requirements ## sync to requirements needed for CI c dev_requirements: clean_tox piptools ## sync to requirements for local development pip-sync -q requirements/dev.txt requirements/private.* +prod_requirements: piptools ## sync virtualenv to requirements needed for production + pip-sync -q requirements/base.txt + validation_requirements: piptools ## sync to requirements for testing & code quality checking pip-sync -q requirements/validation.txt diff --git a/README.rst b/README.rst index 358180c..52724ad 100644 --- a/README.rst +++ b/README.rst @@ -20,11 +20,9 @@ This service is configured with an in-memory database simply to make Django happ Operators ========= -This is intended to be run as a fully internal service with no database or admin frontend, with the LMS and CMS making calls to it unauthenticated. **It should not be callable directly from the internet.** +**It is critical to configure this service securely**, as a misconfigured codejail-service will almost certainly allow an attacker to compromise not just this service, but possibly the rest of your infrastructure. See configuration and deployment docs for details. -While the service uses AppArmor as a first (and really, only) line of defense, it should also be deployed in a way that does not allow direct connections to other IDAs or internal services within the Open edX deployment. In the ideal situation, networking would be set up to only allow outbound connections to a predetermined set of IPs or domains. - -After any significant change to security settings, consider running the tests in ``./api_tests/``. These are run manually against a deployed service and can check for various sandbox weaknesses. +This is intended to be run as a fully internal service with no database or admin frontend, with the LMS and CMS making calls to it unauthenticated. It should not be callable directly from the internet. Getting Started with Development ******************************** @@ -33,21 +31,6 @@ Please see the Open edX documentation for `guidance on Python development`_ in t .. _guidance on Python development: https://docs.openedx.org/en/latest/developers/how-tos/get-ready-for-python-dev.html -Deploying -********* - -The service may be started using gunicorn with a command like this:: - - export DJANGO_SETTINGS_MODULE=codejail_service.settings.production - gunicorn -c codejail_service/docker_gunicorn_configuration.py \ - --bind '0.0.0.0:8080' --workers=10 --max-requests=1000 --name codejail \ - codejail_service.wsgi:application - -There is a healthcheck endpoint at ``/health/`` which responds to a -GET with ``200 OK`` if the service is running and healthy. - -TODO (``__) - Getting Help ************ diff --git a/docs/concepts/index.rst b/docs/concepts/index.rst deleted file mode 100644 index 8a2b4bd..0000000 --- a/docs/concepts/index.rst +++ /dev/null @@ -1,2 +0,0 @@ -Concepts -######## diff --git a/docs/conf.py b/docs/conf.py index a7cbf92..651b6c3 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -57,7 +57,6 @@ def get_version(*file_paths): # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ - 'sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', 'sphinx.ext.ifconfig', @@ -514,27 +513,3 @@ def get_version(*file_paths): intersphinx_mapping = { 'python': ('https://docs.python.org/3.12', None), } - - -def on_init(app): # pylint: disable=unused-argument - """ - Run sphinx-apidoc after Sphinx initialization. - - Read the Docs won't run tox or custom shell commands, so we need this to - avoid checking in the generated reStructuredText files. - """ - docs_path = os.path.abspath(os.path.dirname(__file__)) - root_path = os.path.abspath(os.path.join(docs_path, '..')) - apidoc_path = 'sphinx-apidoc' - if hasattr(sys, 'real_prefix'): # Check to see if we are in a virtualenv - # If we are, assemble the path manually - bin_path = os.path.abspath(os.path.join(sys.prefix, 'bin')) - apidoc_path = os.path.join(bin_path, apidoc_path) - check_call([apidoc_path, '-o', docs_path, os.path.join(root_path, 'codejail_service'), - os.path.join(root_path, 'codejail_service/migrations')]) - - -def setup(app): - """Sphinx extension: run sphinx-apidoc.""" - event = 'builder-inited' - app.connect(event, on_init) diff --git a/docs/deployment.rst b/docs/deployment.rst new file mode 100644 index 0000000..7a011c7 --- /dev/null +++ b/docs/deployment.rst @@ -0,0 +1,66 @@ +Configuration and deployment +############################ + +codejail-service has some unusual requirements due to its sensitive nature. **Please read all sections carefully.** + +Before you begin +**************** + +Ensure your intended deployment environment meets the following prerequisites: + +* Ubuntu, Debian, or another Linux distribution that supports AppArmor +* AppArmor 3 or newer (e.g. in Ubuntu 22.04 or newer) +* The ability to load an AppArmor profile into the host and (if using Docker) apply the profile to a Docker container + +Network +******* + +If at all possible, your codejail-service cluster should be run with the following networking restrictions: + +* No inbound access from the public internet. codejail-service lacks authentication, rate-limiting, audit logs, and other protections, and its functionality should only be exposed via Custom Python-evaluated Input Problems in the LMS and CMS. +* No outbound connections, *especially* to other services within your deployment. This will help limit the impact if an attacker were to figure out how to escape the sandbox. + +Host +**** + +The host or Docker image must be set up with a sandbox user, sandbox virtualenv, and sudoers file according to the documentation of the `codejail library `__. This forms the foundation of the sandboxing, but does not provide much security by itself. + +See `2U's Dockerfile `__ for an example of how you could set up your host or container. (As of March 2025, there is `not yet a Tutor plugin `__.) + +AppArmor +******** + +Again following the codejail library's instructions, create an AppArmor profile and load it into the host. + +This profile will need to be customized according to your service's app user, sandbox setup, and other details. It will also look different if you are using Docker or running the service directly on the host. For reference, here is `2U's AppArmor profile `__; note that the inner profile is the one that actually applies the sandboxing. + +Django settings +*************** + +Now you can set your ``CODE_JAIL`` Django setting, which tells the codejail library where the sandboxed Python executable lives, and how to limit resource usage. See the codejail library's documentation for details. (Note: Leave the ``PROXY`` setting to its default of ``0``.) + +Starting the service +******************** + +To run the service, activate a virtualenv, run ``make prod_requirements``, and start the service with ``gunicorn`` using a command like this:: + + export DJANGO_SETTINGS_MODULE=codejail_service.settings.production + gunicorn -c codejail_service/docker_gunicorn_configuration.py \ + --bind '0.0.0.0:8080' --workers=10 --max-requests=1000 --name codejail \ + codejail_service.wsgi:application + +The service is now listening on port 8080. + +There is a healthcheck endpoint at ``/health/`` which responds to a GET with ``200 OK`` if the service is running and healthy, or a ``503 Service Unavailable`` otherwise. The healthcheck is driven by a handful of security tests that the service performs. If the healthcheck is failing, this likely indicates that there is a misconfiguration that would allow unsandboxed code execution. + +Enabling code execution +*********************** + +Once the healthcheck is passing (indicating that sandboxing is probably functional), the main code-exec endpoint can be enabled with a Django setting: ``CODEJAIL_ENABLED = True``. This should not be enabled until the healthcheck passes, and should be immediately followed with API tests (see next section), which include a more comprehensive set of tests than the healthcheck performs. + +API tests +********* + +After the first setup, and after any significant change to security settings, run the tests in ``./api_tests/`` (see README in that directory). This will probe the service for a variety of possible vulnerabilities. + +These tests can also be incorporated into your deployment pipeline. diff --git a/docs/features.rst b/docs/features.rst deleted file mode 100644 index 073bbe9..0000000 --- a/docs/features.rst +++ /dev/null @@ -1,30 +0,0 @@ -.. _chapter-features: - -Feature Toggling -################ -All new features/functionality should be released behind a feature gate. This allows us to easily disable features -in the event that an issue is discovered in production. This project uses the -`Waffle `_ library for feature gating. - -Waffle supports three types of feature gates (listed below). We typically use flags and switches since samples are -random, and not ideal for our needs. - - Flag - Enable a feature for specific users, groups, users meeting certain criteria (e.g. authenticated or staff), - or a certain percentage of visitors. - - Switch - Simple boolean, toggling a feature for all users. - - Sample - Toggle the feature for a specified percentage of the time. - - -For information on creating or updating features, refer to the -`Waffle documentation `_. - -Permanent Feature Rollout -************************* -Over time some features may become permanent and no longer need a feature gate around them. In such instances, the -relevant code and tests should be updated to remove the feature gate. Once the code is released, the feature flag/switch -should be deleted. diff --git a/docs/getting_started.rst b/docs/getting_started.rst deleted file mode 100644 index cec5418..0000000 --- a/docs/getting_started.rst +++ /dev/null @@ -1,18 +0,0 @@ -Getting Started -############### - -If you have not already done so, create/activate a `virtualenv`_. Unless otherwise stated, assume all terminal code -below is executed within the virtualenv. - -.. _virtualenv: https://virtualenvwrapper.readthedocs.org/en/latest/ - - -Install dependencies -******************** -Dependencies can be installed via the command below. - -.. code-block:: bash - - $ make requirements - - diff --git a/docs/how-tos/index.rst b/docs/how-tos/index.rst deleted file mode 100644 index 5147f80..0000000 --- a/docs/how-tos/index.rst +++ /dev/null @@ -1,2 +0,0 @@ -How-tos -####### diff --git a/docs/index.rst b/docs/index.rst index 92768b8..6069997 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -6,7 +6,7 @@ codejail_service ================ -Run codejail (sandboxed Python execution) as a service +Run codejail (sandboxed Python execution) as a service. Contents: @@ -14,22 +14,14 @@ Contents: :maxdepth: 2 readme - getting_started - quickstarts/index - concepts/index - how-tos/index testing - internationalization - modules + deployment changelog decisions - features - references/index Indices and tables ################## * :ref:`genindex` -* :ref:`modindex` * :ref:`search` diff --git a/docs/internationalization.rst b/docs/internationalization.rst deleted file mode 100644 index bb98088..0000000 --- a/docs/internationalization.rst +++ /dev/null @@ -1,49 +0,0 @@ -.. _chapter-i18n: - -Internationalization -#################### -All user-facing text content should be marked for translation. Even if this application is only run in English, our -open source users may choose to use another language. Marking content for translation ensures our users have -this choice. - -Follow the `internationalization coding guidelines`_ in the edX Developer's Guide when developing new features. - -.. _internationalization coding guidelines: https://edx.readthedocs.org/projects/edx-developer-guide/en/latest/internationalization/i18n.html - -Updating Translations -********************* -This project uses `Transifex`_ to translate content. After new features are developed the translation source files -should be pushed to Transifex. Our translation community will translate the content, after which we can retrieve the -translations. - -.. _Transifex: https://www.transifex.com/ - -Pushing source translation files to Transifex requires access to the edx-platform. Request access from the Open Source -Team if you will be pushing translation files. You should also `configure the Transifex client`_ if you have not done so -already. - -.. _configure the Transifex client: https://docs.transifex.com/client/config/ - -The `make` targets listed below can be used to push or pull translations. - -.. list-table:: - :widths: 25 75 - :header-rows: 1 - - * - Target - - Description - * - pull_translations - - Pull translations from Transifex - -Fake Translations -***************** -As you develop features it may be helpful to know which strings have been marked for translation, and which are not. -Use the `fake_translations` make target for this purpose. This target will extract all strings marked for translation, -generate fake translations in the Esperanto (eo) language directory, and compile the translations. - -You can trigger the display of the translations by setting your browser's language to Esperanto (eo), and navigating to -a page on the site. Instead of plain English strings, you should see specially-accented English strings that look -like this: - - Thé Fütüré øf Ønlïné Édüçätïøn Ⱡσяєм ι# Før änýøné, änýwhéré, änýtïmé Ⱡσяєм # - diff --git a/docs/quickstarts/index.rst b/docs/quickstarts/index.rst deleted file mode 100644 index e3f408f..0000000 --- a/docs/quickstarts/index.rst +++ /dev/null @@ -1,2 +0,0 @@ -Quick Start -########### diff --git a/docs/references/index.rst b/docs/references/index.rst deleted file mode 100644 index ba5ea57..0000000 --- a/docs/references/index.rst +++ /dev/null @@ -1,2 +0,0 @@ -References -########## diff --git a/docs/testing.rst b/docs/testing.rst index 0f29e5b..53528ab 100644 --- a/docs/testing.rst +++ b/docs/testing.rst @@ -17,25 +17,12 @@ To run just the unit tests: $ make test -To run just the unit tests and check diff coverage - -.. code-block:: bash - - $ make diff_cover - To run just the code quality checks: .. code-block:: bash $ make quality -To run the unit tests under every supported Python version and the code -quality checks: - -.. code-block:: bash - - $ make test-all - To generate and open an HTML report of how much of the code is covered by test cases: diff --git a/tox.ini b/tox.ini index 48208f6..9f9535a 100644 --- a/tox.ini +++ b/tox.ini @@ -62,8 +62,6 @@ deps = -r{toxinidir}/requirements/doc.txt commands = doc8 --ignore-path docs/_build README.rst docs - rm -f docs/codejail_service.rst - rm -f docs/modules.rst make -e -C docs clean make -e -C docs html