Catalog edx-platform tests that fail, both for Tutor and for Devstack

[kdmccormick]

Context

There are some tests that don't work in Tutor. There are also some tests that don't work in Devstack. For the rollout of Tutor to 2U developers, it would be reasonable to ensure that the tests that pass in Devstack also pass in Tutor.

To achieve this, we need to catalog which tests currently don't work. In this issue, we focus on edx-platform tests. A future issue will capture tests for IDAs and MFEs.

Acceptance

• Run all edx-platform test suites (python unit, js unit, python quality, js quality) within both Devstack and Tutor on master.
• Catalog the tests or suites that fail.
• Make new issue(s) to fix tests that fail in Tutor but pass in Devstack.

Results

Sheet: https://docs.google.com/spreadsheets/d/1CHiFOh2RpPlljNNF5rltEgOLGpKe1W35oy5Qms0-JXo/edit#gid=0
Issue list: #110

Sheet: https://docs.google.com/spreadsheets/d/1iu-Z3J_P_1m2pGIoTXAqx0YnQ5cJx6k85VnMpRSGvag/edit#gid=778843154
Single issue for devstack parity: #156
Longer-term issue, for resolving all Tutor test failures: openedx/edx-platform#32353

[regisb]

For the record, here are the instructions on how to run tests in Tutor: https://docs.tutor.overhang.io/dev.html#running-edx-platform-unit-tests

It's really too bad that we have to define these environment variables manually to be able to run the test suite. I wish that we could simply run pytest to run all the unit tests.

[kdmccormick]

Strongly agree @regisb . If possible I think that is something we should fix in edx-platform itself--no sense in complicating Tutor or its docs to work around something that is clearly wrong upstream.

[kdmccormick]

@regisb Based on your comment I split out a separate ticket: https://github.com/overhangio/2u-tutor-adoption/issues/48

[Carlos-Muniz]

I'm tackling this issue. Feel free to assign it to me!
I'll be running the tests via Tutor's nightly branch.

[Carlos-Muniz]

Here is the complete list of pytests and their results ran in Devstack and Tutor.

[Carlos-Muniz]

@kdmccormick Completed the xmodule pytests for Tutor and Devstack and recorded it in the spreadsheet, and made new issues for each of the pytest classes.

[kdmccormick]

All issues for fixing tests are collected here: #110

[kdmccormick]

Since @Carlos-Muniz last ran these, looks like the set of failing tests has changed. Furthermore, we've realized that the test randomization behavior causes ddt tests to be renamed (based on their order) which caused issues when comparing tests in the spreadsheet. Carlos will be re-running and cataloging the tests.

[Carlos-Muniz]

@kdmccormick A new issue #155 has been created for the latest running of pytests in Devstack and Tutor. Issue #110, and related issues can also be closed.

[kdmccormick]

Cataloged here, in a new sheet: https://docs.google.com/spreadsheets/d/1iu-Z3J_P_1m2pGIoTXAqx0YnQ5cJx6k85VnMpRSGvag/edit#gid=778843154