Tasks remaining to get to Python 3

[cclauss]

Planning Doc

• Fix tests for sort order differences in Python 3 Fix tests for sort order differences in Python 3 #3330
• Coverstore/warc.py py3 again Coverstore/warc.py py3 again #3331
• The cmp argument again The cmp argument again #3332
• Prevent PRs from breaking Python 3 tests Prevent PRs from breaking Python 3 tests #3345
• Deterministic dicthash() regardless of key order Deterministic dicthash() regardless of key order #3348 (or WIP: Re-fix test_merge_authors.py #3347)
• lending.py: Add logging to exceptions lending.py: Add logging to exceptions #3350
• Log URL generation failures in lending.py Log URL generation failures in lending.py #3363
• Fix coverstore doctests for WARCRecords on Python 3 Fix coverstore doctests for WARCRecords on Python 3 #3389 -> 3420/refactor/archive thumbnail warc #3422
• Dockerfile.olbase: Add Python 3.8 and pip3 Dockerfile.olbase: Add Python 3.8 and pip3 #3411
• Get a Python 3 powered staging.openlibrary.org Get a Python 3 powered staging.openlibrary.org #3416
  • Get Python3 installed in the Docker images Get Python 3 installed in the docker images #3417
  • Create flag to run Docker image using only Python 3 Create flag to run docker image using only python 3 #3418
  • Removing WARCDisk dead code + tests 3420/refactor/archive thumbnail warc #3422
  • Created dev1.openlibrary.org Create docker-based dev1.openlibrary.org #3419
• How do we modernize pytest_funcarg__xyz() tests? How do we modernize pytest_funcarg__xyz() tests? #3586
• Importing render_template is used for its side effects Importing render_template is used for its side effects #3590
• Fix absolute import and bytes for Python 3 Fix absolute import and bytes for Python 3 #3598
• Upgrade select Python dependencies Upgrade select Python dependencies #3619
• Add 2016 LOC MARC dump description to sources Add 2016 LOC MARC dump description to sources #3620
• Python 3: Stats workaround for lang not in web.ctx Python 3: Stats workaround for lang not in web.ctx #3635
• Use list(dict) instead of dict.keys() Use list(dict) instead of dict.keys() #3637
• Fix doctests of UnitParser for Python 2 and 3 Fix doctests of UnitParser for Python 2 and 3 #3657
• Fix mockcache for Python 3 Fix mockcache for Python 3 #3671
• Update and rename test-py3.sh to test_py3.sh Update and rename test-py3.sh to test_py3.sh #3672
• Enable other doctests?
• Re-enable doctests Re-enable doctests #3683
• Add openlibrary/tests/core/test_lists_model.py Add openlibrary/tests/core/test_lists_model.py #3691
• scripts/test_py3.sh --> test results
  • python3 -m pytest -v openlibrary/catalog/marc/tests/test_get_subjects.py MARC Binary is bytes type, clarify code and fix tests #3584
  • python3 -m pytest -v openlibrary/catalog/marc/tests/test_parse.py MARC Binary is bytes type, clarify code and fix tests #3584
  • python3 -m pytest -v openlibrary/tests/catalog/test_get_ia.py MARC Binary is bytes type, clarify code and fix tests #3584
  • python3 -m pytest -v openlibrary/plugins/openlibrary/tests/test_home.py Python 3: pytest openlibrary/plugins/openlibrary/tests/test_home.py fails #3670
• WIP: MARC Binary is bytes type, clarify code and fix tests MARC Binary is bytes type, clarify code and fix tests #3584 & MarcBinary(data): Should the data parameter be bytes or str? #3390
• get_sorted_editions() is not working on books/OL42679M Python 3: get_sorted_editions() is not working on books/OL42679M #3633
• Python 3: Make document a cached_property Python 3: Make document a cached_property #3728
• Fix-pytests-for-openlibrary/records-again Fix pytests for openlibrary/records again #3708
• Ensure that Infogami is ready for Python 3
  • Travis CI: Test on the current infogami master Travis CI: Test on the current infogami master #3579
  • Test openlibrary in travis for regressions as well Test openlibrary in travis for regressions as well infogami#86
  • Achieve Python 3 test parity with Python 2 Achieve Python 3 test parity with Python 2 infogami#87
  • Make LazyObject iterable for Python 3 Make LazyObject iterable for Python 3 infogami#88
  • Upgrade web.py to match Open Library Upgrade web.py to match Open Library infogami#95
  • Add WikiSource().iter() and WikiSource().len() Add WikiSource().__iter__() and WikiSource().__len__() infogami#96
  • Fix import of keyencode Fix import of keyencode infogami#98
  • Server.py: json.dumps(d, default=str) for datetimes Server.py: json.dumps(d, default=str) for datetimes infogami#99
  • Import traceback from the standard library Import traceback from the standard library infogami#100
  • Requirements: web.py==0.39; python_version < '3.5' else 0.61 Requirements: web.py==0.39; python_version < '3.5' else 0.61 infogami#103
  • Register i18n_loadhook() to initialize web.ctx.lang Register i18n_loadhook() to initialize web.ctx.lang infogami#105
  • A fix for work.get_sorted_editions() on Python 3 A fix for work.get_sorted_edition() on Python 3 infogami#107
  • Py3: Fix dictionary changed size during iteration Py3: Fix dictionary changed size during iteration infogami#108
  • Thing: old style class -> new style for Python 3 Thing: old-style class -> new-style for Python 3 infogami#111
  • Fix Thing.setattr() Fix Thing.setattr() infogami#112
  • Cannot add a list to a range Cannot add a list to a range infogami#116
  • Py3: Fix dictionary changed size during iteration again Py3: Fix dictionary changed size during iteration again infogami#117
• pagecount is an int, not a float pagecount is an int, not a float #3730
• Python3: Sort Publish date unknown Python3: Sort Publish date unknown #3731
• Add 16 pytests to make test-py Add 16 pytests to make test-py #3735
• StringIO wants str, not bytes StringIO wants str, not bytes #3752
• fix MARC leader display bug fix MARC leader display bug #3779
• Python 3 compatibility for four files in /scripts Python 3 compatibility for four files in /scripts #3753
• Fix Python 3: Add Cover Image hangs Fix Python 3: Add Cover Image hangs #3733
• Docker Deployment Strategy for ol-web[1,2] Docker Deployment Strategy for ol-web[1,2] #4017
• Fix deployment on ol-web1/ol-web2 Fix deployment on ol-web1/ol-web2 #3993
• Retire ol-web3 & ol-web4 Retire ol-web3 & ol-web4 #3992
• Saving record with non-Latin characters in title throws Internal Server Error Saving record with non-Latin characters in title throws Internal Server Error #4092
• Move ol-covers to Python3 Move ol-covers to Python3 #4054
  • Add Sentry to Coverstore Add Sentry to Coverstore #4055
• Moving ol-home to python3 node Migrate ol-home to ol-home0 #4060
  • Create ol-home0 Create ol-home0 node #4058
  • Move infobase to ol-home0 Move infobase to Python 3 ol-home0 #4059
• Test, test, test...
• Adding book page has problems with Unicode ÄÖÜäöü or ß Adding book page has problems with Unicode ÄÖÜäöü or ß #3704
• Re-enable the Integration Tests in xfail mode Re-enable the Integration Tests in xfail mode again #3682
• Re-enable dependabot for Python dependencies Reenable dependabot for Python dependencies #3580
• Modernizing our Python compound sorting Modernizing our Python compound sorting #3038
• Update dependencies
• Close Run on Both Python 2 and Python 3 #890

BOLD: Pull requests that are ready for review and merge.
ITALIC: Issues and PRs that are optional (nice-to-have).

[tfmorris]

It seems like Infogami is going to be our biggest hurdle. Do we have a strategy for dealing with that?

[cclauss]

Today Infogami appears to be our biggest hurdle but just a few months ago it was not.

Last year I was able to land many modernizations on Infogami so I have a fair understanding of that codebase. There are no miracles here so it will require effort but given the state of web.py (about to drop support for legacy Python) and OL (six files from pytests being green on Py3), I hope that it will be easier to get reviews on Infogami pull requests than it was at the end of 2019.

[cclauss]

Time to close this one! Thanks to everyone for helping to get us to Python 3!!