python3 support or workaround

[jamesbroadshaw]

Are you looking for help?

I have an app I am deploying to heroku and with the latest heroku-20 stack, support for python2 has been removed. I took a quick stab at trying to update play using 2to3 but ran into a bunch of issues (not being a python developer). Perhaps porting to python3 is too much work but is it possible to bundle in the python2 runtime locally for linux as it's done for Windows and use that by default? Open to any ideas and happy to put up a PR that's mostly broken for suggestions.

Play Version (1.5.x / etc)

1.6.0 (python2)

Operating System (Ubuntu 15.10 / MacOS 10.10 / Windows 10)

heroku-20 stack based on Ubuntu 20.04 LTS (without python2)

JDK (Oracle 1.8.0_72, OpenJDK 1.8.x, Azul Zing)

openjdk 16 2021-03-16

Library Dependencies

heroku-20

Expected Behavior

Please describe the expected behavior of the issue, starting from the first action.

1. heroku create
2. deploy using buildpack for play 1
3. application deploys

Actual Behavior

python2 has been removed from heroku-20 stack and running the initial 'play' command gets a syntax error as it is being run with python3 instead.

[tomparle]

Hi @jamesbroadshaw , nice issue you found out !
I'am also using Heroku and after reading https://devcenter.heroku.com/articles/heroku-20-stack it seems that the easiest would be to use the python2 executable instead of python, which could be either by :

• updating the script files in Play to choose the path to the python bin to use
• updating the Heroku Play build pack so that python calls link to python2 with a symlink for example

While the second option may seem easier to implement, I'm not sure yet what would be best.
What do other people think about his ?

[jamesbroadshaw]

Thanks for taking a look @tomparle (and all the other fixes you've contributed to keep play working with modern Java!)

python2 is only present during the build stage (slug compiler) of the app, the actual runtime when the slug is deployed to the dyno does not have it (see yellow highlights here: https://devcenter.heroku.com/articles/stack-packages). I made a new app (replacing the top of the play python script of #!/usr/bin/env python with python2 to get it to build), and on the startup of the dyno:

2021-04-15T01:04:13.320317+00:00 app[web.1]: [heroku-exec] Starting
2021-04-15T01:04:13.367977+00:00 app[web.1]: /usr/bin/env: 'python2': No such file or directory
2021-04-15T01:04:13.441905+00:00 heroku[web.1]: Process exited with status 127
2021-04-15T01:04:13.506011+00:00 heroku[web.1]: State changed from starting to crashed

If you then do a 'heroku run bash' to explore the runtime of heroku-20:

~ $ which python2
~ $ python -V
Python 3.8.5
~ $ ls -la /usr/bin/python*
lrwxrwxrwx 1 root root 7 Apr 15 2020 /usr/bin/python -> python3
lrwxrwxrwx 1 root root 9 Mar 13 2020 /usr/bin/python3 -> python3.8
-rwxr-xr-x 1 root root 5486384 Jan 27 15:41 /usr/bin/python3.8

Here's what I found works with no changes needed in play or the buildpack:

1. Set a pre-buildpack to install packages with apt

heroku buildpacks:add --index 1 heroku-community/apt

Create an 'Aptfile' in the root and add the following:
python2
python2-minimal
python2.7
python2.7-minimal
libpython2-stdlib
libpython2.7-minimal
libpython2.7-stdlib
python-is-python2

2. Add aptfile and commit to your heroku-20 stack-based Play app, deploy!

[tomparle]

Ho, I did not see that python2 was not available on runtime. No wonder you could not find in with Heroku bash.

Congratulations for your Aptfile workaround !
I will try some tests to find out if a migration to Python 3 is faisable easily.

[tomparle]

Hi all,
I spent some time to try to migrate Play scripts to Python 3.
While everything's not perfect, I think I could achieve quite some progress :

• all scripts (play, framework/pym/play/**/*.py and modules/**/*.py) have been migrated to Python 3
• I tested successfully the following commands : (play) run, start, restart, stop, clean, id, test, eclipsify, deps, version, modules, precompile, war

Some things to consider :

• the embedded python distribution in the python folder has not be modified, not sure if it should
• you may want to remove some print "hello" lines in some modules of your project. The syntax causes an error in Python 3, and those lines are just sample code from the module template. They can be removed safely.
• when launching a command, the following warning is displayed : !! Warning: could not load core command file modulesrepo.py due to No module named 'decoder'. I did not find the reason of this error, and since the scripts worked, I don't use the modules command (and not sure anyone does), and I already spent several hours on this, I left it as is. If anyone has an idea how to fix this, you're welcome !

Here's my work in progress : https://github.com/tomparle/play1/tree/python3
I guess it should be better if other people test this as well before I submit a PR to the main repo.
Feedbacks welcome !

[tomparle]

Update : I fixed the modulesrepo error by upgrading the simplejson library to Python 3.
Now, there is no more errors displayed and all commands are available and seem to work fine !

[tomparle]

New update : I fixed some imports (which causes some errors in some pure Python3 environments), and could make this build works in the new cedar stack (heroku-20), so I'm getting quite confident in this upgrade @jamesbroadshaw !

[tomparle]

Update : I did not notice that Python tests failed.
I spent several hours fixing this because of difficult deadlock / Play not properly killed issues in tests.
I'm glad that tests finally pass.
Hope this will help some people and make this upgrade finally available to the master branch !

[sbeigel]

As macOS 12.3 finally removed Python 2 which makes using Play 1.x impossible -- what's the status for the Python 3 migration? :)

[tomparle]

The PR works and is in the process of being merged this week :-)

[sbeigel]

Sounds great, thank you!

[xael-fry]

Yes the python3 should be the last feature to merge before releasing the 1.7.0, it expect to do it soon

[liccoCode]

Hi, When is the python3 support expected to be online

[tomparle]

@liccoCode I encountered some issues when merging with the master codebase, and spent these two last days quite some time to fix new issues, with the help of @xael-fry.

We're closer than ever, and we only have to fix the build process for some reason. Stay tuned !

[liccoCode]

@tomparle great!

[maicaballangan]

Regretted upgrading my macOS to Monterey. Currently using @tomparle's branch on my local

[jameschu]

Monterey 12.3 workaround

https://www.reddit.com/r/macsysadmin/comments/te2c9o/macos_monterey_123_update_is_now_available_python/i14x6fq/?utm_source=reddit&utm_medium=web2x&context=3

[yepelinux]

Thanks for taking a look @tomparle (and all the other fixes you've contributed to keep play working with modern Java!)

python2 is only present during the build stage (slug compiler) of the app, the actual runtime when the slug is deployed to the dyno does not have it (see yellow highlights here: https://devcenter.heroku.com/articles/stack-packages). I made a new app (replacing the top of the play python script of #!/usr/bin/env python with python2 to get it to build), and on the startup of the dyno:

2021-04-15T01:04:13.320317+00:00 app[web.1]: [heroku-exec] Starting 2021-04-15T01:04:13.367977+00:00 app[web.1]: /usr/bin/env: 'python2': No such file or directory 2021-04-15T01:04:13.441905+00:00 heroku[web.1]: Process exited with status 127 2021-04-15T01:04:13.506011+00:00 heroku[web.1]: State changed from starting to crashed

If you then do a 'heroku run bash' to explore the runtime of heroku-20:

~ $ which python2 ~ $ python -V Python 3.8.5 ~ $ ls -la /usr/bin/python* lrwxrwxrwx 1 root root 7 Apr 15 2020 /usr/bin/python -> python3 lrwxrwxrwx 1 root root 9 Mar 13 2020 /usr/bin/python3 -> python3.8 -rwxr-xr-x 1 root root 5486384 Jan 27 15:41 /usr/bin/python3.8

Here's what I found works with no changes needed in play or the buildpack:

1. Set a pre-buildpack to install packages with apt

heroku buildpacks:add --index 1 heroku-community/apt

Create an 'Aptfile' in the root and add the following: python2 python2-minimal python2.7 python2.7-minimal libpython2-stdlib libpython2.7-minimal libpython2.7-stdlib python-is-python2

2. Add aptfile and commit to your heroku-20 stack-based Play app, deploy!

Any idea on how to make it work on heroku-22? I get this error at the final line:
"python-is-python2" has no installation candidate