Python 3 compatibility

[dirk-thomas]

Getting ROS core (up to ros_tutorials) to build:

• python 3 compatibility ros-infrastructure/catkin_pkg#72 0.1.23
• fix handling of special characters when using Python 3 ros-infrastructure/catkin_pkg#80 0.1.24 released as python3-catkin-pkg
• python3-empy package is available now
• python 3 compatibility ros-infrastructure/rosdep#279 0.10.24 released as python3-rosdep
• python 3 compatibility ros-infrastructure/rosdistro#32 0.2.17 released as python3-rosdistro
• python 3 compatibility ros-infrastructure/rosinstall_generator#12 0.1.5 released as python3-rosinstall-generator
• python 3 compatibility ros-infrastructure/rospkg#48 1.0.26 released as python3-rospkg
• python 3 compatibility catkin#544, add option to specify Python version (fix #570) catkin#571, expose PYTHON_EXECUTABLE to environment hook .em templates catkin#645, use PYTHON_EXECUTABLE for _setup_util.py catkin#646
• python 3 compatibility genmsg#35, python 3 compatibility genmsg#37
• python 3 compatibility gencpp#11
• python 3 compatibility genpy#22
• python 3 compatibility common_msgs#18
• python 3 compatibility ros#41
• python 3 compatibility rospack#25, python 3 compatibility rospack#27, use specific python version catkin has decided on rospack#29
• python 3 compatibility ros_tutorials#13

[dirk-thomas]

Now getting the unit tests to pass again...

• python 3 compatibility genpy#26 (incomplete)
• python 3 compatibility ros#45
• Python 3 compatibility ros_comm#358, Python 3 compatibility ros_comm#426, Python 3 compatibility ros_comm#427, Python 3 compatibility ros_comm#429, Python 3 compatibility ros_comm#430 (incomplete)
• ros/rospack#upcoming
• python3-netifaces package is available now
• no python3-paramiko package available, fork for Py3 at https://pypi.python.org/pypi/paramiko-3k, needs to be released upstream @j-rivero is tracking
• python 3 compatibility dynamic_reconfigure#22 (build issues only)

[jack-oquin]

What package versions are required to run this, and on what operating system?

[dirk-thomas]

Currently I am building everything on Saucy from source.

[jack-oquin]

I made the mistake of installing Saucy on one of my systems recently, in hopes of doing Python3 migration for some the many packages I maintain. Working on Precise has been fruitless, so far.

But, there are no ROS Saucy packages available yet (AFAICT), not even ros-infrastructure.

Can you post a .rosinstall or other hints so I can check this code out on Saucy and try to test and convert my own modules? So far, this seemingly simple task has been amazingly difficult. A wiki page for building from source on Saucy would be great. I am surely not the only person needing that information.

[vrabaud]

Hi, I have done that before (but have NOT tested it with Saucy) while
waiting for the final ROS version. Keep the following in your
/etc/apt/sources.list
deb http://packages.ros.org/ros-shadow-fixed/ubuntu raring main
(yes, raring !)
It might totally fail because of dependencies but I'll cross my fingers for
you :)

On Thu, Dec 5, 2013 at 4:41 PM, jack-oquin notifications@github.com wrote:

I made the mistake of installing Saucy on one of my systems recently, in
hopes of doing Python3 migration for some the many packages I maintain.
Working on Precise has been fruitless, so far.

But, there are no ROS Saucy packages available yet (AFAICT), not even
ros-infrastructure.

Can you post a .rosinstall or other hints so I can check this code out on
Saucy and try to test and convert my own modules? So far, this seemingly
simple task has been amazingly difficult. A wiki page for building from
source on Saucy would be great. I am surely not the only person needing
that information.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2427#issuecomment-29908132
.

[dirk-thomas]

@vrabaud You should definitely not use the raring repository for Saucy. Period.

@jack-oquin All Python packages have already been released for Saucy some weeks ago (the version numbers are listed above). Installing them (from the saucy repo) should work fine. All ROS packages need to be compiled from source. Some have the latest patches already committed to the latest branch (if checked above) then it is sufficient to checkout the source repo. For some others (if not yet checked) the latest patches are still in a pull request - in that case you will need to checkout the branch (probably coming from a forked repo) of the pull request.

Currently you should not yet expect anything to really run. E.g. even starting roscore does not yet work well. So it might be too early to work on converting custom packages (beside general Python 3 fixes in your code base).

[jack-oquin]

Thanks, Dirk.

It would save me hours of effort if you have a .rosinstall file I could use.

[dirk-thomas]

Well, I hope it would not take that long to checkout a dozen repos but I have created a gist with the repos: https://gist.github.com/dirk-thomas/7832421
It contains all default branches of the repos (which is either hydro-devel or groovy-devel) except the ones listed above which need my the forked versions of the repos

[jack-oquin]

Thanks!

What takes time is figuring out exactly what repos to get and which branch in each one to use. It does not take very many wrong guesses before the subsequent {debug; try_something_else} cycle uses up an hour or two.

[dirk-thomas]

Update packages to use catkin_install_python() for installing their Python scripts with shebang rewriting. The following repos have a catkin_install_python branch:

• use catkin_install_python() to install Python scripts actionlib#21
• use catkin_install_python() to install Python scripts catkin#596
• use catkin_install_python() to install Python scripts common_msgs#29
• use catkin_install_python() to install Python scripts gencpp#18
• use catkin_install_python() to install Python scripts genlisp#6
• use catkin_install_python() to install Python scripts genpy#25
• use catkin_install_python() to install Python scripts ros#46
• use catkin_install_python() to install Python scripts ros_comm#361
• use catkin_install_python() to install Python scripts ros_tutorials#17

[dirk-thomas]

I merged most of the huge ros_comm PR to the indigo-devel branch. If I find more time I will look into the remaining issues with genpy and some ros_comm tests.

[dirk-thomas]

The "best" way to test Python 3 support on Trusty is:

• create virtual environment and activate it
• rm venv/bin/python and replace it with a dummy shell script which returns an error code 1
• avoid bash to use system python executable (even if the venv is in the beginning of the PATH): alias python=venv/bin/python
  • ensure that when invoking python from a a different folder it fails
• install python packages in venv using pip
• build packages with python3 ./src/catkin/bin/catkin_make_isolated -DPYTHON_VERSION=3 --install

[jack-oquin]

Need to mention catkin_install_python() here, along with detailed "how to test" instructions:

• http://wiki.ros.org/python_2_and_3_compatible_code

[dirk-thomas]

I don't think that wiki page is the right place. It is completely unrelated to catkin and ROS but only describes differences in Python 2 and Python 3 code. And I think it should stay that way (beside linking other pages).

A separate page on how to use Python 3 should be created - but I don't have much spare time to write it in the near future. My previous comment was more of a brain dump when I ran into the alias issue.

[jack-oquin]

It is discouraging that telling people how they should write Python code for ROS is something done in one's "spare time".

[dirk-thomas]

We made a decision several months ago that we won't proceed with working on Python 3 support for Indigo.

I am using my free time to merge already existing pull requests in order to not loose all the work done back then (since they were already not mergable out-of-the-box anymore). Please excuse me for not spending more of my free time on writing additional tutorials and documentation.

[dirk-thomas]

I will close this ticket since there is no specific work on this item. In the meantime a couple of platforms are using Python 3 by default and over time patches have been contributed to various parts of ROS. If further work is necessary please just create tickets / pull requests against the specific ROS package.