rosdep 0.16.0 introduced errant carriage returns ('\r') to /usr/local/bin/rosdep

[acarrillo]

Immediately after the release of rosdep 0.16.0, our builds began to fail on execution of rosdep in bash:

root@5f367a5c2cdc:/usr/local/bin# rosdep
/usr/bin/env: ‘python\r’: No such file or directory

On inspection via hexdump, I was able to confirm that there are some carriage returns ('\r', or 0x0d) that have been injected before the line feeds ('\n', or 0x0a), which implies perhaps that a Windows build agent got a hold on rosdep during packaging and injected some Windows-style CR-LF terminators instead of the UNIX-style LF? Regardless of how it happened, this is totally broken on Linux!

Old

acarrillo /usr/local/bin $ xxd rosdep                              
00000000: 2321 2f75 7372 2f62 696e 2f70 7974 686f  #!/usr/bin/pytho
00000010: 6e0a 0a66 726f 6d20 726f 7364 6570 322e  n..from rosdep2.
00000020: 6d61 696e 2069 6d70 6f72 7420 726f 7364  main import rosd
00000030: 6570 5f6d 6169 6e0a 726f 7364 6570 5f6d  ep_main.rosdep_m
00000040: 6169 6e28 290a                           ain().          
acarrillo /usr/local/bin $ python rosdep --version
0.15.2                                            

New

root@5f367a5c2cdc:/usr/local/bin# xxd rosdep
00000000: 2321 2f75 7372 2f62 696e 2f65 6e76 2070  #!/usr/bin/env p
00000010: 7974 686f 6e0d 0a0d 0a66 726f 6d20 726f  ython....from ro
00000020: 7364 6570 322e 6d61 696e 2069 6d70 6f72  sdep2.main impor
00000030: 7420 726f 7364 6570 5f6d 6169 6e0d 0a72  t rosdep_main..r
00000040: 6f73 6465 705f 6d61 696e 2829 0d0a       osdep_main()..
root@5f367a5c2cdc:/usr/local/bin# python rosdep --version
0.16.0

We've worked around this internally by calling python /usr/local/bin/rosdep in our build scripts instead of using rosdep directly.

Side by side

[nuclearsandwich]

Thanks for the report. I'm looking into this now.

[tfoote]

I just tried upgrading and don't see this issue:

tfoote@snowman4:/usr/bin Last: [0] (0s Seconds)
$ xxd rosdep
00000000: 2321 202f 7573 722f 6269 6e2f 7079 7468  #! /usr/bin/pyth
00000010: 6f6e 0a0d 0a66 726f 6d20 726f 7364 6570  on...from rosdep
00000020: 322e 6d61 696e 2069 6d70 6f72 7420 726f  2.main import ro
00000030: 7364 6570 5f6d 6169 6e0d 0a72 6f73 6465  sdep_main..rosde
00000040: 705f 6d61 696e 2829 0d0a                 p_main()..
tfoote@snowman4:/usr/bin Last: [0] (0s Seconds)
$ rosdep --version
0.16.0
tfoote@snowman4:/usr/bin Last: [0] (0s Seconds)
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.3 LTS
Release:	18.04
Codename:	bionic
tfoote@snowman4:/usr/bin Last: [0] (0s Seconds)
$ apt-cache policy python-rosdep
python-rosdep:
  Installed: 0.16.0-1
  Candidate: 0.16.0-1
  Version table:
 *** 0.16.0-1 500
        500 http://packages.ros.org/ros/ubuntu bionic/main amd64 Packages
        500 http://packages.ros.org/ros/ubuntu bionic/main i386 Packages
        500 http://packages.ros.org/ros2/ubuntu bionic/main amd64 Packages
        500 http://packages.ros.org/ros2/ubuntu bionic/main arm64 Packages
        100 /var/lib/dpkg/status

[nuclearsandwich]

I just tried upgrading and don't see this issue:

Did you install via pip or the debian packages? I was able to reproduce it with the pip package on Archlinux.

[acarrillo]

For posterity my build agents are Ubuntu 16.04.6. Indeed looks like we installed our rosdep via PyPI

root@5f367a5c2cdc:/usr/local/bin# pip show rosdep
Name: rosdep
Version: 0.16.0
Summary: rosdep package manager abstraction tool for ROS
Home-page: http://wiki.ros.org/rosdep
Author: Tully Foote, Ken Conley
Author-email: tfoote@osrfoundation.org
License: BSD
Location: /usr/local/lib/python2.7/dist-packages
Requires: PyYAML, rosdistro, catkin-pkg, rospkg
You are using pip version 9.0.2, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

root@5f367a5c2cdc:/usr/local/bin# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.6 LTS
Release:        16.04
Codename:       xenial

Snippet showing pip install within build

  Downloading https://files.pythonhosted.org/packages/2d/06/4bdb45c1a252a236e5267435d252e52f91fdd0526f7f186b71bdbdf3557d/rosdep-0.16.0.tar.gz (84kB)

[tfoote]

I used the debian packages, I added more details above.

[nuclearsandwich]

Okay. It looks like an autocrlf I had thought was set on a specific repository was global and for whatever reason my local clone of rosdep decided to add crlf line endings and those were propagated to the pypi release. Since I cannot re-upload to pypi without bumping the version I will create a 0.16.1 release which fixes the issue.

[nuclearsandwich]

@acarrillo 0.16.1 has been uploaded to pypi. I tested the wheel before uploading and tested the upload once it was done so I'm pretty sure this is resolved.

[acarrillo]

Thanks @nuclearsandwich !! You rock! You're welcome to close this Issue on my account.

(venv) acarrillo ~ $ xxd venv/bin/rosdep
00000000: 2321 2f68 6f6d 652f 6163 6172 7269 6c6c  #!/home/acarrill
00000010: 6f2f 7665 6e76 2f62 696e 2f70 7974 686f  o/venv/bin/pytho
00000020: 6e0a 0a66 726f 6d20 726f 7364 6570 322e  n..from rosdep2.
00000030: 6d61 696e 2069 6d70 6f72 7420 726f 7364  main import rosd
00000040: 6570 5f6d 6169 6e0a 726f 7364 6570 5f6d  ep_main.rosdep_m
00000050: 6169 6e28 290a                           ain().
(venv) acarrillo ~ $ venv/bin/rosdep --version
0.16.1

[ruffsl]

@nuclearsandwich , @tfoote , I'm not sure if this is related, but I'm having issues running init:

FROM ubuntu:bionic

# setup timezone
RUN echo 'Etc/UTC' > /etc/timezone && \
    ln -s /usr/share/zoneinfo/Etc/UTC /etc/localtime && \
    apt-get update && apt-get install -q -y tzdata && rm -rf /var/lib/apt/lists/*

# install packages
RUN apt-get update && apt-get install -q -y \
      dirmngr \
      gnupg2 \
      lsb-release \
    && rm -rf /var/lib/apt/lists/*

# setup ros2 keys
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654

# setup sources.list
RUN echo "deb http://packages.ros.org/ros2/ubuntu `lsb_release -sc` main" > /etc/apt/sources.list.d/ros2-testing.list

# install bootstrap tools
RUN apt-get update && apt-get install --no-install-recommends -y \
    python3-rosdep \
    && rm -rf /var/lib/apt/lists/*

# setup environment
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8

# install setup files
RUN rosdep init

Step 9/9 : RUN rosdep init
 ---> Running in da6098e15d71
Traceback (most recent call last):
  File "/usr/bin/rosdep", line 3, in <module>
    from rosdep2.main import rosdep_main
ImportError: No module named rosdep2.main

[ruffsl]

but I'm having issues running init:

Please see #706

[nuclearsandwich]

@nuclearsandwich , @tfoote , I'm not sure if this is related, but I'm having issues running init:

I am able to reproduce the same issue. I tried the newly built 0.16.1 package from the ros_bootstrap repository directly and it doesn't exhibit the issue so it seems like a similar issue with the same fix: clearing the autocrlf behavior and generating the release from a clean clone.

[nuclearsandwich]

I'm reasonably confident that the 0.16.1 release resolves @ruffsl's problem as well.