Unicode bytearray error in 2018.3.4

[pnickerson-cashstar]

Description of Issue/Question

One of my Salt minions recently upgraded from 2018.3.3 to 2018.3.4, and it started flooding the log file with a single repeated error:

2019-02-28T05:24:49.036345-05:00 mockproc-doc01.semi.aws salt-minion[3572]: [CRITICAL] Unexpected error while connecting to bastion.semi.aws
2019-02-28T05:24:49.036533-05:00 mockproc-doc01.semi.aws salt-minion[3572]: Traceback (most recent call last):
2019-02-28T05:24:49.036636-05:00 mockproc-doc01.semi.aws salt-minion[3572]:   File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1020, in _connect_minion
2019-02-28T05:24:49.036734-05:00 mockproc-doc01.semi.aws salt-minion[3572]:     yield minion.connect_master(failed=failed)
2019-02-28T05:24:49.036834-05:00 mockproc-doc01.semi.aws salt-minion[3572]:   File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
2019-02-28T05:24:49.036931-05:00 mockproc-doc01.semi.aws salt-minion[3572]:     value = future.result()
2019-02-28T05:24:49.037023-05:00 mockproc-doc01.semi.aws salt-minion[3572]:   File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
2019-02-28T05:24:49.037114-05:00 mockproc-doc01.semi.aws salt-minion[3572]:     raise_exc_info(self._exc_info)
2019-02-28T05:24:49.037203-05:00 mockproc-doc01.semi.aws salt-minion[3572]:   File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 876, in run
2019-02-28T05:24:49.037292-05:00 mockproc-doc01.semi.aws salt-minion[3572]:     yielded = self.gen.throw(*exc_info)
2019-02-28T05:24:49.037388-05:00 mockproc-doc01.semi.aws salt-minion[3572]:   File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1217, in connect_master
2019-02-28T05:24:49.037483-05:00 mockproc-doc01.semi.aws salt-minion[3572]:     master, self.pub_channel = yield self.eval_master(self.opts, self.timeout, self.safe, failed)
2019-02-28T05:24:49.037573-05:00 mockproc-doc01.semi.aws salt-minion[3572]:   File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
2019-02-28T05:24:49.037959-05:00 mockproc-doc01.semi.aws salt-minion[3572]:     value = future.result()
2019-02-28T05:24:49.038114-05:00 mockproc-doc01.semi.aws salt-minion[3572]:   File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
2019-02-28T05:24:49.038264-05:00 mockproc-doc01.semi.aws salt-minion[3572]:     raise_exc_info(self._exc_info)
2019-02-28T05:24:49.038408-05:00 mockproc-doc01.semi.aws salt-minion[3572]:   File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 230, in wrapper
2019-02-28T05:24:49.038550-05:00 mockproc-doc01.semi.aws salt-minion[3572]:     yielded = next(result)
2019-02-28T05:24:49.038691-05:00 mockproc-doc01.semi.aws salt-minion[3572]:   File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 694, in eval_master
2019-02-28T05:24:49.038845-05:00 mockproc-doc01.semi.aws salt-minion[3572]:     opts.update(prep_ip_port(opts))
2019-02-28T05:24:49.038988-05:00 mockproc-doc01.semi.aws salt-minion[3572]:   File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 265, in prep_ip_port
2019-02-28T05:24:49.039157-05:00 mockproc-doc01.semi.aws salt-minion[3572]:     host, port = parse_host_port(opts['master'])
2019-02-28T05:24:49.039300-05:00 mockproc-doc01.semi.aws salt-minion[3572]:   File "/usr/lib/python2.7/dist-packages/salt/utils/network.py", line 1956, in parse_host_port
2019-02-28T05:24:49.039451-05:00 mockproc-doc01.semi.aws salt-minion[3572]:     host_ip = ipaddress.ip_address(host).compressed
2019-02-28T05:24:49.039597-05:00 mockproc-doc01.semi.aws salt-minion[3572]:   File "/usr/lib/python2.7/dist-packages/salt/ext/ipaddress.py", line 115, in ip_address
2019-02-28T05:24:49.039739-05:00 mockproc-doc01.semi.aws salt-minion[3572]:     return IPv6Address(address)
2019-02-28T05:24:49.039890-05:00 mockproc-doc01.semi.aws salt-minion[3572]:   File "/usr/lib/python2.7/dist-packages/salt/_compat.py", line 175, in __init__
2019-02-28T05:24:49.040033-05:00 mockproc-doc01.semi.aws salt-minion[3572]:     elif self._is_packed_binary(address):
2019-02-28T05:24:49.043157-05:00 mockproc-doc01.semi.aws salt-minion[3572]:   File "/usr/lib/python2.7/dist-packages/salt/_compat.py", line 194, in _is_packed_binary
2019-02-28T05:24:49.043319-05:00 mockproc-doc01.semi.aws salt-minion[3572]:     packed = bool(int(str(bytearray(data)).encode('hex'), 16))
2019-02-28T05:24:49.043470-05:00 mockproc-doc01.semi.aws salt-minion[3572]: TypeError: unicode argument without an encoding

I believe this is the same or similar problem seen in #51258
I put in some print statements, and I see that the "data" variable mentioned above is a unicode type, containing "bastion.semi.aws". Note that this is 16 characters long.
It looks like this has been fixed in version 2019.2, but now the problem has appeared in 2018.3.

Setup

salt-minion 2018.3.4 (Oxygen)
Ubuntu 16.04.6 LTS
Kernel: 4.4.0-1075-aws
Python: 2.7.12

Relevant line from /etc/salt/minion:
master: bastion.semi.aws

Steps to Reproduce Issue

1. Upgrade Salt minion to 2018.3.4 on Ubuntu 16.04 with Python 2.7
2. Set the master setting to a 16 character string.
3. Run the Salt minion.

Versions Report

Salt Version:
Salt: 2018.3.4

Dependency Versions:
cffi: 1.11.5
cherrypy: Not Installed
dateutil: 2.7.3
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: 0.21.1
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: 2.18
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.12 (default, Nov 12 2018, 14:36:49)
python-gnupg: 0.3.8
PyYAML: 3.13
PyZMQ: 15.2.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: Ubuntu 16.04 xenial
locale: UTF-8
machine: x86_64
release: 4.4.0-1075-aws
system: Linux
version: Ubuntu 16.04 xenial

[pnickerson-cashstar]

A note for the urgency of this issue: To trigger the bug, all I did was run apt-get upgrade. After that, the Salt minion immediately began recording several hundred lines per second in two log files (/var/log/salt/minion and /var/log/syslog), filling up the disk by several GB per day, and also stopped responding to the Salt master.

I believe this bug is new in 2018.3.4, because it started on upgrade, and my boxes that still have 2018.3.3 have not had the bug.

This bug might only affect those who have 16 character master names, but for those who it does hit, it'll take down their servers after a simple apt upgrade.

[twangboy]

Probably related to #51831
There is a fix in for this: #51918

[pnickerson-cashstar]

Hmm, you're right, this could be a duplicate of #51831, with the addition that it's affecting Python 2.7 as well.

[Ch3LL]

duplicate of #51917 you can test the backport PR i added there.

[Ch3LL]

heh i added the wrong duplicate issue. #51827