2018.3.4 TypeError: unicode argument without an encoding #51827

gregor-tb · 2019-02-26T13:06:36Z

Description of Issue/Question

Seems similar to other issues (#51298) involving 16 byte long strings, in our case the "master" entry on the minion has exactly 16 bytes and causes an exception.

We didn't address this problem with prior 2018.3 branch.

Steps to Reproduce Issue

Just installed fresh Debian 9.8. minimal, installed salt-minion, run salt-call grains.items (or any other command).

[ERROR   ] An un-handled exception was caught by salt's global exception handler:
TypeError: unicode argument without an encoding
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 410, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 47, in run
    caller = salt.cli.caller.Caller.factory(self.config)
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 79, in factory
    return ZeroMQCaller(opts, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 291, in __init__
    super(ZeroMQCaller, self).__init__(opts)
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 102, in __init__
    self.minion = salt.minion.SMinion(opts)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 806, in __init__
    lambda: self.eval_master(self.opts, failed=True)
  File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 457, in run_sync
    return future_cell[0].result()
  File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 285, in wrapper
    yielded = next(result)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 694, in eval_master
    opts.update(prep_ip_port(opts))
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 265, in prep_ip_port
    host, port = parse_host_port(opts['master'])
  File "/usr/lib/python2.7/dist-packages/salt/utils/network.py", line 1956, in parse_host_port
    host_ip = ipaddress.ip_address(host).compressed
  File "/usr/lib/python2.7/dist-packages/salt/ext/ipaddress.py", line 115, in ip_address
    return IPv6Address(address)
  File "/usr/lib/python2.7/dist-packages/salt/_compat.py", line 175, in __init__
    elif self._is_packed_binary(address):
  File "/usr/lib/python2.7/dist-packages/salt/_compat.py", line 194, in _is_packed_binary
    packed = bool(int(str(bytearray(data)).encode('hex'), 16))
TypeError: unicode argument without an encoding
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 410, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 47, in run
    caller = salt.cli.caller.Caller.factory(self.config)
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 79, in factory
    return ZeroMQCaller(opts, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 291, in __init__
    super(ZeroMQCaller, self).__init__(opts)
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 102, in __init__
    self.minion = salt.minion.SMinion(opts)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 806, in __init__
    lambda: self.eval_master(self.opts, failed=True)
  File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 457, in run_sync
    return future_cell[0].result()
  File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 285, in wrapper
    yielded = next(result)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 694, in eval_master
    opts.update(prep_ip_port(opts))
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 265, in prep_ip_port
    host, port = parse_host_port(opts['master'])
  File "/usr/lib/python2.7/dist-packages/salt/utils/network.py", line 1956, in parse_host_port
    host_ip = ipaddress.ip_address(host).compressed
  File "/usr/lib/python2.7/dist-packages/salt/ext/ipaddress.py", line 115, in ip_address
    return IPv6Address(address)
  File "/usr/lib/python2.7/dist-packages/salt/_compat.py", line 175, in __init__
    elif self._is_packed_binary(address):
  File "/usr/lib/python2.7/dist-packages/salt/_compat.py", line 194, in _is_packed_binary
    packed = bool(int(str(bytearray(data)).encode('hex'), 16))
TypeError: unicode argument without an encoding

Versions Report

2018.3.4+ds-1

The text was updated successfully, but these errors were encountered:

Ch3LL · 2019-02-26T20:05:20Z

can you verify that the PR fixes it on 2018.3.4 as well?

gregor-tb · 2019-02-27T13:49:03Z

Sorry, I don't know what information you need from me.
I installed the latest deb from http://repo.saltstack.com/apt/debian/9/amd64/2018.3
The fix is not there.

dawidmalina · 2019-02-27T20:00:27Z

I have exactly the same issue with pure saltstack 2018.3.4 installation on ubuntu 18.04 with python3 (previously it was working perfectly).

@Ch3LL I can confirmed that #51298 is working and fixing this issue (I manually applied this patch).

dawidmalina · 2019-02-27T20:04:49Z

Fix definitely is not applied to 2018.3 branch: https://github.com/saltstack/salt/blob/2018.3/salt/_compat.py#L192

9numbernine9 · 2019-02-28T12:08:05Z

I can also confirm that I'm seeing the exact same issue (Salt 2018.3.4, CentOS 7.5 with Python 2.7) and manually applying the patch from #51298 resolves the issue.

Ch3LL · 2019-03-01T15:45:44Z

thanks for confirming. i will work on getting it backported right now.

Ch3LL · 2019-03-01T16:03:32Z

I went ahead and backported both #51298 and #51342 since we had to revert some of the changes. Any chance anyone can re-confirm that this PR #51931 indeed fixes it since we had to roll back changes from the initial PR i asked you to test.

9numbernine9 · 2019-03-04T12:01:18Z

@Ch3LL I applied the changes in PR #51931 to a couple of my 2018.3.4 minions and everything seemed to work just fine! 👍

Ch3LL · 2019-03-07T16:21:21Z

thanks for trying that @9numbernine9 I really appreciate that :)

patrick7 · 2019-04-28T23:03:42Z

Will this bug also be fixed in upcoming debian buster?

tlamr · 2019-05-01T17:26:35Z

This broke our debian stretch upgrade to debian buster as well. Minion just prints these errors into log until it fills whole disk.
This patch fixes it.

Thanks!

Ch3LL · 2019-05-07T18:08:12Z

thanks for all the verifications. i'll go ahead and close since its verified.

@patrick7 we do not have debian buster packages up on repo.saltstack.com but you should be able to install via pip when this fix is included in the 2018.3.4 release. Or you can patch the file.

tlamr · 2019-05-07T19:01:37Z

We actually also rely on debian and thanks to Benjamin Drung this was already backported to sid and is available in 2018.3.4+dfsg1-5 and newer

see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=928337

Thanks!

jacksontj · 2019-05-17T03:44:02Z

@Ch3LL It doesn't seem that this fix is available in a 2018.3.x release yet? Presumably it will be in https://docs.saltstack.com/en/latest/topics/releases/2018.3.5.html which is yet unreleased; any EA for release?

Ch3LL · 2019-05-20T14:53:56Z

ping @KChandrashekhar do we have a timeline on a 2018.3.5 release?

floridop · 2019-05-22T08:37:55Z

Same problem here on Ubuntu 18.04 and release 2018.3.4+ds-1 from your repositories.
For 6 years I kept 2015.x because every update sucked more and more. Now that I decided to move to a "previous" the minion fills the disks with warnings (and KILLS A SERVER).

Seriously guys, you have no quality control. I can understand a bug but not an exception handling that allows code to flood disks. I understand why there is no concept of "stable" version.

I am really surprised you made a commercial product out of this. I hope your stable commercial stuff don't do the same with your paying customers....

So can you give me a suggestion for some release that just works or I have to move to ansible? I'd stick to my old 2015 but it has been out of support for quite long time now.

I think this should be reopened.

Tanoti · 2019-05-24T10:56:04Z

Please can we get an update on when 2018.3.5 will be released?

floridop · 2019-05-24T13:25:31Z

FYI for other people struggling with this, I had to switch all the infrastructure to 2017.7 latest, which seems stable enough now, although different version of 2017.7 wreaked havoc within 2015.x and 2017.x minions (different versions), some started filling logs saying that they could not recognize the master. It took few minions update to latest figure out the issue was in the master which I had to update to the latest 2017 as well. It's kind of a mess but looks stable now.

Btw better like this since with 2018.x I had problems with some of the conditions in my states, lots of stuff was not picked up, I guess changes in Jinja parsing. Maybe there is a bug or release note about that as well. Maybe in 5 years from now when 2018 will be ok I'll take a look and report in more detail...

My machines are Ubuntu 12.04, 14.04, 16.04, 18.04 . master runs on a 18.04 now. Other minions are on Centos6/7, versions 2015.x and 2017.x and it all looks stable right now. Which is nice with such variation. You guys are good when you don't behave like python.

KChandrashekhar · 2019-05-30T23:09:43Z

Please can we get an update on when 2018.3.5 will be released?

Will get the approximate 2018.3.5 release dates by Tuesday (4th June)

KChandrashekhar · 2019-06-04T17:03:21Z

@Tanoti @jacksontj Currently we are working on 2019.2.1 which is estimated for July. Neon is estimated to come out in Aug-sept. 2018.3.5 is estimated to be released in Oct-Nov of 2019.

tlamr · 2019-06-05T10:33:58Z

❤️ Debian Buster fixed https://tracker.debian.org/news/1040463/salt-201834dfsg1-6-migrated-to-testing/

Tanoti · 2020-01-21T14:28:42Z

In case anyone comes here looking for answers, the fix is not in 2018.3.5.

Ch3LL added the info-needed waiting for more info label Feb 26, 2019

Ch3LL added this to the Blocked milestone Feb 26, 2019

Ch3LL added Bug broken, incorrect, or confusing behavior severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around P3 Priority 3 team-core and removed info-needed waiting for more info labels Mar 1, 2019

Ch3LL added the fixed-pls-verify fix is linked, bug author to confirm fix label Mar 1, 2019

This was referenced Mar 1, 2019

Backport both #51298 and #51342 #51931

Merged

Unicode bytearray error in 2018.3.4 #51917

Closed

Ch3LL closed this as completed May 7, 2019

csanate mentioned this issue Aug 14, 2019

Hybrid hunter installation failure Security-Onion-Solutions/securityonion#63

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2018.3.4 TypeError: unicode argument without an encoding #51827

2018.3.4 TypeError: unicode argument without an encoding #51827

gregor-tb commented Feb 26, 2019

Ch3LL commented Feb 26, 2019

gregor-tb commented Feb 27, 2019 •

edited

Loading

dawidmalina commented Feb 27, 2019 •

edited

Loading

dawidmalina commented Feb 27, 2019

9numbernine9 commented Feb 28, 2019

Ch3LL commented Mar 1, 2019

Ch3LL commented Mar 1, 2019

9numbernine9 commented Mar 4, 2019

Ch3LL commented Mar 7, 2019

patrick7 commented Apr 28, 2019

tlamr commented May 1, 2019

Ch3LL commented May 7, 2019

tlamr commented May 7, 2019

jacksontj commented May 17, 2019

Ch3LL commented May 20, 2019

floridop commented May 22, 2019

Tanoti commented May 24, 2019

floridop commented May 24, 2019

KChandrashekhar commented May 30, 2019

KChandrashekhar commented Jun 4, 2019

tlamr commented Jun 5, 2019

Tanoti commented Jan 21, 2020

2018.3.4 TypeError: unicode argument without an encoding #51827

2018.3.4 TypeError: unicode argument without an encoding #51827

Comments

gregor-tb commented Feb 26, 2019

Description of Issue/Question

Steps to Reproduce Issue

Versions Report

Ch3LL commented Feb 26, 2019

gregor-tb commented Feb 27, 2019 • edited Loading

dawidmalina commented Feb 27, 2019 • edited Loading

dawidmalina commented Feb 27, 2019

9numbernine9 commented Feb 28, 2019

Ch3LL commented Mar 1, 2019

Ch3LL commented Mar 1, 2019

9numbernine9 commented Mar 4, 2019

Ch3LL commented Mar 7, 2019

patrick7 commented Apr 28, 2019

tlamr commented May 1, 2019

Ch3LL commented May 7, 2019

tlamr commented May 7, 2019

jacksontj commented May 17, 2019

Ch3LL commented May 20, 2019

floridop commented May 22, 2019

Tanoti commented May 24, 2019

floridop commented May 24, 2019

KChandrashekhar commented May 30, 2019

KChandrashekhar commented Jun 4, 2019

tlamr commented Jun 5, 2019

Tanoti commented Jan 21, 2020

gregor-tb commented Feb 27, 2019 •

edited

Loading

dawidmalina commented Feb 27, 2019 •

edited

Loading