Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update simple.py,fix win10 UnicodeDecodeError #651

Closed
wants to merge 1 commit into from
Closed

Update simple.py,fix win10 UnicodeDecodeError #651

wants to merge 1 commit into from

Conversation

nmweizi
Copy link

@nmweizi nmweizi commented Mar 1, 2021

fix win10 UnicodeDecodeError

    nr = InitNornir(config_file="config.yaml")
  File "d:\python38\lib\site-packages\nornir\init_nornir.py", line 64, in InitNornir
    inventory=load_inventory(config),
  File "d:\python38\lib\site-packages\nornir\init_nornir.py", line 18, in load_inventory
    inv = inventory_plugin(**config.inventory.options).load()
  File "d:\python38\lib\site-packages\nornir\plugins\inventory\simple.py", line 103, in load
    hosts_dict = yml.load(f)
  File "d:\python38\lib\site-packages\ruamel\yaml\main.py", line 343, in load
    return constructor.get_single_data()
  File "d:\python38\lib\site-packages\ruamel\yaml\constructor.py", line 111, in get_single_data
    node = self.composer.get_single_node()
  File "_ruamel_yaml.pyx", line 701, in _ruamel_yaml.CParser.get_single_node
  File "_ruamel_yaml.pyx", line 902, in _ruamel_yaml.CParser._parse_next_event
  File "_ruamel_yaml.pyx", line 911, in _ruamel_yaml.input_handler
UnicodeDecodeError: 'gbk' codec can't decode byte 0xac in position 11744: illegal multibyte sequence

fix win10 UnicodeDecodeError

    nr = InitNornir(config_file="config.yaml")
  File "d:\python38\lib\site-packages\nornir\init_nornir.py", line 64, in InitNornir
    inventory=load_inventory(config),
  File "d:\python38\lib\site-packages\nornir\init_nornir.py", line 18, in load_inventory
    inv = inventory_plugin(**config.inventory.options).load()
  File "d:\python38\lib\site-packages\nornir\plugins\inventory\simple.py", line 103, in load
    hosts_dict = yml.load(f)
  File "d:\python38\lib\site-packages\ruamel\yaml\main.py", line 343, in load
    return constructor.get_single_data()
  File "d:\python38\lib\site-packages\ruamel\yaml\constructor.py", line 111, in get_single_data
    node = self.composer.get_single_node()
  File "_ruamel_yaml.pyx", line 701, in _ruamel_yaml.CParser.get_single_node
  File "_ruamel_yaml.pyx", line 902, in _ruamel_yaml.CParser._parse_next_event
  File "_ruamel_yaml.pyx", line 911, in _ruamel_yaml.input_handler
UnicodeDecodeError: 'gbk' codec can't decode byte 0xac in position 11744: illegal multibyte sequence
@ktbyers
Copy link
Collaborator

ktbyers commented Mar 1, 2021

Hmmm, on the face of it this looks wrong to me. These are text files why would we open them as binary?

I would have thought the fix would have been to specify the file encoding as utf-8?

@dbarrosop
Copy link
Contributor

Agreed, this doesn't seem entirely right. This is most likely an issue with the way you saved your files so it'd be great if you could share a sample triggering the issue so we can see what's happening and why.

@nmweizi
Copy link
Author

nmweizi commented Mar 1, 2021

cannot upload zip and yaml file,something wrong.how upload?
hosts.yaml edit by pycharm,is encoding as utf-8
create new hosts.yaml ,Paste the content below,The error is the same
The error only occurs in win32,linux none

host2华为.cmh:
    hostname: 1.1.1.1
    port: 22
    username: test
    password: "123"
    platform: huawei
    groups:
        - test
    data:
        ping_check:
            ping_cmd: "ping -s 1000 -c 3"
            ping_address:
                - [ 1.1.1.1, "ping网关地址" ]
                - [ 2.2.2.2, "ping本机地址" ]

@nmweizi
Copy link
Author

nmweizi commented Mar 1, 2021

python3 Built-in Functions

open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None)
Open file and return a corresponding file object. If the file cannot be opened, an OSError is raised. See Reading and Writing Files for more examples of how to use this function.

file is a path-like object giving the pathname (absolute or relative to the current working directory) of the file to be opened or an integer file descriptor of the file to be wrapped. (If a file descriptor is given, it is closed when the returned I/O object is closed, unless closefd is set to False.)

mode is an optional string that specifies the mode in which the file is opened. It defaults to 'r' which means open for reading in text mode. Other common values are 'w' for writing (truncating the file if it already exists), 'x' for exclusive creation and 'a' for appending (which on some Unix systems, means that all writes append to the end of the file regardless of the current seek position). In text mode, if encoding is not specified the encoding used is platform dependent: locale.getpreferredencoding(False) is called to get the current locale encoding.  (For reading and writing raw bytes use binary mode and leave encoding unspecified.)
In [13]: f = open("./inventory/hosts.yaml",'r')

In [14]: locale.getpreferredencoding(False)
Out[14]: 'cp936'

In [15]: f.readlines()
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-15-055d0c37aeda> in <module>
----> 1 f.readlines()

UnicodeDecodeError: 'gbk' codec can't decode byte 0xac in position 336: illegal multibyte sequence

In [16]:

@nmweizi
Copy link
Author

nmweizi commented Mar 1, 2021

Or force encoding ='utf8' when open file

@dbarrosop
Copy link
Contributor

Ok, I just read online that in windows you need to specify the encoding or this happens. I opened #654 to:

  1. Try to see if we can reproduce the issue in a test in our windows' CI
  2. Fix it by allowing users to specify the encoding, defaulting to utf-8

@dbarrosop
Copy link
Contributor

Or force encoding ='utf8' when open file

exactly, I will add that to #654. Sorry for taking over your PR but wanted to make sure we could reproduce the error first in the CI to make sure we don't reintroduce it in the future by mistake

@nmweizi
Copy link
Author

nmweizi commented Mar 1, 2021

thank you very much.
good work.

@dbarrosop
Copy link
Contributor

Closing in favor of #654

@dbarrosop dbarrosop closed this Mar 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants