Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Encoding clash when reading sources #1196

Closed
shimizukawa opened this issue Jan 3, 2015 · 7 comments
Closed

Encoding clash when reading sources #1196

shimizukawa opened this issue Jan 3, 2015 · 7 comments
Labels
closed:invalid type:bug

Comments

@shimizukawa
Copy link
Member

When building Sphinx documentation on the Launchpad buildds for Ubuntu Raring, I'm getting the following error:

#!text
reading sources... �[  6%] �                      
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sphinx/environment.py", line 810, in read_doc
    pub.publish()
  File "/usr/lib/python3/dist-packages/docutils/core.py", line 217, in publish
    self.settings)
  File "/usr/lib/python3/dist-packages/docutils/readers/__init__.py", line 72, in read
    self.parse()
  File "/usr/lib/python3/dist-packages/docutils/readers/__init__.py", line 78, in parse
    self.parser.parse(self.input, document)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/__init__.py", line 172, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 174, in run
    input_source=document['source'])
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2727, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 331, in section
    self.new_subsection(title, lineno, messages)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 399, in new_subsection
    node=section_node, match_titles=True)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 286, in nested_parse
    node=node, match_titles=match_titles)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 199, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2300, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2312, in explicit_construct
    return method(self, expmatch)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2055, in directive
    directive_class, match, type_name, option_presets)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2104, in run_directive
    result = directive_instance.run()
  File "/build/buildd/s3ql-beta-2.1/util/sphinx_pipeinclude.py", line 48, in run
    handle_io_errors=None)
  File "/usr/lib/python3/dist-packages/docutils/io.py", line 246, in __init__
    (self.encoding, self.source.encoding))
UnicodeError: Encoding clash: encoding given is "utf-8" but source is opened with encoding "ANSI_X3.4-1968".

I have no idea why Sphinx or Python would open anything with this encoding, and I cannot reproduce the problem locally either. I have set LANG=en_US.UTF-8 explicitly for this command, but this didn't change anything. The full build output is available on https://launchpadlibrarian.net/142540817/buildlog_ubuntu-raring-i386.s3ql-beta_2.1-1~raring6_FAILEDTOBUILD.txt.gz and also includes the output of a printenv for debugging.

The package is built using Sphinx 1.1.3 and Python 3.3.

Sorry for not being more specific.. but maybe you still have a hunch what might be causing this?
@shimizukawa
Copy link
Member Author

From Takayuki Shimizukawa on 2013-06-16 15:25:40+00:00

Sphinx-1.1.3 did not support Python-3.3.x. Please use Sphinx-1.2b1 for Python-3.3.

@shimizukawa
Copy link
Member Author

From Nikolaus Rath on 2013-10-19 22:46:24+00:00

Could you point me to the commit that fixed this issue?

As long as there are only sphinx 1.2 beta releases, they are not going to make it into Debian unstable (I checked with the maintainer). However, I would very much like to support building under Debian, so maybe there's some way I can monkeypatch around the problem?

@shimizukawa
Copy link
Member Author

From Takayuki Shimizukawa on 2013-10-25 02:02:54+00:00

Nikolaus Rath patches to support python-3.3 is this:

Though, I don't know whether this completely.

# for developers: we need release 1.2 in near future..

@shimizukawa
Copy link
Member Author

From Nikolaus Rath on 2013-10-25 21:01:31+00:00

Thanks for the links! I didn't manage to backport them properly to fix the above issue, but I did find the following workaround (to be placed on top of setup.py):

#!python

# Work around #1196
import locale
import os
os.environ['LC_ALL'] = 'C.UTF-8'
os.environ['LANG'] = 'C.UTF-8'
locale.setlocale(locale.LC_ALL, '')

probably very ugly, but it seems to work :-).

@shimizukawa
Copy link
Member Author

From Nikolaus Rath on 2014-06-28 20:58:48+00:00

I'm having the same problem with Sphinx 1.2.2:

#!text

root@vostro:~/s3ql-2.9+dfsg# ./setup.py build_sphinx -a -E
running build_sphinx
Running html builder...
Running Sphinx v1.2.2
building [html]: targets for 31 source files that are out of date
updating environment: 31 added, 0 changed, 0 removed
Traceback (most recent call last):                                                     
  File "./setup.py", line 316, in <module>
    main()
  File "./setup.py", line 194, in main
    command_options={ 'sdist': { 'formats': ('setup.py', 'bztar') } },
  File "/usr/lib/python3.4/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.4/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "./setup.py", line 101, in run
    app.builder.build_update()
  File "/usr/lib/python3/dist-packages/sphinx/builders/__init__.py", line 214, in build_update
    'out of date' % len(to_build))
  File "/usr/lib/python3/dist-packages/sphinx/builders/__init__.py", line 234, in build
    purple, length):
  File "/usr/lib/python3/dist-packages/sphinx/builders/__init__.py", line 134, in status_iterator
    for item in iterable:
  File "/usr/lib/python3/dist-packages/sphinx/environment.py", line 477, in update_generator
    self.read_doc(docname, app=app)
  File "/usr/lib/python3/dist-packages/sphinx/environment.py", line 624, in read_doc
    pub.publish()
  File "/usr/lib/python3/dist-packages/docutils/core.py", line 217, in publish
    self.settings)
  File "/usr/lib/python3/dist-packages/docutils/readers/__init__.py", line 72, in read
    self.parse()
  File "/usr/lib/python3/dist-packages/docutils/readers/__init__.py", line 78, in parse
    self.parser.parse(self.input, document)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/__init__.py", line 172, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 171, in run
    input_source=document['source'])
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2727, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 328, in section
    self.new_subsection(title, lineno, messages)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 396, in new_subsection
    node=section_node, match_titles=True)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 283, in nested_parse
    node=node, match_titles=match_titles)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/usr/lib/python3/dist-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2300, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2312, in explicit_construct
    return method(self, expmatch)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2055, in directive
    directive_class, match, type_name, option_presets)
  File "/usr/lib/python3/dist-packages/docutils/parsers/rst/states.py", line 2104, in run_directive
    result = directive_instance.run()
  File "/tmp/buildd/s3ql-2.9+dfsg/util/sphinx_pipeinclude.py", line 48, in run
    handle_io_errors=None)
  File "/usr/lib/python3/dist-packages/docutils/io.py", line 246, in __init__
    (self.encoding, self.source.encoding))
UnicodeError: Encoding clash: encoding given is "utf-8" but source is opened with encoding "ANSI_X3.4-1968".
root@vostro:~/s3ql-2.9+dfsg# Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='ANSI_X3.4-1968'>
BrokenPipeError: [Errno 32] Broken pipe

The above workaround still works.

@shimizukawa
Copy link
Member Author

From Takayuki Shimizukawa on 2014-06-29 06:04:23+00:00

I think it is not a Sphinx issue. because:

  File "/tmp/buildd/s3ql-2.9+dfsg/util/sphinx_pipeinclude.py", line 48, in run
    handle_io_errors=None)
  File "/usr/lib/python3/dist-packages/docutils/io.py", line 246, in __init__
    (self.encoding, self.source.encoding))
UnicodeError: Encoding clash: encoding given is "utf-8" but source is opened with encoding "ANSI_X3.4-1968"

sphinx_pipeinclude.py", line 48 is a cause of this problem.

child.stdout file like object has been opened with encoding "ANSI_X3.4-1968", however encoding is specified as "utf-8". It is not adaptable. I think there is several ways to correct it; simpler one is that it use child.stdout.encoding instead of encoding for a io.FileInput parameter.

@shimizukawa
Copy link
Member Author

From Nikolaus Rath on 2014-06-29 21:13:48+00:00

You are right, apologies for the noise.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
closed:invalid type:bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@shimizukawa