UnicodeDecodeError when presenting an object with non-ASCII characters in title

[jmce]

With collective.contentleadimage 1.3.4 in a Plone 4.1.6 site, visiting an object [a link object in the example below] with non-ASCII characters in its title and a lead image leads to a UnicodeDecodeError generating the img tag:

Traceback (innermost last):
  Module ZPublisher.Publish, line 126, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 46, in call_object
  Module Products.CMFCore.FSPythonScript, line 127, in __call__
  Module Shared.DC.Scripts.Bindings, line 322, in __call__
  Module Shared.DC.Scripts.Bindings, line 359, in _bindAndExec
  Module Products.PythonScripts.PythonScript, line 344, in _exec
  Module script, line 25, in link_redirect_view
   - <FSPythonScript at /ludicum_org/link_redirect_view used for /ludicum_org/ev/rm/13/orgs/apm>
   - Line 25
  Module Shared.DC.Scripts.Bindings, line 322, in __call__
  Module Shared.DC.Scripts.Bindings, line 359, in _bindAndExec
  Module Products.CMFCore.FSPageTemplate, line 237, in _exec
  Module Products.CMFCore.FSPageTemplate, line 177, in pt_render
  Module Products.PageTemplates.PageTemplate, line 79, in pt_render
  Module zope.pagetemplate.pagetemplate, line 113, in pt_render
  Module zope.tal.talinterpreter, line 271, in __call__
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 888, in do_useMacro
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 533, in do_optTag_tal
  Module zope.tal.talinterpreter, line 518, in do_optTag
  Module zope.tal.talinterpreter, line 513, in no_tag
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 954, in do_defineSlot
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 533, in do_optTag_tal
  Module zope.tal.talinterpreter, line 518, in do_optTag
  Module zope.tal.talinterpreter, line 513, in no_tag
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 858, in do_defineMacro
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 954, in do_defineSlot
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 533, in do_optTag_tal
  Module zope.tal.talinterpreter, line 518, in do_optTag
  Module zope.tal.talinterpreter, line 513, in no_tag
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 954, in do_defineSlot
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 533, in do_optTag_tal
  Module zope.tal.talinterpreter, line 518, in do_optTag
  Module zope.tal.talinterpreter, line 513, in no_tag
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 852, in do_condition
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 742, in do_insertStructure_tal
  Module Products.PageTemplates.Expressions, line 218, in evaluateStructure
  Module zope.tales.tales, line 696, in evaluate
   - URL: main_template
   - Line 148, Column 29
   - Expression: <StringExpr u'plone.abovecontentbody'>
   - Names:
      {'container': <PloneSite at /ludicum_org>,
       'context': <ATLink at /ludicum_org/ev/rm/13/orgs/apm>,
       'default': <object object at 0xb76237d8>,
       'here': <ATLink at /ludicum_org/ev/rm/13/orgs/apm>,
       'loop': {},
       'nothing': None,
       'options': {'args': ()},
       'repeat': <Products.PageTemplates.Expressions.SafeMapping object at 0x10549e3c>,
       'request': <HTTPRequest, URL=http://3.ludicum.org/ev/rm/13/orgs/apm/link_redirect_view>,
       'root': <Application at >,
       'template': <FSPageTemplate at /ludicum_org/link_view used for /ludicum_org/ev/rm/13/orgs/apm>,
       'traverse_subpath': [],
       'user': <PloneUser 'jmce'>}
  Module zope.contentprovider.tales, line 80, in __call__
  Module plone.app.viewletmanager.manager, line 154, in render
  Module plone.app.viewletmanager.manager, line 85, in render
  Module collective.contentleadimage.browser.viewlets, line 51, in render
  Module plone.app.layout.viewlets.common, line 48, in render
  Module Products.Five.browser.pagetemplatefile, line 125, in __call__
  Module Products.Five.browser.pagetemplatefile, line 59, in __call__
  Module zope.pagetemplate.pagetemplate, line 113, in pt_render
  Module zope.tal.talinterpreter, line 271, in __call__
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 583, in do_setLocal_tal
  Module zope.tales.tales, line 696, in evaluate
   - URL: /srv/plone/fortuna2/buildout-cache/eggs/collective.contentleadimage-1.3.4-py2.6.egg/collective/contentleadimage/browser/leadimage-body.pt
   - Line 1, Column 0
   - Expression: <PythonExpr ( view.bodyTag(css_class='newsImage'))>
   - Names:
      {'args': (),
       'container': <ATLink at /ludicum_org/ev/rm/13/orgs/apm>,
       'context': <ATLink at /ludicum_org/ev/rm/13/orgs/apm>,
       'default': <object object at 0xb76237d8>,
       'here': <ATLink at /ludicum_org/ev/rm/13/orgs/apm>,
       'loop': {},
       'nothing': None,
       'options': {},
       'repeat': <Products.PageTemplates.Expressions.SafeMapping object at 0x1052a11c>,
       'request': <HTTPRequest, URL=http://3.ludicum.org/ev/rm/13/orgs/apm/link_redirect_view>,
       'root': <Application at >,
       'template': <Products.Five.browser.pagetemplatefile.ViewPageTemplateFile object at 0xceac06c>,
       'traverse_subpath': [],
       'user': <PloneUser 'jmce'>,
       'view': <Products.Five.viewlet.viewlet.SimpleViewletClass from /srv/plone/fortuna2/buildout-cache/eggs/collective.contentleadimage-1.3.4-py2.6.egg/collective/contentleadimage/browser/leadimage-body.pt object at 0xfb996ac>,
       'views': <Products.Five.browser.pagetemplatefile.ViewMapper object at 0x1061652c>}
  Module zope.tales.pythonexpr, line 59, in __call__
   - __traceback_info__: ( view.bodyTag(css_class='newsImage'))
  Module <string>, line 1, in <module>
  Module collective.contentleadimage.browser.viewlets, line 25, in bodyTag
  Module Products.Archetypes.Field, line 2621, in tag
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)

[miohtama]

Looks like this is an issue with Archetypes tag()

https://github.com/collective/collective.contentleadimage/blob/master/src/collective/contentleadimage/browser/viewlets.py#L20

We pass image in as is and do nothing for it. Archetypes should be able to handle it what we throw at it.

[jmce]

Using pdb with a breakpoint in the method tag() of ImageField class in Archetype's Field.py, to print values and title before the the exception

        values = {'src' : url,
                  'alt' : escape(alt, quote=True),
                  'title' : escape(title, quote=True),
                  'height' : height,
                  'width' : width,
                 }

        result = '<img src="%(src)s" alt="%(alt)s" title="%(title)s" '\
                 'height="%(height)s" width="%(width)s"' % values

got

(Pdb) pp title
'Associa\xc3\xa7\xc3\xa3o de Professores de Matem\xc3\xa1tica'
(Pdb) pp values
{'alt': 'Associa\xc3\xa7\xc3\xa3o de Professores de Matem\xc3\xa1tica',
 'height': 43,
 'src': u'http://3.ludicum.org/ev/rm/13/orgs/apm/leadImage_preview',
 'title': 'Associa\xc3\xa7\xc3\xa3o de Professores de Matem\xc3\xa1tica',
 'width': 160}
(Pdb) s
> /srv/plone/fortuna2/buildout-cache/eggs/Products.Archetypes-1.7.14-py2.6.egg/Products/Archetypes/Field.py(2623)tag()
-> 'height="%(height)s" width="%(width)s"' % values
(Pdb) 
UnicodeDecodeError: UnicodeD...ge(128)')
> /srv/plone/fortuna2/buildout-cache/eggs/Products.Archetypes-1.7.14-py2.6.egg/Products/Archetypes/Field.py(2623)tag()
-> 'height="%(height)s" width="%(width)s"' % values

Curiously (?), doing a similar assignment with a similar format applied to that utf-8 title, also in a dict, but OUTSIDE tag(), in the caller, results in no exception...

[miohtama]

You cannot repeat unicode errors in pdb prompt because your terminal messes the encoding for you

[jmce]

miohtama: I'm aware the terminal messes with what I see, tried to see what was in strings, checked e.g. decoding/reencoding with values['title'].decode('utf-8').encode('utf-8') ... but, of course, not surprising if I'm unaware of subtler effects.
[update: ok, I believe I was forgetting something about encoding in interactive access [even though I must see details on it again]

[jmce]

I forgot to mention that I met a similar problem with redturtle.smartlink [now not installed, failed to report it for that product]. For now I'll probably handle the exception directly inside the tag method in Archetypes' Field.py, using decode('utf-8') to adjust the title when this happens...

[miohtama]

I made a similar fix to plone.app.vocabularies recently:

miohtama/plone.app.vocabularies@9864e6b

I suggest we apply this to Archetypes?

[jmce]

Yep, in the service I'm building I ended up just patching Archetypes' Field.py, for now.

In the tag method of ImageField I added the following immediately preceding assignment of the values dictionary (used isinstance just in case "type (…) is str" wouldn't cover all needs [but maybe it does])

    if isinstance (alt, str):
        alt = alt.decode('utf-8', 'replace')
    if isinstance (title, str):
        title = title.decode('utf-8', 'replace')

[miohtama]

Have you created a patch for Archetypes yet?

[jmce]

No problems until now with the Archetypes change I already mentioned above:

--- Products.Archetypes-1.7.14-py2.6.egg/Products/Archetypes/Field.py-ORIGINAL  2012-07-26 12:02:31.000000000 +0100
+++ Products.Archetypes-1.7.14-py2.6.egg/Products/Archetypes/Field.py   2012-09-29 19:11:02.000000000 +0100
@@ -2608,10 +2608,17 @@
         if alt is None:
             alt = instance.Title()
         if title is None:
             title = instance.Title()

+        # Temporary(?) hack by jmce@artenumerica.com to solve
+        # https://github.com/collective/collective.contentleadimage/issues/6
+        if isinstance(alt, str):
+            alt = alt.decode('utf-8', 'replace')
+        if isinstance(title, str):
+            title = title.decode('utf-8', 'replace')
+
         values = {'src' : url,
                   'alt' : escape(alt, quote=True),
                   'title' : escape(title, quote=True),
                   'height' : height,
                   'width' : width,