EN DASH disappears from PDF bookmarks in Japanese documents

[jfbu]

Consider following index.rst

Welcome to FOO's documentation!
===============================

もしもし–––さようなら
---------------------

何もない

and language = 'ja'. Then make latexpdf includes following console output:

dvipdfmx:warning: No character mapping available.
 CMap name: EUC-UCS2
 input str: <85>

and after enquiry it comes from the PDF bookmarks. Indeed the EN DASH are ok in PDF, but absent from bookmark. Picture:

Sphinx escapes the U+2013 (EN DASH) to \textendash{} in LaTeX, but this is not the cause of the problem. It happens also when using the approach at https://github.com/jfbu/sphinx/tree/fixplatexendash and inserting literal – in the tex file.

Environment info

• OS: Mac
• Python version: 3.5.4
• Sphinx version: 1.6.5
• <Extra tools e.g.: Browser, tex or something else>

[jfbu]

The issue may have to do with this

  \RequirePackage{atbegshi}
  \ifx\ucs\undefined
    \ifnum 42146=\euc"A4A2
      \AtBeginShipoutFirst{\special{pdf:tounicode EUC-UCS2}}
    \else
      \AtBeginShipoutFirst{\special{pdf:tounicode 90ms-RKSJ-UCS2}}
    \fi
  \else
    \AtBeginShipoutFirst{\special{pdf:tounicode UTF8-UCS2}}
  \fi

Is this a problem Japanese LaTeX users know how to solve?

Using \PassOptionsToPackage{pdfencoding=unicode}{hyperref} fixes the EN DASH problem in bookmarks but destroys all Japanese characters there...

[tk0miya]

To resolve this, we need to migrate to uplatex. With platex, we use EUC-JP as internal encoding of PDF bookmarks. But the encoding does not contain EN-DASH. As a workaround, we can replace them by other characters (ex. hyphen in ASCII) before outputting .tex code. But to keep using pLaTeX is not important. So it would be better to migrate.