Cannot import Chinese articles #459

slipbox · 2023-11-07T15:18:12Z

Cannot import Chinese articles, it displays as garbled characters.
For example: http://www.nopss.gov.cn/n1/2023/0802/c431029-40049194.html

lable:bug

mdroidian · 2023-11-09T20:39:07Z

Initial exploration:

workbench/src/features/article.tsx

Lines 107 to 118 in 63edf81

    
               const enc = charset(r.headers) || "utf-8"; 
        
               const buffer = iconv.encode(r.encoded, "base64"); 
        
               const html = iconv.decode(buffer, enc); 
        
               const headIndex = html.indexOf("<head>") + "<head>".length; 
        
               const base = document.createElement("base"); 
        
               base.href = url; 
        
               base.target = "_blank"; 
        
               const htmlWithBase = `${html.substring(0, headIndex)}${ 
        
                 base.outerHTML 
        
               }${html.substring(headIndex)}`; 
        
               const doc = new DOMParser().parseFromString(htmlWithBase, "text/html"); 
        
               const parsedDoc = new Readability(doc).parse();

not frontend / roam issue: copy/paste works and html (ln 109) shows untranslatable characters
charset(r.headers) is returning utf8
iconv-lite says "Untranslatable characters are set to � or ?. No transliteration is currently supported."

Options for next steps

check what is happening in apiPost, r.encoded
find different character encodings provider

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot import Chinese articles #459

Cannot import Chinese articles #459

slipbox commented Nov 7, 2023

mdroidian commented Nov 9, 2023

Cannot import Chinese articles #459

Cannot import Chinese articles #459

Comments

slipbox commented Nov 7, 2023

mdroidian commented Nov 9, 2023