In HtmlRenderer, escape_html() returns a string with encoding #<Encoding:ASCII-8BIT>

[andreas12345]

Hi,

when trying to override HtmlRenderer#text, I noticed that the encoding of the string after escape_html() is #Encoding:ASCII-8BIT:

  def text(node)
    puts node.string_content.encoding.inspect # prints #<Encoding:UTF-8>
    puts escape_html(node.string_content).encoding.inspect # prints #<Encoding:ASCII-8BIT>
  end

A workaround I used was to apply force_encoding('utf-8') before continuing processing of the string.

• Andreas

[kivikakk]

Thanks for reporting this! That's a good find. We should probably propagate the encoding of rb_text here:

commonmarker/ext/commonmarker/commonmarker.c

Lines 1150 to 1154 in fc1d7c0

	if (houdini_escape_html0(&buf, (const uint8_t *)RSTRING_PTR(rb_text),
	RSTRING_LEN(rb_text), 0)) {
	result = (char *)cmark_strbuf_detach(&buf);
	return rb_str_new2(result);
	}

(and in a few other places around there.)