[Gumbo] Entity Ö is converted to Ö;, leaving the semicolon behind

[dfguo]

I'm not so sure why

val = "Ö"
=> "Ö"
r = Sanitize.fragment(val, Sanitize::Config.merge(Sanitize::Config::BASIC))
=> "Ö;"

Was this intentional? It breaks German characters by adding an extra ';'.

[rgrove]

I can't reproduce this. What version of Ruby and Sanitize are you using?

[dfguo]

@rgrove Sorry that my input got rendered to HTML. Here is the code version.

> val = "Ö"
=> "Ö"
> r = Sanitize.fragment(val, Sanitize::Config.merge(Sanitize::Config::BASIC))
=> "Ö;"

I'm using ruby 2.0.0p481, sanitize (3.0.0) Thank you for looking into this.

[rgrove]

Thanks.

So, it's normal for the entity Ö to be converted into Ö as part of the parsing step, but it shouldn't have a trailing semicolon after the conversion. I'll see if I can figure out what's causing that.

[rgrove]

Yep, looks like this is a Gumbo parser bug. I'll try to get it fixed upstream.

https://github.com/google/gumbo-parser/blob/3a61e9ad963cacfb3246468feab28c5058f621c1/src/char_ref.c#L467-L468

[dfguo]

Awesome! Glad you found the issue!

[dfguo]

@rgrove when will the next release be? Thanks!

[rgrove]

@dfguo Next step is to get a new Nokogumbo gem with the Gumbo fix: rubys/nokogumbo#11

Once that happens, I'll release Sanitize 3.0.1 with an updated Nokogumbo dependency.

[rgrove]

@dfguo Just pushed out the 3.0.1 gem with this fix.

Edit: Correction, 3.0.2 has the fix. Looks like Nokogumbo 1.1.11 silently reverted the change we wanted. Oops!

[dfguo]

Awesome! Thanks!