use-after-free in tidy-html5 5.1.25

[gaa-cifasis]

Hello,

We found a use-after-free in tidy-html5 (last version: 5.1.25). Find attached a test case to reproduce it here. You can see the ASAN report:

==4395== ERROR: AddressSanitizer: heap-use-after-free on address 0xb5e02210 at pc 0x8095ee0 bp 0xbfffeb68 sp 0xbfffeb5c
READ of size 1 at 0xb5e02210 thread T0
#0 0x8095edf (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x8095edf)
#1 0x8230d41 (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x8230d41)
#2 0x815dee5 (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x815dee5)
#3 0x80ef513 (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x80ef513)
#4 0x80f04cf (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x80f04cf)
#5 0x80f04cf (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x80f04cf)
#6 0x80fd9fe (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x80fd9fe)
#7 0x8106468 (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x8106468)
#8 0x8116713 (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x8116713)
#9 0x8123ae4 (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x8123ae4)
#10 0x80ac845 (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x80ac845)
#11 0x80ad3aa (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x80ad3aa)
#12 0x8052d0b (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x8052d0b)
#13 0xb6851a82 (/lib/i386-linux-gnu/libc-2.19.so+0x19a82)
#14 0x8057785 (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x8057785)
0xb5e02210 is located 0 bytes inside of 3-byte region [0xb5e02210,0xb5e02213)
freed by thread T0 here:
#0 0xb69fc774 (/usr/lib/i386-linux-gnu/libasan.so.0.0.0+0x16774)
#1 0x822efcf (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x822efcf)
previously allocated by thread T0 here:
#0 0xb69fc854 (/usr/lib/i386-linux-gnu/libasan.so.0.0.0+0x16854)
#1 0x81f7aac (/home/vagrant/afl-tests/progs/tidy-html5-5.1.25/tidy+0x81f7aac)
Shadow bytes around the buggy address:
0x36bc03f0: fa fa 00 fa fa fa 06 fa fa fa 05 fa fa fa fd fa
0x36bc0400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x36bc0410: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x36bc0420: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x36bc0430: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fd fa
=>0x36bc0440: fa fa[fd]fa fa fa 03 fa fa fa fd fa fa fa fd fa
0x36bc0450: fa fa 00 03 fa fa 06 fa fa fa 05 fa fa fa fd fa
0x36bc0460: fa fa fd fa fa fa fd fa fa fa 07 fa fa fa 06 fa
0x36bc0470: fa fa 00 fa fa fa 03 fa fa fa 05 fa fa fa 02 fa
0x36bc0480: fa fa 00 fa fa fa 06 fa fa fa 05 fa fa fa 00 04
0x36bc0490: fa fa 03 fa fa fa 02 fa fa fa fd fa fa fa 05 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap righ redzone: fb
Freed Heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
ASan internal: fe

and the gdb backtrace is here:

(gdb) bt
#0 0xb7fdd428 in __kernel_vsyscall ()
#1 0xb6866607 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#2 0xb6869a33 in __GI_abort () at abort.c:89
#3 0xb6a042e4 in ?? () from /usr/lib/i386-linux-gnu/libasan.so.0
#4 0xb69f858a in ?? () from /usr/lib/i386-linux-gnu/libasan.so.0
#5 0xb6a00f4b in ?? () from /usr/lib/i386-linux-gnu/libasan.so.0
#6 0xb69ffd3a in __asan_report_error () from /usr/lib/i386-linux-gnu/libasan.so.0
#7 0xb69f88ff in __asan_report_load1 () from /usr/lib/i386-linux-gnu/libasan.so.0
#8 0x08095ee0 in prvTidytmbstrlen (str=0xb5e02211 "m") at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/tmbstr.c:98
#9 prvTidytmbstrdup (allocator=0x82c1540 <prvTidyg_default_allocator>, str=str@entry=0xb5e02210 "em")

at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/tmbstr.c:18

#10 0x08230d42 in prvTidyInsertedToken (doc=doc@entry=0xb6601d00) at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/istack.c:275
#11 0x0815dee6 in prvTidyGetToken (doc=doc@entry=0xb6601d00, mode=mode@entry=MixedContent)

at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/lexer.c:2276

#12 0x080ef514 in prvTidyParseInline (doc=0xb6601d00, element=0xb5604690, mode=MixedContent)

at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/parser.c:1672

#13 0x080f04d0 in ParseTag (mode=, node=0xb5604690, doc=)

at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/parser.c:775

#14 prvTidyParseInline (doc=0xb6601d00, element=, mode=MixedContent)

at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/parser.c:2202

#15 0x080f04d0 in ParseTag (mode=, node=0xb5604700, doc=)

at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/parser.c:775

#16 prvTidyParseInline (doc=0xb6601d00, element=, mode=MixedContent)

at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/parser.c:2202

#17 0x080fd9ff in ParseTag (mode=IgnoreWhitespace, node=0xb5604770, doc=)

at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/parser.c:775

#18 prvTidyParseRow (doc=0xb6601d00, row=0xb5601590, mode=IgnoreWhitespace) at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/parser.c:2689
#19 0x08106469 in ParseTag (mode=IgnoreWhitespace, node=, doc=)

at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/parser.c:775

#20 prvTidyParseTableTag (doc=0xb6601d00, table=0xb5601520, mode=IgnoreWhitespace)

at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/parser.c:3059

#21 0x08116714 in ParseTag (mode=IgnoreWhitespace, node=0xb5601520, doc=)

at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/parser.c:775

#22 prvTidyParseBody (doc=0xb6601d00, body=0xb56017c0, mode=IgnoreWhitespace) at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/parser.c:4081
#23 0x08123ae5 in prvTidyParseDocument (doc=doc@entry=0xb6601d00) at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/parser.c:4750
#24 0x080ac846 in prvTidyDocParseStream (doc=doc@entry=0xb6601d00, in=in@entry=0xb6003c60)

at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/tidylib.c:1233

#25 0x080ad3ab in tidyDocParseFile (filnam=0xbffff81e "use-after-free.html", doc=0xb6601d00)

---Type to continue, or q to quit---
at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/tidylib.c:896
#26 tidyParseFile (tdoc=tdoc@entry=0xb6601d00, filnam=filnam@entry=0xbffff81e "use-after-free.html")

at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/src/tidylib.c:844

#27 0x08052d0c in main (argc=2, argv=0xbffff6b4) at /home/vagrant/afl-tests/progs/tidy-html5-5.1.25/console/tidy.c:1336

Regards,
Gustavo.

[geoffmcl]

@gaa-cifasis, Gustavo, thank you for the detailed reports, with version, ASAN, gdb, and the sample page... this is great...

But unfortunately I can NOT reproduce the problem, using the use-after-free.html page, in Ubuntu 14.04 linux, nor in Windows 7, both 64-bits...

And that github attachment seems a very messed up page?? Did something happen with the cut/paste/drop???

But I found perhaps the original page here, but still no problem found running either Tidy version 5.1.25, nor later versions, so can do nothing... I will keep trying this... experimenting...

Now, we did have a buffer memory overrun problem - see #319 - where the fix was added to 5.1.26 plus I think... I do not know if this is the same or a different problem...

And it would be appreciated if I could run ASAN in linux. What do I need to download, install? How do I set this up?

Did you build the binary you are using? Or did you download and install it? From where?

For testing, I built a 5.1.25 binary from source...

I did a clone of tidy, and a checkout, but you could also use the zip source... Or you could GitHub Fork tidy, and clone from your fork...

$ git clone git@github.com:htacg/tidy-html5.git tidy-html5-5.1.25
$ cd tidy-html5-5.1.25
$ git checkout 5.1.25
git usual warning about detached HEAD state
$ cd build/cmake
$ ./build-me.sh DEBUG
y
$ ./tidy -v
HTML Tidy for Linux version 5.1.25
$ gbd --args tidy $HOME/downloads/use-after-free.html
GNU gdb ... 7.7.1
...
Reasing symbols from tidy...done.
(gbd) run
...
tidy output -
453 warnings, 15 errors...
...
[Interior 1 ... exited with code 02]
(gdb) q
$

So how do I now get an ASAN report?

And then I tried a test using the original W3C page and just get just two warnings -

> tidy5-5.1.25d -v
HTML Tidy for Windows version 5.1.25
> tidy5-5.1.25d in_314-4.html
line 40 column 28 - Warning: replacing invalid UTF-8 bytes (char. code U+0007)
line 760 column 1 - Warning: <table> lacks "summary" attribute
Info: Doctype given is "-//W3C//DTD XHTML 1.0 Strict//EN"
Info: Document content looks like XHTML 1.0 Strict
2 warnings, 0 errors were found!

And that tidy5-5.1.25d executable is a special Debug build of tidy which will also detect a write/read to freed memory, as well as lots of other memory problems... but it shows nothing, with either file... nor under MSVC debug mode which always detects this... so...

We need to narrow down the problem by reducing the html to a minimum sample, and get a repeatable bug... and test with later versions... get an ASAN report... get to a fix ;=))

Anything you can do to help would be most appreciated... thanks again...

[gaa-cifasis]

@gaa-cifasis, Gustavo, thank you for the detailed reports, with version, ASAN, gdb, and the sample page... this is great...

You are welcome!

But unfortunately I can NOT reproduce the problem, using the use-after-free.html page, in Ubuntu 14.04 linux, nor in Windows 7, both 64-bits...
And that github attachment seems a very messed up page?? Did something happen with the cut/paste/drop???

If you want a quick check of this issue without recompiling with ASAN support, compile a tidy binary (disable the debug flag just in case) and use valgrind to run the test case:

$ valgrind tidy use-after-free.html

I'm using a 32-bit version of Ubuntu 14.04, but this is unlikely related with this issue. Btw, i had to upload the html webpage to a private Gist, otherwise Github won't let me attach it here. It should be fine, despite it looks corrupted (it was discovered by afl)

But I found perhaps the original page here, but still no problem found running either Tidy version 5.1.25, nor later versions, so can do nothing... I will keep trying this... experimenting...

Let me know you can see some invalid memory read, and we can try more complicated stuff later..

[geoffmcl]

@gaa-cifasis thanks, tried valgrind, first on the good, original page - and no problems -

~/projects/html_tidy/tidy-html5-5.1.25/build/cmake$ valgrind tidy -o temp1.html $HOME/downloads/HTML-Test-Suite-Documentation.html
==4694== Memcheck, a memory error detector
==4694== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==4694== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==4694== Command: tidy -o temp1.html /home/geoff/downloads/HTML-Test-Suite-Documentation.html
==4694== 
line 40 column 28 - Warning: replacing invalid UTF-8 bytes (char. code U+0007)
line 760 column 1 - Warning: <table> lacks "summary" attribute
Info: Doctype given is "-//W3C//DTD XHTML 1.0 Strict//EN"
Info: Document content looks like XHTML 1.0 Strict
2 warnings, 0 errors were found!
==4694== 
==4694== HEAP SUMMARY:
==4694==     in use at exit: 0 bytes in 0 blocks
==4694==   total heap usage: 2,911 allocs, 2,911 frees, 278,364 bytes allocated
==4694== 
==4694== All heap blocks were freed -- no leaks are possible
==4694== 
==4694== For counts of detected and suppressed errors, rerun with: -v
==4694== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Then with the MESS downloaded - this is a really broken file -

~/projects/html_tidy/tidy-html5-5.1.25/build/cmake$ valgrind tidy -o temp2.html $HOME/downloads/use-after-free.html
==4700== Memcheck, a memory error detector
==4700== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==4700== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==4700== Command: tidy -o temp2.html /home/geoff/downloads/use-after-free.html
==4700== 
line 1 column 32 - Warning: replacing invalid UTF-8 bytes (char. code U+0004)
line 1 column 1 - Warning: <!DOCTYPE> converting backslash in URI to slash
line 1 column 1 - Warning: <!DOCTYPE> escaping malformed URI reference
line 4 column 12 - Warning: replacing invalid UTF-8 bytes (char. code U+0080)
line 4 column 13 - Warning: replacing invalid UTF-8 bytes (char. code U+00FF)
line 4 column 14 - Warning: replacing invalid UTF-8 bytes (char. code U+00FF)
line 4 column 15 - Warning: replacing invalid UTF-8 bytes (char. code U+00FF)
... many MANY warnings skipped ...
line 40 column 9 - Warning: <ension> attribute ".html" lacks value
line 40 column 9 - Error: <ension> is not recognized!
==4700== Invalid read of size 1
==4700==    at 0x4130F9: prvTidytmbstrdup (in /usr/bin/tidy)
==4700==    by 0x446573: prvTidyInsertedToken (in /usr/bin/tidy)
==4700==    by 0x42C771: prvTidyGetToken (in /usr/bin/tidy)
==4700==    by 0x41E6E2: prvTidyParseInline (in /usr/bin/tidy)
==4700==    by 0x41E8A4: prvTidyParseInline (in /usr/bin/tidy)
==4700==    by 0x41E8A4: prvTidyParseInline (in /usr/bin/tidy)
==4700==    by 0x41FF36: prvTidyParseRow (in /usr/bin/tidy)
==4700==    by 0x420C10: prvTidyParseTableTag (in /usr/bin/tidy)
==4700==    by 0x4225CC: prvTidyParseBody (in /usr/bin/tidy)
==4700==    by 0x423B1F: prvTidyParseDocument (in /usr/bin/tidy)
==4700==    by 0x416619: prvTidyDocParseStream (in /usr/bin/tidy)
==4700==    by 0x416735: tidyParseFile (in /usr/bin/tidy)
==4700==  Address 0x5290580 is 0 bytes inside a block of size 3 free'd
==4700==    at 0x4C2BDEC: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==4700==    by 0x4463C8: prvTidyPopInline (in /usr/bin/tidy)
==4700==    by 0x41F233: prvTidyParseInline (in /usr/bin/tidy)
==4700==    by 0x41E8A4: prvTidyParseInline (in /usr/bin/tidy)
==4700==    by 0x41E8A4: prvTidyParseInline (in /usr/bin/tidy)
==4700==    by 0x41FF36: prvTidyParseRow (in /usr/bin/tidy)
==4700==    by 0x420C10: prvTidyParseTableTag (in /usr/bin/tidy)
==4700==    by 0x4225CC: prvTidyParseBody (in /usr/bin/tidy)
==4700==    by 0x423B1F: prvTidyParseDocument (in /usr/bin/tidy)
==4700==    by 0x416619: prvTidyDocParseStream (in /usr/bin/tidy)
==4700==    by 0x416735: tidyParseFile (in /usr/bin/tidy)
==4700==    by 0x4093D0: main (in /usr/bin/tidy)
==4700== 

And this block is repeated 3 or 4 times...

So this only happens for me with the really messed file, but given that tidy should handle everything, maybe it is still a valid sample? But maybe not??

These appear to be in prvTidytmbstrdup()... just about out of time today, but will re-check #319 ... what was the fix there? Was this in 5.1.25 or not? etc... And will take a look at Tidy's strdup function... And I note Tidy's internal TY_(tmbstrlen) returns only byte count until the first zero, and your sample file does contain quite a few '0' bytes...

Using $ xxd -g 1 file on the bad downloaded html file I can see (linux and WIN32) -

0000020: 54 44 20 48 00 10 4c 20 34 2e 30 31 2f 16 45 4e  TD H..L 4.01/.EN
and
00000b0: 47 62 41 64 79 3e 0d 0a 3c 64 72 69 66 79 20 00  GbAdy>..<drify .
and
00000f0: 4c 45 2e 3c 2f 64 69 b3 3e 0d 00 40 64 69 76 20  LE.</di.>..@div 
and
0000110: 73 54 00 20 04 00 64 22 3e 54 65 73 74 20 66 6f  sT. ..d">Test fo
and
0000140: 68 34 20 6e 20 31 ff ff 33 2e ff 7f 00 00 2f 64  h4 n 1..3...../d
and
0000160: 6f 6c 6c 6f 77 69 6e 67 00 00 80 00 6c 50 2c 20  ollowing....lP, 
0000170: 74 68 65 20 00 04 72 64 65 72 20 61 74 74 72 69  the ..rder attri
and
0000230: 0a 3c 74 72 3e 3c 74 64 00 00 60 10 6c 65 20 44  .<tr><td..`.le D
and
0000640: 77 77 2e 6d 69 63 72 6f 73 00 10 74 2e 63 6f 6d  ww.micros..t.com
and
0000730: 2f 6c 69 3e 0d 00 01 00 00 69 3e 3c 61 20 68 72  /li>.....i><a hr
and maybe many MORE!!!

How is this handled in Tidy? Etc...

And loading in say gedit, warns it is not valid utf-8, and puts some in red...

Maybe the solution would be for tidy to abort a file if it contains 0x00, stating it feels like a binary file... 0x00 is not valid... or something...

And now I can see this same valgrind error using say later tidy version 5.1.32, which means it is not tidy version related either...

As stated will continue tomorrow, or soonest... see what else I can find... interesting, tidying a BINARY file ;=)) why not?

And separately would like to learn how to compile an ASAN enabled version...

[gaa-cifasis]

My thoughts on this issue: tidy should handle any type of input without unsafe memory operations (even broken/binary files) and it's ok to abort in any case to avoid this type of bugs.

Btw, in order to re-compile with ASAN support, you can check this tutorial. Since tidy is using cmake to compile, you should check the cmake manual to add gcc/g++ flags to a build.

Let me know if you need anything else.

[geoffmcl]

The Problem

@gaa-cifasis have been able to narrow it down... and if I try to modify that input file too much, even using a good editor which operates virtually in binary mode, it stops happening...

It happens around line 67 - [NUL] representing a 0 byte - but may be setup before this...

Sample hyperlinks from test suites are <em>not[NUL][NUL][NUL]an style="text-decoration:underline">underlined</span></em>

That <em> is put on the istack, to be propogated... When tidy gets to the </span> it must close the <em>, so pops the istack... Then something strange - it thinks it is dealing with an <a> so pops and frees too much including the <em> but then wants to duplicate that node, which is an <em>, which has been freed... so three bytes are read e, m, plus the [NUL], twice - once getting the length, and again to do the copy == 6 read errors...

But something does not make sense here... why is an <em> mistaken as an <a>???????

It shows up differently in Windows MSVC Debug, but it is there. In this heavy debug mode, freed memory is not actually freed, but filled with a pattern, to check if it is ever later written to... But it is not written to, so no warning, only read after being freed... so no auto-detection... but looking at the node added to the tree it now has a name of that pattern up until the first nul... So you get a strange warning message like - line 67 column 102 - Warning: inserting implicit <e¦e¦e¦e¦e...>...

And I also installed Dr.Memory for Windows (32-bit), and it confirms it, in a report similar to valgrind... so am adding back the bug classification...

But as I say, if you let your editor modify that file too much, it stops happening... So it is quite a unique sequence that triggers it...

In the original file that is line 81 -

Sample hyperlinks from test suites are <em>not <span style="text-decoration:underline">underlined</span></em>

It does not matter how it got munged, just that it did and provides this sample...

The Solution

Now, as you point out, we could just abort file processing if we find a [NUL]. That's an option... and it is agreed Tidy must not have this memory problem...

But in tracing through the code, Tidy handles well all control characters from the file stream. It specifically deals with \n, \r, \t, \f, and simply tosses all others less than 32... just go an read the next character from the stream. It also seems to deal well with invalid utf-8 sequences... makes a lot of noise about them...

So much noise that I had to remember a little used options concerning seeing all warnings and errors, namely --show-errors 600, to see them all...

So it should be able to handle binary files, with any random sequence of characters, if I can find and fix the specific trigger in this case... and convince myself this is closed, and can not happen again... then maybe, just maybe, we do not need to use the abort option...

Will work on it some more... Thanks for an interesting bug ;=))

ASAN Testing

And also thanks for the ASAN tutorial pointer. Just adding the following cmake options, I got it compiled in -

cmake-clean # a script I have to blow away ALL the last build files...
cmake ../.. -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_C_FLAGS=-fsanitize=address -DCMAKE_BUILD_TYPE=Debug -DENABLE_DEBUG_SYMBOLS:BOOL=TRUE
make

And running the resulting Tidy, HTML Tidy for Linux version 5.1.33Exp in this case, on the bad file, I get a similar report -

$ ./tidy --show-errors 600 path/to/bad/file
... skipped ...
line 56 column 21 - Warning: missing </a> before </li>
line 58 column 1 - Warning: <p> isn't allowed in <tr> elements
line 8 column 1 - Info: <tr> previously mentioned
line 58 column 4 - Warning: inserting implicit <span>
line 58 column 4 - Warning: inserting implicit <a>
line 63 column 1 - Warning: plain text isn't allowed in <tr> elements
line 8 column 1 - Info: <tr> previously mentioned
line 63 column 1 - Warning: <p> isn't allowed in <tr> elements
line 8 column 1 - Info: <tr> previously mentioned
line 64 column 1 - Warning: inserting implicit <span>
line 64 column 1 - Warning: inserting implicit <a>
line 67 column 40 - Warning: replacing unexpected span by </span>
line 67 column 95 - Warning: discarding unexpected </span>
=================================================================
==5035== ERROR: AddressSanitizer: heap-use-after-free on address 0x60040000ba30 at pc 0x4201c1 bp 0x7fff8bc3da80 sp 0x7fff8bc3da78
READ of size 1 at 0x60040000ba30 thread T0
    #0 0x4201c0 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x4201c0)
    #1 0x41faf9 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x41faf9)
    #2 0x48c3df (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x48c3df)
    #3 0x458db5 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x458db5)
    #4 0x43e15c (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x43e15c)
    #5 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #6 0x43e11c (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x43e11c)
    #7 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #8 0x43e11c (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x43e11c)
    #9 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #10 0x4407e5 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x4407e5)
    #11 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #12 0x442975 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x442975)
    #13 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #14 0x447662 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x447662)
    #15 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #16 0x44942a (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x44942a)
    #17 0x44a55c (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x44a55c)
    #18 0x425e8f (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x425e8f)
    #19 0x4246e9 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x4246e9)
    #20 0x424344 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x424344)
    #21 0x40e159 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x40e159)
    #22 0x7ff184c67ec4 (/lib/x86_64-linux-gnu/libc-2.19.so+0x21ec4)
    #23 0x409138 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x409138)
0x60040000ba30 is located 0 bytes inside of 3-byte region [0x60040000ba30,0x60040000ba33)
freed by thread T0 here:
    #0 0x7ff18502033a (/usr/lib/x86_64-linux-gnu/libasan.so.0.0.0+0x1533a)
    #1 0x47cc3b (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x47cc3b)
    #2 0x48b6ae (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x48b6ae)
    #3 0x48b701 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x48b701)
    #4 0x48b93b (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x48b93b)
    #5 0x43cbcc (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x43cbcc)
    #6 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #7 0x43e11c (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x43e11c)
    #8 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #9 0x43e11c (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x43e11c)
    #10 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #11 0x4407e5 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x4407e5)
    #12 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #13 0x442975 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x442975)
    #14 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #15 0x447662 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x447662)
    #16 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #17 0x44942a (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x44942a)
    #18 0x44a55c (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x44a55c)
    #19 0x425e8f (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x425e8f)
    #20 0x4246e9 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x4246e9)
    #21 0x424344 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x424344)
    #22 0x40e159 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x40e159)
    #23 0x7ff184c67ec4 (/lib/x86_64-linux-gnu/libc-2.19.so+0x21ec4)
previously allocated by thread T0 here:
    #0 0x7ff18502041a (/usr/lib/x86_64-linux-gnu/libasan.so.0.0.0+0x1541a)
    #1 0x47cb4e (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x47cb4e)
    #2 0x41fb5b (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x41fb5b)
    #3 0x48b37c (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x48b37c)
    #4 0x43aeb8 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x43aeb8)
    #5 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #6 0x43e11c (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x43e11c)
    #7 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #8 0x43e11c (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x43e11c)
    #9 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #10 0x43e11c (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x43e11c)
    #11 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #12 0x4407e5 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x4407e5)
    #13 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #14 0x442975 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x442975)
    #15 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #16 0x447662 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x447662)
    #17 0x436f80 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x436f80)
    #18 0x44942a (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x44942a)
    #19 0x44a55c (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x44a55c)
    #20 0x425e8f (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x425e8f)
    #21 0x4246e9 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x4246e9)
    #22 0x424344 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x424344)
    #23 0x40e159 (/home/geoff/projects/html_tidy/tidy-html5/build/asan/tidy+0x40e159)
    #24 0x7ff184c67ec4 (/lib/x86_64-linux-gnu/libc-2.19.so+0x21ec4)
Shadow bytes around the buggy address:
  0x0c00ffff96f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c00ffff9700: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c00ffff9710: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c00ffff9720: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c00ffff9730: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c00ffff9740: fa fa fd fa fa fa[fd]fa fa fa 03 fa fa fa fd fa
  0x0c00ffff9750: fa fa fd fa fa fa 00 03 fa fa 06 fa fa fa 05 fa
  0x0c00ffff9760: fa fa fd fa fa fa fd fa fa fa fd fa fa fa 07 fa
  0x0c00ffff9770: fa fa 06 fa fa fa 00 00 fa fa 03 fa fa fa 05 fa
  0x0c00ffff9780: fa fa 02 fa fa fa 00 fa fa fa 06 fa fa fa 05 fa
  0x0c00ffff9790: fa fa 00 04 fa fa 03 fa fa fa 02 fa fa fa fd fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap righ redzone:     fb
  Freed Heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==5035== ABORTING

And that report again indicates some of the sequence... an <a> gets on the istack, the <em> is not shown... and just after discarding the </span>... big trouble...

So you can see I am now on its tail ;=)) Thanks...

[gaa-cifasis]

Great work!, it is nice to see that this nasty bug can be isolated.

[gaa-cifasis]

Also, another test case but with a different callstack is available here. In the unlikely case of a different bug, let me know, and i will create another bug report.

[monsieurp]

Hi gents, Gentoo dev chiming in. We filed a report due to this issue in our bug tracking system: https://bugs.gentoo.org/show_bug.cgi?id=571872. Is there an ETA as to when this is meant to be solved? Just asking, if it takes ages, no worries. Cheers.