Is there a way to do "best-effort" XMP metadata parsing? (to overcome invalid XML)

[ribtoks]

There's a file that has a corrupted XMP packet, which can be read by exiftool, but exiv2 says

Error: XMP Toolkit error 201: Error in XMLValidator
Warning: Failed to decode XMP metadata.

I understand that it's broken and it's OK. But maybe there's a way to overcome this?

I'm using exiv2 as a C++ library.

[clanmills]

@ribtoks There are options to delete/print/insert raw XMP packets.

$ exiv2 -pX https://clanmills.com/Stonehenge.jpg | xmllint --pretty 1 -
<?xml version="1.0"?>
<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 4.4.0-Exiv2">
  <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
    <rdf:Description xmlns:xmp="http://ns.adobe.com/xap/1.0/" xmlns:cm2e="http://clanmills.com/exiv2/" xmlns:dc="http://purl.org/dc/elements/1.1/" rdf:about="" xmp:Rating="0" xmp:ModifyDate="2015-07-16T20:25:28+01:00" cm2e:Father="Robin Mills">
      <cm2e:Family>
        <rdf:Bag/>
      </cm2e:Family>
      <dc:description>
        <rdf:Alt>
          <rdf:li xml:lang="x-default">Classic View</rdf:li>
        </rdf:Alt>
      </dc:description>
      <dc:Family>
        <rdf:Bag>
          <rdf:li>Robin</rdf:li>
        </rdf:Bag>
      </dc:Family>
    </rdf:Description>
  </rdf:RDF>
</x:xmpmeta>
<?xpacket end="w"?>

Before you can read that XMP, could you figure out why the validator rejected it? Once you understand, it may be possible to modify the validator code to forgive and accept.

[clanmills]

I'm up early this morning and had an idea:

Fix 1 Works
Add a namespace definition for iptc.

557 rmills@rmillsm1:~/temp $ xmllint --pretty 1 foo.xml 
<?xml version="1.0"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" xmlns:iptc="clanmills:ns:iptc">
  <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
    <rdf:Description xmlns:dc="http://purl.org/dc/elements/1.1/" rdf:about="">
      <dc:title>Redacted...</dc:title>
      <iptc:ObjectName>Redacted...</iptc:ObjectName>
      <dc:description>Redacted...</dc:description>
      <iptc:Caption-Abstract>Redacted...</iptc:Caption-Abstract>
      <dc:subject>Redacted...</dc:subject>
      <iptc:Keywords>Redacted...</iptc:Keywords>
    </rdf:Description>
  </rdf:RDF>
</x:xmpmeta>
558 rmills@rmillsm1:~/temp $ curl -O https://clanmills.com/Stonehenge.jpg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6599k  100 6599k    0     0  3459k      0  0:00:01  0:00:01 --:--:-- 3458k
$ exiv2 -dX Stonehenge.jpg 
$ cat foo.xml | exiv2 -iX- Stonehenge.jpg 
$ exiv2 -px Stonehenge.jpg 
Xmp.dc.title                                 LangAlt     1  lang="x-default" Redacted...
Xmp.dc.description                           LangAlt     1  lang="x-default" Redacted...
Xmp.dc.subject                               XmpBag      1  Redacted...
Xmp.iptc.ObjectName                          XmpText    11  Redacted...
Xmp.iptc.Caption-Abstract                    XmpText    11  Redacted...
Xmp.iptc.Keywords                            XmpText    11  Redacted...
562 rmills@rmillsm1:~/temp $ 

Fix 2 Works
Blank out the iptc: entries in the XML:

$ bbedit "foo.xml"
$ cat foo.xml
<?xml version="1.0"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/">
  <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
    <rdf:Description xmlns:dc="http://purl.org/dc/elements/1.1/" rdf:about="">
      <dc:title>Redacted...</dc:title>
                                                       
      <dc:description>Redacted...</dc:description>
                                              
      <dc:subject>Redacted...</dc:subject>
                                                  
    </rdf:Description>
  </rdf:RDF>
</x:xmpmeta>
$ cat foo.xml | exiv2 -iX- Stonehenge.jpg 
$ exiv2 -px Stonehenge.jpg 
Xmp.dc.title                                 LangAlt     1  lang="x-default" Redacted...
Xmp.dc.description                           LangAlt     1  lang="x-default" Redacted...
Xmp.dc.subject                               XmpBag      1  Redacted...
$ 

Editing xmpPacket in C++ is a couple of lines of C++. Be careful:

1. adding xmlns:iptc="clanmills:ns:iptc" will expand xmpPacket
2. don't use iptc:ns:clanmills. I made that up for testing. There might an existing suitable NS:

$ exiv2 -vV | grep -i iptc
xmlns=Iptc4xmpCore:http://iptc.org/std/Iptc4xmpCore/1.0/xmlns/
xmlns=Iptc4xmpExt:http://iptc.org/std/Iptc4xmpExt/2008-02-29/
xmlns=acdsee:http://ns.acdsee.com/iptc/1.0/
$ 

[ribtoks]

Hi Robin. Thank you for your ideas!
Is there a way to "pre-add" iptc namespace for parsing (e.g. "register" it)? Remember, I'm doing it from C++ with exiv2 as a library. I can do these "push-ups" for this particular file in my command line, but the fix in question should be a generic one, that will be deployed in my software. That's why I framed the question in the beginning as "is it possible to ignore XML parsing errors" as, if it was, it would allow to parse well-known fields and ignore unknown ones.

[ribtoks]

an update regarding your "Fix 2" I cannot edit xmpPacket in C++ as parsing fails before that... Was there a way to extract the whole packet in C++ as a bytes buffer?

[clanmills]

@ribtoks The XML is invalid, so every XML parser (including xmllint) will barf. You can't forgive or trick it, you have to repair the XML code.

readMetadata() reads the raw XML from the file.

            // Read the rest of the segment.
            DataBuf buf(size);
            if (size > 0) {
                readOrThrow(*io_, buf.pData_ + 2, size - 2, kerFailedToReadImageData);
                memcpy(buf.pData_, sizebuf, 2);
            }

At that moment, buf.pData_ has something like this:

1012 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance $ exiv2 -pS https://clanmills.com/Stonehenge.jpg
STRUCTURE OF JPEG FILE: https://clanmills.com/Stonehenge.jpg
 address | marker       |  length | data
       0 | 0xffd8 SOI  
       2 | 0xffe1 APP1  |   15272 | Exif..II*......................
   15276 | 0xffe1 APP1  |    2786 | http://ns.adobe.com/xap/1.0/.<?x
                             ----   --------------------------------------->
   18064 | 0xffed APP13 |      96 | Photoshop 3.0.8BIM.......'.....
   18162 | 0xffe2 APP2  |    4094 | MPF.II*...............0100.....
   22258 | 0xffdb DQT   |     132 
   22392 | 0xffc0 SOF0  |      17 
   22411 | 0xffc4 DHT   |     418 
   22831 | 0xffda SOS  
1013 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance $ 

The little dance in the code with 29 and 31 are there to ignore the 29 bytes of "http://ns.adob......" PLUS the 2 bytes containing the unsigned short segment length (in this case 2786).

Andreas wrote that code about 20 years ago, so I don't know exactly why we have xmpPacket_ AND xmpData_. A little digging with the debugger will tell you.

You'll need to add code between the readOrThrow() and sending the data to Adobe. Something like this:

std::string XMP((char*) buf.pData_+31,size-31);  // or something like that
if ( XMP.find("<iptc:") > 0 ){
    XMP.replace(   "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\">"
               ,   "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\" xmlns:iptc=\"clanmills:ns:iptc\">" );
    memcpy(buf.pData_,(char*)XMP,XMP.size());   // or something like that
}

I don't want to fire up Xcode to work on this. I've hardly written any code for about 3 years. I'm old and rusty. However, I'm confident that you'll tame the beast.

[ribtoks]

Thank you, Robin!

[clanmills]

Thank you, @ribtoks, for bringing up this subject. It's raining in England, and thinking about this puzzle has been fun.

The answer to the question: Is there a way to overcome this? is NO. It is possible to delete the XMP data. It's also possible to extract the XMP data, externally repair the XMP, and replace the XMP data.

[kevinbackhouse]

You can try commenting out this line of code:

exiv2/src/xmp.cpp

Line 685 in bf95347

XMLValidator::check(xmpPacket.data(), len);

It's the source of the error message that you're getting. It's only there as a security mitigation: the Adobe XMP library that we use is very buggy, so the purpose of that line of code is to do a pre-check of the XML to make sure it's valid before we call the Adobe library. If you trust the file that you're running exiv2 on, then you can safely skip the pre-check.

[ribtoks]

So you mean that Adobe XMP library will "eat" my invalid XML, just if the exiv2 will allow it?

[kevinbackhouse]

Not sure, but I think it's worth a try. Adobe XMP definitely isn't very careful about validating the XML, so it might accept it without crashing.

[clanmills]

Gosh, thanks, @kevinbackhouse Kevin. That's valuable insight. I didn't know what you explained.

[ribtoks]

It was a good attempt, but still got an error from XMP toolkit

Error: XMP Toolkit error 201: XML parsing failure
Warning: Failed to decode XMP metadata.

Anyway, thank you for the idea!

[kevinbackhouse]

Yes, unfortunately that error message is from the Adobe code:

exiv2/xmpsdk/src/ExpatAdapter.cpp

Line 143 in bf95347

XMP_StringPtr errMsg = "XML parsing failure";

So I think @clanmills suggestion of trying to edit the file to fix the XML is the only option.