XMLParseError after fetching categoryscheme from BIS

[cbenz]

Running this code:

import sdmx

bis_client = sdmx.Client("BIS")
bis_client.get("categoryscheme")

I get the following stack trace:

--- SS without structure ---
1 (132176382615648) False

--- <class 'sdmx.message.StructureMessage'> ---
2 (132176121483568) <sdmx.StructureMessage>
  <Header>
    id: 'IDREF0bff5885-7552-417a-af1e-b36dcd3b3b4e'
    prepared: '2024-05-10T05:01:55+00:00'
    receiver: <Agency not_supplied>
    sender: <Agency UNKNOWN>
    source: 
    test: False

--- <class 'sdmx.model.common.AgencyScheme'> ---
AGENCIES (132176121488464) AGENCIES

--- <class 'sdmx.model.v21.DataStructureDefinition'> ---
BIS_CBPOL (132176137674944) BIS_CBPOL
BIS_CBS (132176121490768) BIS_CBS
CPMI_CASHLESS (132176121491824) CPMI_CASHLESS
CPMI_CT1 (132176121492880) CPMI_CT1
CPMI_CT2 (132176121493936) CPMI_CT2
CPMI_DEVICES (132176121494944) CPMI_DEVICES
CPMI_INSTITUTIONS (132176121495328) CPMI_INSTITUTIONS
CPMI_MACRO (132176117663120) CPMI_MACRO
CPMI_PARTICIPANTS (132176117664128) CPMI_PARTICIPANTS
CPMI_SYSTEMS (132176117665232) CPMI_SYSTEMS
BIS_PROP_PRICES (132176117666144) BIS_PROP_PRICES
BIS_CREDIT_GAP (132176117667248) BIS_CREDIT_GAP
BIS_DEBT_SEC2 (132176117668256) BIS_DEBT_SEC2
BIS_DER (132176117669360) BIS_DER
BIS_DSR (132176117670800) BIS_DSR
BIS_EER (132176117672096) BIS_EER
BIS_GLI (132176117673200) BIS_GLI
BIS_LBS_DISS (132176117674256) BIS_LBS_DISS
BIS_LONG_CPI (132176117675264) BIS_LONG_CPI
NA_SEC (132176117676320) NA_SEC
BIS_SELECTED_PP (132176117677808) BIS_SELECTED_PP
BIS_TOTAL_CREDIT (132176121494848) BIS_TOTAL_CREDIT
BIS_XR (132176117744800) BIS_XR
BIS_XTD_DERIV (132176117745904) BIS_XTD_DERIV

--- <class 'sdmx.model.common.Agency'> ---
BIS (132176121490000) BIS

--- <class 'sdmx.model.v21.DataflowDefinition'> ---
WS_CBPOL (132176121489952) WS_CBPOL
WS_CBS_PUB (132176121491248) WS_CBS_PUB
WS_CPMI_CASHLESS (132176121492304) WS_CPMI_CASHLESS
WS_CPMI_CT1 (132176121493360) WS_CPMI_CT1
WS_CPMI_CT2 (132176121494416) WS_CPMI_CT2
WS_CPMI_DEVICES (132176121495424) WS_CPMI_DEVICES
WS_CPMI_INSTITUT (132176117662592) WS_CPMI_INSTITUT
WS_CPMI_MACRO (132176117663600) WS_CPMI_MACRO
WS_CPMI_PARTICIP (132176117664560) WS_CPMI_PARTICIP
WS_CPMI_SYSTEMS (132176117665664) WS_CPMI_SYSTEMS
WS_CPP (132176117666624) WS_CPP
WS_CREDIT_GAP (132176117667728) WS_CREDIT_GAP
WS_DEBT_SEC2_PUB (132176117668736) WS_DEBT_SEC2_PUB
WS_DER_OTC_TOV (132176117669744) WS_DER_OTC_TOV
WS_DPP (132176117670224) WS_DPP
WS_DSR (132176117671280) WS_DSR
WS_EER (132176117672576) WS_EER
WS_GLI (132176117673632) WS_GLI
WS_LBS_D_PUB (132176117674688) WS_LBS_D_PUB
WS_LONG_CPI (132176117675744) WS_LONG_CPI
WS_NA_SEC_DSS (132176117676800) WS_NA_SEC_DSS
WS_OTC_DERIV2 (132176117676704) WS_OTC_DERIV2
WS_SPP (132176117743888) WS_SPP
WS_TC (132176117677472) WS_TC
WS_XRU (132176117745280) WS_XRU
WS_XTD_DERIV (132176117746336) WS_XTD_DERIV

--- <class 'sdmx.model.common.CategoryScheme'> ---
BISWEB_CATSCHEME (132176117749792) BISWEB_CATSCHEME
BIS_DP (132176117921840) BIS_DP

--- Name ---
561 (132176347510528) ('en', "Category 'BIS:BIS_DP(1.0).BANKING.LBS.LBS_PT.LBS_PT_COUNTRY' linking to PublicationTable 'BIS:LBS_A7(1.0)'")


Ignore:
 {132176382706752, 132176121490000}
<str:Source xmlns:str="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/structure" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:mes="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/message" xmlns:com="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/common">
  <Ref package="publicationtable" agencyID="BIS" id="LBS_A7" version="1.0" class="PublicationTable"/>
</str:Source>

Traceback (most recent call last):
  File "/home/cbenz/Dev/vendor/sdmx/sdmx/reader/xml/common.py", line 204, in read_message
    result = func(self, element)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/cbenz/Dev/vendor/sdmx/sdmx/reader/xml/v21.py", line 403, in _ref
    reader.push(QName(elem).localname, reader.reference(elem, cls_hint))
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cbenz/Dev/vendor/sdmx/sdmx/reader/xml/common.py", line 462, in reference
    return self.Reference(self, elem, cls_hint=cls_hint)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cbenz/Dev/vendor/sdmx/sdmx/reader/xml/common.py", line 87, in __init__
    self.maintainable = issubclass(target_cls, common.MaintainableArtefact)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: issubclass() arg 1 must be a class

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cbenz/Dev/dbnomics/dbnomics-fetchers/bis-fetcher/local/bug-repro.py", line 4, in <module>
    bis_client.get("categoryscheme")
  File "/home/cbenz/Dev/vendor/sdmx/sdmx/client.py", line 496, in get
    msg = reader.read_message(response_content, structure=kwargs.get("dsd", None))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cbenz/Dev/vendor/sdmx/sdmx/reader/xml/__init__.py", line 45, in read_message
    import_module(f"sdmx.reader.xml.{version}")
  File "/home/cbenz/Dev/vendor/sdmx/sdmx/reader/xml/common.py", line 221, in read_message
    raise XMLParseError from exc
sdmx.exceptions.XMLParseError: TypeError: issubclass() arg 1 must be a class

I understand that the following XML fragment is not supported, in particular the class="PublicationTable" part:

<str:Source xmlns:str="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/structure" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:mes="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/message" xmlns:com="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/common">
  <Ref package="publicationtable" agencyID="BIS" id="LBS_A7" version="1.0" class="PublicationTable"/>
</str:Source>

This has worked until yesterday. Maybe BIS updated their SDMX API reponse content? Sadly I did not keep a copy of the working response.

Could a maintainer of this library tell me whether this is really a new SDMX feature, not implemented yet?

As of now I don't really get how to add a parser for that, but I'm going to try. (I don't find any mention of PublicationTable in the SDMX 2.1 information model PDF)

[khaeru]

Hi @cbenz, thanks for the issue and clear summary!

Could a maintainer of this library tell me whether this is really a new SDMX feature, not implemented yet? […] (I don't find any mention of PublicationTable in the SDMX 2.1 information model PDF)

Indeed, I don't find it either: not in the SDMX 2.1 standard you linked, nor in:

• the summary of changes in SDMX 3.0.0,
• Any content under the @sdmg-twg GitHub organization: https://github.com/search?q=org%3Asdmx-twg+publication+table&type=code,
• A pre-release announcement for SDMX 3.1 on the SDMX website, (I have just seen this for the first time.) or,
• BIS' own SDMX web service documentation that is linked from within the docs of this package.

So this seems to not be standards-compliant SDMX-ML, idiosyncratic to this data provider. It's possible that the XML returned by this query might pass XML Schema validation (that is, the XML tag is a well-formed <Ref …> with the expected attributes), but we (and thus the code) have no way of understanding what it is supposed to mean.

Some possibilities:

1. This is a draft implementation of something that is/will be proposed for SDMX 3.1.0 or a later version, or
2. It is solely used for BIS' own purposes, internal or external.
3. It is a remnant of some older version, like SDMX 2.0, that has somehow leaked into their usage of SDMX 2.1. (I am not familiar enough with pre-2.1 standards to say for sure.)

In any case, the design goals for sdmx1 do not cover non-standard usages. So I think the preferred approach would be to catch this exception, ignore it, and perhaps log a warning message. This would prevent the invalid reference from blocking your use of the other contents of the CategorySchemes. I could take care of making this change.

Meanwhile, could you perhaps please contact the BIS via the means given on their SDMX documentation website, directing them to this issue, and ask them to please clarify what is going on? If it is case (1) above, sdmx1 will eventually need to support this. If (2) or (3), the provider may not be aware that their web service is returning invalid SDMX-ML.

[khaeru]

Here's a reply from one Glenn Tice, via the “SDMX User Forum” (linked via the header at https://sdmx.org; unfortunately not public/requires registration):

PublicationTable is one of the Fusion Metadata Registry (FMR) 'extended' structural metadata artefacts. These are not official SDMX artefacts, but follow all the same principles. Publication Tables are 'identifiable' so can be referenced by other structures including Category Schemes as in this case.

In FMR, Publication Tables are 'presentational metadata' objects which describe complex data tables for publication on websites and other media. A full description can be found on the FMR Wiki here. There are examples of materialised publication tables on the BIS Data Portal: Locational banking statistics publication table: BIS,LBS_A1,1.0.

There are no plans to add Publication Tables to the standard SDMX information model. However the SDMX Technical Working Group is examining in general how presentational metadata can be supported, and new official artefacts for that purpose are likely to emerge as a result, maybe in SDMX 3.2.0.

For your use case, a solution may be to modify the code to gracefully ignore references to artefacts with unrecognised packages or classes.

The suggestion aligns with what I wrote earlier (“catch this exception, ignore it, and perhaps log a warning message”). I'll open a PR shortly with these changes and ask for your review.