Merge stable extensions into the standard

[yolile]

Discussion

From #891 (comment):

Adding more extensions to core would be good – extensions have issues as described here. A simple example is that there is no probability of two EU publishers choosing the correct/same combination of extensions for sub-threshold procedures.

We should analyze and consider which current and widely used extensions should be added to the standard itself.

One of these extensions could be Bid statistics and details extension. This information is important because as the extension itself mentions:

Information on bids submitted as part of a contracting process is important for many forms of analysis, including:

• Market analysis for understanding the competitiveness of a given marketplace;
• Red flag analysis for understanding potential corruption risks; and
• Value for money analysis;

This information is available at least in:

• Argentina (example)
• Paraguay for auctions (OCDS example)
• Chile (example)

[jpmckinney]

Also related: #1183

[duncandewhurst]

Since the implementation research is more than a year old now, I used the OCDS Downloads stats to generate a report of extension usage by publisher.

I used the Domain metadata from the Kingfisher Collect spiders to identify the publisher, to avoid counting different endpoints for the same publication, e.g. HondurasPortalBulk, HondurasPortalRecords and HondurasPortalReleases. I also counted the following domains as a single publication:

• All digiwhist domains
• 'Oficina Normativa de Contratación y Adquisiciones del Estado (ONCAE)' and 'Oficina Normativa de Contratación y Adquisiciones del Estado (ONCAE) / Secretaria de Finanzas de Honduras (SEFIN)'
• 'Instituto Nacional de Transparencia, Acceso a la Información y Protección de Datos Personales (INAI)' and 'Instituto Estatal de Transparencia, Acceso a la Información Pública y Protección de Datos Personales\n(INAIP) - Yucatán'

Some extensions are declared a different URLs, so I grouped the results by name.en from extension.json, where available:

The following extensions are used by more than 2 more publishers:

extension_name	publisher_count_deduped
Location	8
Budget Breakdown	7
Lots	6
Bid statistics and details	5
Status Details	4
Member Of	3
Contract suppliers	3
Guarantees	3
Additional Contact Points	3

We already plan to move some of these extensions to the standard in 1.2:

• Merge Lots extension #891
• Add statusDetails field to Tender, Award, Contract #764

There are issues with some of the other extensions:

• Member Of's readme has this disclaimer: We strongly recommend against using memberOf unless a use case supports it.
• Contract suppliers - @jpmckinney I seem to remember you had some concerns about this extension, but I can't find the issue in which you described them
• Guarantees - This is a community extension, we might want to solicit more input before adding it to the standard

Otherwise, Location, Budget Breakdown, Bid statistics and details, and Additional Contact Points seem like good candidates to move to the schema.

Many other extensions are used by 2 publishers, but I'm assuming we wouldn't consider that to be 'widely' used.

Edit: A handful of publishers were not included in this analysis because there were issues scraping their data:

• Tenders Electronic Daily (TED) by TenderBase
• moldova positive initiative
• openopps
• Paraguay Dirección Nacional de Contrataciones Públicas (DNCP)
• Paraguay Ministerio de Hacienda

If I remember correctly, Paraguay is a heavy user of extensions, so we might want to add their extensions to the total above.

[duncandewhurst]

@jpmckinney @yolile based on the above, are you happy for me to start on preparing PRs to merge the following extensions?

• Location
• Budget Breakdown
• Bid statistics and details
• Additional Contact Points

[yolile]

If I remember correctly, Paraguay is a heavy user of extensions, so we might want to add their extensions to the total above.

From the >=3 list, Paraguay will add 1 to Lots, Budget Breakdown and Bid statistics and details (and an additional extension over that one to include items, see open-contracting/ocds-extensions#126)

Many other extensions are used by 2 publishers, but I'm assuming we wouldn't consider that to be 'widely' used.

From the = 2 list, Paraguay will add 1 to Item attributes, Enquiries, Budget and spending classification and Parties scale. We could consider adding these too if the criteria is >= 3.

But happy to start with the ones that are used for more than 3

[jpmckinney]

Contract suppliers - @jpmckinney I seem to remember you had some concerns about this extension, but I can't find the issue in which you described them

open-contracting/ocds-extensions#86 (comment)

4+

• Location has some open issues to review. It has geographically-diverse adoption. It seems like a good candidate.
• Budget Breakdown and Budget and spending classification have overlapping semantics, which is not a good situation for having a standardized approach. Budget Breakdown has more publishers because it was publicized earlier. All that said, it's not clear that a contracting standard is the right place for budget details. I'd leave them both out for now.
• Lots: Merge Lots extension #891
• The Bids extension has many open issues. We need to review whether they are still valid and whether they need to be addressed before inclusion.

<=3

• Additional Contact Points: It's only Mexico (multiple) and Moldova. What are the additional contact types for those publishers?
• Parties scale: No open issues. Very small extension. OK to add.
• Enquiries: It's a larger extension for a common process. I think we'd need more review to merge.
• Item attributes: A fairly new extension that risks being used for concepts that ought to be modelled differently. I think we should wait and monitor its use.

FYI, OCDS Downloads hasn't set the API keys for OpenOpps and Paraguay – so it can't collect their data.

[duncandewhurst]

• Location has some open issues to review. It has geographically-diverse adoption. It seems like a good candidate.

I reviewed the open extensions. open-contracting/ocds-extensions#173 and open-contracting/ocds-extensions#115 seem ready to address at the same time as merging the extension. open-contracting/ocds-extensions#142 is more about the Address object and it looks like consensus wasn't yet achieved, but I don't think it blocks merging the extension. The other issues are unrelated. Are you happy to go ahead with merging the extension on that basis?

[duncandewhurst]

Additional Contact Points: It's only Mexico (multiple) and Moldova. What are the additional contact types for those publishers?

Of the publications that declare the extension, only Mexico SHCP actually publishes parties.additionalContactPoints. The additional contact points are just a list of individual names, so I can't tell the 'type' from the data. In any case, since only one publisher actually uses the extension's fields, I suggest we don't need to merge the extension into the standard.

[jpmckinney]

Are you happy to go ahead with merging the extension on that basis?

Let's make a PR against the Location extension first. That way, we can release a 1.1.6 of that extension to be in sync with the version that is merged to OCDS 1.2.0.

[duncandewhurst]

The Bids extension has many open issues. We need to review whether they are still valid and whether they need to be addressed before inclusion.

I've reviewed and commented on the open issues. Some require further discussion.

[yolile]

Should we consider adding the organization classification extension as well? As there have been requests from different partners on how to disclose gender info in OCDS.

[jpmckinney]

Sure!

[jpmckinney]

Noting that, if we merge bids into OCDS, we probably want awards to refer to bids (which contains tenderers / suppliers), since a principle is to only have one way to model a given concept. (I think it's also more correct that a bid is successful, rather than a specific supplier is awarded.)

[jpmckinney]

I opened a new issue for merging organization classification #1622. Location is covered in #1484.

The remaining candidate is Bids, but like Lots #891, there are some outstanding (and significant) issues to address e.g. open-contracting/ocds-extensions#125, over which it will be difficult to achieve consensus.

Postponing bids (and any future candidates) to 1.3/2.0.