Additional fields missing from OCDS Validator results

[duncandewhurst]

When loading this data (results here) into the OCDS Validator there two additional fields in the tender section which are not reported:

• procuringEntity_id
• procuring_entity

Since the data at the above link is likely to change I have copied an extract from the tender section below:

"procuringEntity_id": 11,
"documents": [],
"milestones": [],
"amendment": {},
"system": "egpII",
"procuring_entity": {
	"id": 704,
	"procuringEntity_id": 11,
	"identifier": {
		"id": "11",
		"legalName": "DepartmentofWaterSupply&Sewerage"

[timgdavies]

Is this because of our language pattern validation approach?

I.e. _id could be procuringEntity in Indonesian...

We might need to do some rethinking around this, possibly taking @kindly's approach of having auto-generated language-specific extensions

[edugomez]

@duncandewhurst , @timgdavies is right, this is due to pattern properties in the schema. This does need some rethinking, as e.g. procuringEntity_hello won't be caught as an additional field either.

[robredpath]

I've added a non-sprint work issue so that this doesn't get lost

[duncandewhurst]

Flagging that some of the implementers we met with in Georgia were looking at publishing multi-lingual OCDS data using this approach.

[jpmckinney]

Submitting the following to http://standard.open-contracting.org/validator/ warns about whatevername but not whatever_name:

{
    "version": "1.1",
    "uri": "http://www.example.com",
    "publishedDate": "2017-09-19T00:00:00Z",
    "publisher": {
        "name": "Open Data Services Co-operative Limited"
    },
    "releases": [
        {
            "ocid": "ocds-213czf-000-00001",
            "id": "ocds-213czf-000-00001-01-planning",
            "date": "2017-09-19T00:00:00Z",
            "initiationType": "tender",
            "tag": [
                "planning"
            ],
            "tender": {
                "id": "ocds-213czf-000-00001-01-planning",
                "whatever_name": "",
                "whatevername": ""
            }
        }
    ]
}

This is because of the test on this line being insufficient:

cove/cove/lib/common.py

Line 286 in f36ac9e

if LANGUAGE_RE.search(field.split('/')[-1]):

That test checks that the field name is suffixed with what could potentially be a BCP 47 language tag. In this case, _name qualifies. However, the test doesn't check that the root field name (whatever) exists in the schema, but it should.

[jpmckinney]

Also, procuringEntity_id is not covered by patternProperties. Perhaps a more robust implementation would add "additionalProperties": false to all objects in the schema, collect the errors, and then report the additional fields.

[robredpath]

@jpmckinney I've added this to the bottom of the Priority board so that we can attend to it in due course. Does that work for you?

[jpmckinney]

That works. In terms of priority, it can move up as it seems simpler/faster to fix that some others, but I'll leave that to ODS to decide.

[jpmckinney]

I think this duplicates / is related to #402.

[robredpath]

Definitely related, but I don't think it's a duplicate. The link is helpful though, thanks!

[timgdavies]

The handling of patternProperties, this is also important for the new ocds_budgets_and_spend_extension which allows arbitrary key-value pairs within two objects (classifications and measures).

At present, validating the examples reports all the fields in these objects as additional fields, whereas the pattern properties regEx in the schema means they should not be reported as a problem.

I think with this, plus #402, this should push this up the priority list - ideally with us finding a good way to respect use of patternProperties in the schema.

[jpmckinney]

This could be a candidate for the Apr-May 2019 sprint.

[robredpath]

Handle additional fields with _ in them

[robredpath]

Moved to https://github.com/open-contracting/cove-ocds/issues/74