"(giant block of JSON) has non-unique elements" validation error should be better presented

[robredpath]

Added 2019-09-10 by @Bjwebb:

In #895 (comment) @duncandewhurst wrote:

I came across another variant of this issue today when validating a record package from Uganda, the path to the error was records:

(giant block of JSON) has non-unique elements.

[robredpath]

@Bjwebb is it fairly straightforward to remove the giant block of JSON and give some sort of meaningful feedback on where the error is so that the user can track it down?

[Bjwebb]

Data Review Tool: Correct reporting of large blocks of JSON

[Bjwebb]

I'm having a look at this to see what we can easily do. @duncandewhurst can you share the data that had this issue? (cove link has expired)

[duncandewhurst]

https://standard.open-contracting.org/review/data/5a270762-383e-4450-ab53-8111a9c68d89

[Bjwebb]

Looking into this, we already overwrite the uniqueItems validation function, to check the uniqueness of IDs instead, where they are available (#155). However, this doesn't work for records, which only have OCIDs.

We can check OCIDs instead, if we know it's a record*. This should also speed up validation of records in CoVE, because checking the OCIDs for uniqueness is quicker than checking the whole record (see #155).

.

* From the validation function, the context we have is a bit limited (the key records doesn't get passed through). Here is the view we have of the schema:

{'description': 'The records for this data package.', 'type': 'array', 'minItems': 1, 'items': {'$ref': '#/definitions/record'}, 'uniqueItems': True}

So, we can check this is about records using the $ref, but we must be aware that this could change.

[Bjwebb]

There's a lib-cove PR for this now: OpenDataServices/lib-cove#35

Here's what it looks like in CoVE [link]:

There's a new message for arrays missing ids/ocids too [link]:

There's also been a change of case from "Non-unique ID Values" to "Non-unique id values", as this seems more consistent with what we're doing elsewhere: [link]:

[duncandewhurst]

Thanks @Bjwebb, this looks good to me!

[Bjwebb]

This is fixed in the live deploy now — https://standard.open-contracting.org/review/?source_url=https://raw.githubusercontent.com/OpenDataServices/lib-cove/master/tests/lib/fixtures/common/records_non_unique.json