Make releases optional on Record definition

[jpmckinney]

Background

For some publishers, it's only feasible/possible to prepare the compiled release – that is, a release that provides the full information about a contracting process at a given time. This might be the case if, for example, the IT system doesn't implement versioning, events, good identifiers and/or proper controls (e.g. it allows a user to just change a supplier's name instead of requiring the user to register a new supplier in the system). This also might be the case for publishers who collect non-OCDS data from a source and re-publish it as OCDS, since they don't have direct access to the source's system.

Presently, some publishers publish these kinds of "full" releases as individual releases, e.g. each time a significant event occurs in the contracting process.

A problem can occur if new information has different array entries from earlier information, that were intended to replace that earlier information. Due to how the merge routine works for most arrays of objects, the new information wouldn't replace it, but would append to it. Due to the data quality and system limitations described earlier, it might not be possible to remedy this problem on the publisher's side.

Discussion

The semantically appropriate way to publish such releases would be as compiled releases. However, the Record definition presently requires an array of releases.

Proposal

Removed "releases" from "required" in the Record definition.

[jpmckinney]

@yolile I think we might have recommended this to a publisher. Do you remember the CRM issue?

This proposal came up in discussion with @dwasyl.

[jpmckinney]

We might want to add caveats to the easy releases page, to help publishers determine whether they can support that approach, or whether they should just publish compiled releases.

I can't find the relevant sentence after a quick look in the docs, but we should also avoid any confusion about whether "full" releases replace previous releases; we should be clear that they are merged the same as smaller "diff" releases.

[yolile]

I think this is related to #1031, as publishing historical data is another case for using only compileReleases. And if I don't remember wrong the discussion came up while implementing Paraguay's OCDS API with historical data.

[duncandewhurst]

Should the issue title be 'Make releases optional on Record definition' per the proposal?

[jpmckinney]

I think this is related to #1031

Yes! That's what I was looking for. I guess this case is a little different, because it's for recent data, but where the constraints are similar as for historical data.

Make releases optional on Record definition

Indeed, typo!

[martinszy]

We have found that for most of our use cases, only the compiledRelease is relevant, and the releases could be requested separately. It should be possible for an API to answer with only the compiledRelease instead of the full record. The releases could be obtained together or separately from the compiledRelease.

[jpmckinney]

Thank you, @martinszy! In #1084, we discuss adding a bulk format, in which the response is JSON Lines of releases or records. We can similarly allow JSON Lines of compiled releases.

As part of this issue and #1084, we should review our requirements on API responses, in order to balance (1) having a small number of response formats (which is simpler for users) and (2) having simpler response formats.

Simply responding with a compiled release (rather than a record containing only a compiled release) is definitely simpler with respect to (2). We can catalog all the possible response formats, but I think it should also be fine with respect to (1).

[jpmckinney]

@yolile Can you create a follow-up issue to update relevant guidance? For example, I assume it would be relevant to make the approach of only publishing compiled releases known on some of the change history or easy releases pages.

[yolile]

Done in #1412