Metadata in download: aminoAcidInsertions included, serialization to be discussed

[corneliusroemer]

I've just tried the metadata download from the ebola-zaire instance on main. This is an example row. Few points for discussion:

• Why are aminoAcidInsertions (and nuc ones) included at all? We can include them but it's more related to alignments rather than metadata proper. After all we don't include snps, or deletions. Insertions allow the reconstruction of a sequence from alignments, but that's all.
• If aminoAcidInsertions and (nuc ones) null? It seems we use null for not provided, should we use that or rather use something like N/A? null seems to be simply the default in kotlin.
• submittedAt and released at is a unix timestamp, should we rather serialize it as human readable timestamp, something like 2023-01-05Z23:11:56?

[chaoran-chen]

Why are aminoAcidInsertions (and nuc ones) included at all?

They shouldn't be, that's a bug.

It seems we use null for not provided

That's also a bug. In a TSV/CSV, I think that unknown values should just be left empty (please not N/A..) -> GenSpectrum/LAPIS#708

submittedAt and released at is a unix timestamp

LAPIS does not yet support timestamps as a type. We could provide it as 2023-01-05Z23:11:56 but that would then be considered as a string and would not support range filters. We could: (1) leave it as is for now and wait until we add timestamps in LAPIS, or (2) add an additional column, i.e., have submittedAt and submittedAtFormatted?

[corneliusroemer]

Thanks! How do we distinguish an empty list from null/None for things like lists of frame shifts?

I guess having a unix timestamp is ok for now, was just not sure if this was intended.

[theosanderson]

How do we distinguish an empty list from null/None for things like lists of frame shifts?

what's would the former mean? (if the answer is: "preprocessing pipeline configured not to provide frameshifts" then let's just not have the column at all in those cases)

[corneliusroemer]

Maybe frameshifts is not the best example, what about something else like associated biosamples: there might be a difference between not knowing about them and knowing there are none.

[chaoran-chen]

Thanks to GenSpectrum/LAPIS-SILO#354 and GenSpectrum/LAPIS#715, insertions are no longer included in the metadata file.

[corneliusroemer]

The insertions have been fixed, there's a new issue for the timestamp