feat(chore): Update to update to silo 0.3.0, lapis to 0.3.7, fix e2e tests

[anna-parker]

resolves #

preview URL: https://fix-e2e.loculus.org/

Summary

Removee type: pango_lineage, also remove partitionBy option as this is only a lineage-feature (why?)

Screenshot

PR Checklist

• All necessary documentation has been adapted.
• The implemented feature is covered by an appropriate test.

[corneliusroemer]

Do we use that field anywhere? It's only in dummy I think? We could also just drop it there, it doesn't serve much purpose.

[fengelniederhammer]

I'm quite sure you can just drop the pango lineage field (or make it a regular field - that's maybe even the better option, because there will be fewer changes).

[anna-parker]

@fengelniederhammer I tried dropping the type first so making it a regular field - but it seems that SILO sees the name as meaning a pango_lineage - so I got the same error. I didn't want to remove the entire field as I think this will mean having to change all e2e tests.

[fengelniederhammer]

No, there is no meaning attached to field names in SILO. The SILO database config should look like:

    - name: pango_lineage
      type: string

(generateIndex: true is optional).
I'm very sure that this works (from a technical perspective).

[fengelniederhammer]

I just read the error message careful enough:
partitionBy 'pangoLineage' must be of type STRING and needs 'generateLineageIndex' set

Simply remove partitionBy in that config. That fixes it.

[anna-parker]

@fengelniederhammer what does partitionBy do actually? - it looks like it just splits up the table - so why does it have to be a lineage now?

[fengelniederhammer]

@fengelniederhammer what does partitionBy do actually? - it looks like it just splits up the table - so why does it have to be a lineage now?

https://lapis.cov-spectrum.org/open/v2/docs/maintainer-docs/references/database-configuration

It also had to be a lineage before. (Before: type: pango_lineage, now: generateLineageIndex: true).
It divides the data into partitions.

This provides optimizations:

• It speeds up the queries, because we can process partitions in parallel.
• We basically sort the data by lineage. Thus similar data is closer together. Thus similar mutations are closer together. That improves compression, because Roaring bitmaps compress better if the they are almost full or almost empty.

[corneliusroemer]

Progress!

[corneliusroemer]

Breaking change in LAPIS?

On main we get this instead, FASTA not JSON

[corneliusroemer]

Ah the problem is that we pass some accept headers and they are used to change the default:

The default (if nothing is passed in accept header) is actually still FASTA

So it's only breaking in the sense that now you return what people ask for

[corneliusroemer]

Nice, now it works!