RangeError: Invalid string length

[hugo-nl]

🐛 When adding long reports we get a RangeError from ChromaDb
⚡ A potential fix is to introduce batching to chroma since it seem to be happening for longer reports

Ex: https://group.vattenfall.com/globalassets/corporate/who-we-are/sustainability/vattenfall-annual-and-sustainability-report-2023.pdf

[Greenheart]

The same error arises for another long report: Atlas Copco, 2023

[Greenheart]

This consistently happens for reports where we find more than 20 tables. Then we likely get a too long document to store in chromadb.

Could we split the tables into their own collection? Or could we remove duplicate emissions/economy table data by solving #274 ?

[Greenheart]

This is not caused by chromadb per se, but actually the V8 JS runtime. Reference: nodejs/node#35973

We should try to batch process the report and its tables. Since we can't store the full length of the string in memory, perhaps we have to refactor this job to use temporary files and streams, to allow processing the report chunk by chunk.