M4.1: Two Station Modules, Spark is Public

[patrickwoodhead]

This milestone is complete when Station is running two modules and the data collected by both is available to the audience that the module funders would like to expose it to.

The first module is Spark. The module funders would like the data collected by Spark to be publicly available in a public dashboard. This will help everyone to improve the Retrieval Success Rate on Filecoin which is the ultimate goal of Spark.

The second module is likely to be Saturn. The module funders would like the data to be available to the Saturn team and maybe publicly.

To make a third module, we could bifurcate Spark into two separate modules, partitioned by protocol or by piece/payload.

Running two or more modules on Station will make the Station team think in Meridian platform and framework terms. A bonus goal in this milestone is to have the beginnings of the Meridian Platform and Framework laid out.

Tasks

Preview Give feedback

1. Reduce load on SPs space-meridian/roadmap#52
   💥 Spark +1
   
2. Scale SPARK PG DB beyond 500 GiB of measurements filecoin-station/roadmap#60
   4 of 4
   +0
   
3. Automatic Module Updates space-meridian/roadmap#53
   9 of 9
   +0
   
4. Gas Cost Optimisation space-meridian/roadmap#55
   2 of 2
   +0
   
5. Saturn client module on Station space-meridian/roadmap#56
   2 of 2
   +0
   
6. Public Spark Dashboard MVP space-meridian/roadmap#57
   5 of 7
   💥 Spark +1
   
7. CID sampling v0.2 filecoin-station/spark#40
   +0
   
8. Add healthcheck endpoints to services without HTTP filecoin-station/roadmap#62
   +0
   
9. Reduce RAM size of Spark VM's filecoin-station/roadmap#63
   +0
   

Stretch Tasks

Preview Give feedback

1. Report duration of failed retrievals filecoin-station/spark#44
   +0
   
2. Shorten Spark rounds to 20 minutes filecoin-station/spark#50
   +0
   

ETA: 2024-02-09

[patrickwoodhead]

Review of milestone

• This was a good amount of work to complete
• We are really happy with what we have shipped
• We feel like we completed about 80-90% of the work, and were blocked by some external dependencies from completing things further.