Have you considered using Ceres for multi-fork harvesting

[colbyboles]

There hasn't been a lot of activity on this project lately, but it makes a lot of sense to me, especially when you get to larger farm sizes:

https://github.com/hulatang/ceres-combineharvester

In the current Machinaris (or any other multi-chia-fork) setup, each fork harvester instance separately enumerates all of the plots which can take some time when you have over 100k plots. It also takes a lot of memory to cache that information for harvesting - I have a 128GB 5950X machine to run full nodes on and harvest from 8 DAS storage boxes via SAS HBAs. Running 9 forks plus chia is taking more than the 128GB I have due to the harvester plot caches in RAM, which causes that big block of plots to drop out from my farm on some forks, or even chia, depending on who gets the RAM first. I'm not sure what Ceres does currently, but there is also a lot that could be done to optimize simultaneous lookups on large numbers of drives to minimize lookup times across multiple forks.

Is this something you have looked into already?

[guydavis]

Hi, thanks for the tip. Not something I had been aware of previously. Will keep it in mind. Thanks again.

[colbyboles]

I guess the real question is one of maintenance if the various forks begin to diverge from the chia codebase. From what I have seen they all seem to be very close copies of the original. Ceres is also a close copy of the chia codebase, but is listening on all of the fork harvester ports. If some optimizations could be made to the lookups (e.g. better understanding of which plot paths correspond what types of physical media so that simultaneous lookups can be performed where appropriate, and prioritization of chia (or other more valuable forks) for the lookup queues that need to be serialized on a given disk).