Support not counting final transfer phase against plot processes

[doma2345]

When using madmax plotter on a windows machine as soon as it starts the transfer to the final location the next plot starts, however, when using madmax on plotman it seems to wait for the transfer to the final destination to complete before starting the next plot, is there a way to set the amount of plots or inifite loop as per the madmax wiki so that this happen and you dont need to wait for the transfer to finish

Describe the bug
It appears plotmans integration with MadMax allows for 1 single plot at a time where as in MadMax standalone you can set the -n tag which can be set for infinite or an amount and this allows for cross over when copying to the final dest drive. In my testing with 16 cores on a windows 10 machine I am able to get 40 minute plots as it doesnt include the copy to dest, however on my linux machine using plotman with 24 cores I am only able to achieve 50 minute plots as the next plot doesnt start until the copy is finished.

To Reproduce
Run plotman using madmax and see when the next plot starts after the copy starts or after the copy finishes

Expected behavior
The next plot should start as soon as the copy to final dest starts

[doma2345]

i believe plotman just runs madmax at -n 1 over and over where as it should be set to -n -1 which is infinite plots within MadMax

-n, --count arg Number of plots to create (default = 1, -1 = infinite)

[altendky]

Sounds like the plotting itself is fine, not slow. But, you don't want plot processes in the final transfer stage to count against the configured limits. That was actually discussed even before madMAx.

Also, you commented about being limited to one total job at a time? I ran it with more than one at a time so I will need more info to debug that.

[doma2345]

Yes, so when you run vanilla MadMax in windows, when plot1 hits stage 4 and starts the transfer to the final dest plot2 starts and the transfer continues in the background, (this is achieved by setting the -n switch in madmax to anything over 1 (-1 for infinite)) however the plotman integration waits for the final transfer of plot 1 to complete before the process starts again for plot 2.
I hope this is clear.
The plotting isnt actually slow but this delay in waiting for the transfer to complete is the difference between 32 plots a day and 41 so nearly 25% slower for me.

[doma2345]

the functionality and usage of -n switch is discussed here madMAx43v3r/chia-plotter#662

[altendky]

Yeah, doesn't the Chia Network plotter have -n as well? That seems unlikely to be a thing that makes a lot of sense to use with plotman what with that being a scheduling feature outside of plotman's control. But yeah, even with the Chia Network plotter there has been talk about not counting the the final transfer phase against the plot process limits. No promises when but I would generally expect this to get implemented, maybe "soon".

[mk01]

@doma2345

I had the same expectation - thus adapted phases to recognise p4 + p5 (which starts as soon as p4 is done). mk01@bd46a56

then we make use of stagger also for madmax, and by setting it to those values:

stagger_phase_major: 5
stagger_phase_minor: 0
stagger_phase_limit: 1

and increasing working dir max jobs to 2, we make plotman spawn new plotter process once the copying stage starts (but otherwise computation/sorting/io stopped already).

[doma2345]

@mk01 do you have any idea when this will be merged into main?

[mk01]

@doma2345

oh, perhaps my mistake, I should have been clear in the msg. the snippet linked from my fork was meant more like - go grab it, apply to job.py, adapt config and report back (kind of do it yourself fix for time being) - so that @altendky doesn't need to waste time with further investigation - as it indeed falls back under the topic

to implement: exclude jobs at configured stages from being counted into job# limits

(but it definitelly starts with tagging additional stages pass 3:x.)

all this being said, there was no legit PR for that as from @altendky's comment #825 (comment) I got the impression that there was some thinking already spent on that, so there might be something in the backlog already.

@altendky if not on your top list i'm fine to prepare some starting PR.

[doma2345]

Hi @mk01 I am running plotman within the Machinaris docker container so it is not so simple for me to just edit the job.py and I have never had a go at building a docker container. I will try and give it a go and report back if I am able to get it working

[altendky]

plotman already tracks into major phase 4 for the Chia Network plotter. I thought we had that for madMAx as well, but there were also some recent fixes. And more might be needed...

I wouldn't say there's been a lot of thinking on this but I do have some structural work I want to do that would probably make PRs need to be reworked so it might be good to hold off at this point.

[badi95]

Any news on this? It's costing me ~15 min per plot waiting for the plot to transfer over the network. If it's going somewhere I'll hold off on using the fix from @mk01 otherwise I'll try his code out.

[altendky]

I think the merged PR https://github.com/ericaltendorf/plotman/pull/869/files included the "extra" phase?

[badi95]

Thanks! Seems like next release will have some good madmax improvements