Customize run amount

[Klemensas]

Noticed that running time metric has around ±10% variance between runs, I assume this depends on available resources.
While this is a nice metric the variance is a bit too high for my taste.

Looking at the source it seems like this is an average between 3 runs.
Any chance to allow customizing this via an option to pass the number of runs?

[ai]

Are you talking about the time metric or size metric?

[mbalabash]

@ai Just FYI, you can pass runs option inside estimo to get median value as a result.

But it won't guarantee that the results will always be the same. Because time metric depends on available on your device resources.

[ai]

@mbalabash right now we call estimo multiple times (thank for mentioning runs options, it will simplify code). But it is not enough to stabilize the time option.

The real way to have 100% stable result for time metric is to track CPU steps, rather than time. We are waiting for this issue to start using CPU steps mbalabash/estimo#5

[Klemensas]

Sorry, I had time in mind.

The linked issue is very promising and indeed seems like the best solution for this. But it seems rather stuck puppeteer/puppeteer#5223 (comment) or am I misunderstanding something?

I agree that runs won't fully stabilize the timings but allowing to customize that should still be helpful in the meantime. Or do you think it's not worth it?

[ai]

Or do you think it's not worth it?

Yeap. We already run it several times, increasing the runs number will not dramatically improve test stability.

The best way is to avoid calling any other task in parallel, which hard to archive on developers machines and CI.

Only size metric is stable.

Do you want to send PR to docs with a note about stability problem?

[Klemensas]

Sure