modify benchmark to make early stop work #73
Conversation
| while True: | ||
| s = get_time() | ||
| x = backend.to_device(x) | ||
| x = backend.to_device(sample) |
There was a problem hiding this comment.
I think it's wrong that we pass some constant sample instead of part of our dataset. It might be cached in some way
There was a problem hiding this comment.
The main purpose of this change is to avoid data_loader generating data during benchmark. Otherwise this is data_loader benchmark instead of MLP benchmark. Data_loader won't be used in production inference environment, instead it's possible to use same buffer holding different samples passing to the network. It means in real production environment, the input can be cached in some way, it's not a problem.
There was a problem hiding this comment.
There is no data generation during benchmarking. Right now we benchmark passing pre-generated data to compute device and forward pass. Main benchmarking happens with fw_times that is limited to net(backend.to_device(x))
There was a problem hiding this comment.
Maybe we could have an alternative change in PR#75.
We could still pass the pre-generated test loader data to the forward pass. But then we will need to drop the first 3 steps' performance data in benchmarking period. Because we notice the previous warm up does not work. According to the onednn verbose, the first 3 steps in benchmarking period show quite poor performance.
I notice that you have an issue mentioned you would move warmup directly into main pass. If this issue is solved, maybe we can close this PR.
I don't mind point |
|
Merged alternative version of this PR |
This PR
whilerather thanfor loopin benchmark, so that the early stop can be truly enabled. Otherwise, once min_batches is done, the benchmark will be over, which may generate unstable performance results whenmin_batchesis small.min_bacthesandmin_secondsintobenchmark_params.