Corrected performance data when batch size is greater than 1

[wgzintel]

openvino.genai/src/cpp/src/perf_metrics.cpp

Line 115 in fa324cf

raw_metrics.m_durations[i] /= batch_sizes[i];
@m_durations[i] should be the duration for one inference which may generate one or more (batch_size > 1) tokens. In the case batch_size>1, it means @batch_size tokens are generated together in @m_durations[i] time, not one token is generated in @m_durations[i]/@batch_size time

[as-suvorov]

m_durations is used to calculate ttop (Time (in ms) per output token (TPOT)) below: https://github.com/openvinotoolkit/openvino.genai/pull/1100/files#diff-99518905570503f302fd3af3fdede4154abef49e74385eb130d958bce9b22756R121

A note "// If in 10 ms a batch of 5 new tokens is generated then TPOT is 10 / 5 = 2 tok/ms." seems valid to me (except it should be 2 ms / tok).
Should ttop calculation be modified as well?
cc: @pavel-esir

[peterchen-intel]

@pavel-esir Looks like we can't simply remove this line, can you create a PR? The request is to give the duration time for a batch.

Related PRs #940, #1100

[pavel-esir]

@pavel-esir Looks like we can't simply remove this line, can you create a PR? The request is to give the duration time for a batch.

Related PRs #940, #1100

thanks for opening this discussion, i will take a look

[pavel-esir]

@pavel-esir Looks like we can't simply remove this line, can you create a PR? The request is to give the duration time for a batch.

Related PRs #940, #1100

@peterchen-intel if you need time points/duration of each batch you can get them from raw_metrics.m_new_token_times

eg:

result = pipe.generate(["The Sun is yellow because"], max_new_tokens=20)
new_token_times = result.perf_metrics.raw_metrics.m_new_token_times

All fields of openvino_genai.RawPerfMetrics can be found here
https://github.com/openvinotoolkit/openvino.genai/blob/master/src/cpp/include/openvino/genai/perf_metrics.hpp#L18-L44

Do we still a new PR?

[pavel-esir]

@peterchen-intel i have added example of getting times of generation of each token/batch of tokens from raw performance metrics here #1118

[peterchen-intel]

@pavel-esir Can we expose m_durations[i] with the time for the token(s) from one inference (generates batch_sizes[i] tokens)? Current m_durations[i] is not so convinced, since it can't generate one token in m_durations[i] when batch_sizes[i] >1, but generates batch_sizes[i] tokens with one inference in m_durations[i].