optimize pytorch engine inference with falcon model

[grimoire]

Fix falcon tp

before

concurrency: 256
elapsed_time: 184.839s

first token latency(s)(min, max, ave): 0.645, 13.974, 4.690
per-token latency(s) percentile(50, 75, 95, 99): [0.139, 0.147, 0.263, 0.414]

number of prompt tokens: 242197
number of completion tokens: 220686
token throughput (completion token): 1193.938 token/s
token throughput (prompt + completion token): 2504.254 token/s
RPS (request per second): 5.410 req/s
RPM (request per minute): 324.607 req/min


after

concurrency: 256
elapsed_time: 128.932s

first token latency(s)(min, max, ave): 0.270, 10.948, 3.440
per-token latency(s) percentile(50, 75, 95, 99): [0.095, 0.102, 0.232, 0.441]

number of prompt tokens: 242197
number of completion tokens: 220686
token throughput (completion token): 1711.640 token/s
token throughput (prompt + completion token): 3590.119 token/s
RPS (request per second): 7.756 req/s
RPM (request per minute): 465.360 req/min

Only tested on origin falcon-7b

[lvhan028]

在未修改之前，这里带来的问题是什么？

[grimoire]

没有带来问题，不过 q_state 和 k_state 会从 3d 变成 4d，我觉得是个潜在风险

[RunningLeon]

LGTM