Fixes #1512 (#1522) #1455

Job	Run time
ubuntu-20.04 3.10 transducer	10m 52s
ubuntu-20.04 3.10 paraformer	12m 7s
ubuntu-20.04 3.10 zipformer2-ctc	11m 5s
ubuntu-22.04 3.10 transducer	7m 43s
ubuntu-22.04 3.10 paraformer	8m 57s
ubuntu-22.04 3.10 zipformer2-ctc	4m 35s
windows-latest 3.10 transducer	6m 30s
windows-latest 3.10 paraformer	3m 6s
windows-latest 3.10 zipformer2-ctc	6m 42s
macos-latest 3.10 transducer	3m 49s
macos-latest 3.10 paraformer	5m 43s
macos-latest 3.10 zipformer2-ctc	3m 46s
macos-14 3.10 transducer	4m 5s
macos-14 3.10 paraformer	5m 1s
macos-14 3.10 zipformer2-ctc	4m 17s
	1h 38m 18s

Provide feedback