[CI] Speedup CI by removing some unnecessary targets #3900

Job	Run time
build (X64, Release, ubuntu-latest)	1m 36s
build (X64, Trace, ubuntu-latest)	1m 48s
build (X64, Box32, ubuntu-latest)	1m 37s
build (RISCV, Release, ubuntu-latest)	5m 42s
build (RISCV, Trace, ubuntu-latest)	6m 37s
build (RISCV, StaticBuild, ubuntu-latest)	4m 11s
build (RISCV, Box32, ubuntu-latest)	4m 9s
build (RK3588, Release, ubuntu-latest)	5m 38s
build (RK3588, Trace, ubuntu-latest)	5m 47s
build (RK3588, StaticBuild, ubuntu-latest)	3m 51s
build (RK3588, Box32, ubuntu-latest)	4m 43s
build (ARM64, Release, ubuntu-latest)	5m 22s
build (ARM64, Trace, ubuntu-latest)	5m 32s
build (ARM64, StaticBuild, ubuntu-latest)	4m 3s
build (ARM64, Box32, ubuntu-latest)	4m 37s
build (ANDROID, Release, ubuntu-latest)	2m 8s
build (ANDROID, Trace, ubuntu-latest)	2m 21s
build (TERMUX, Release, ubuntu-latest)	2m 9s
build (TERMUX, Trace, ubuntu-latest)	2m 21s
build (LARCH64, Release, ubuntu-latest)	3m 34s
build (LARCH64, Trace, ubuntu-latest)	3m 57s
build (LARCH64, StaticBuild, ubuntu-latest)	1m 59s
build (LARCH64, Box32, ubuntu-latest)	3m 14s
	1h 26m 56s

Provide feedback