[bazel] Allocate CW310s based on extra resources

[drewmacrae]

WIP requires custom bazel binary. Add's the extra resource "cw310" so that 1 test can be dispatched concurrently to other testing.

Signed-off-by: Drew Macrae drewmacrae@google.com

[drewmacrae]

Tested with a bazel test //... --test_tag_filters=-broken

[3,766 / 3,860] 136 / 1364 tests; 75 actions, 7 running; last test: //third_party/riscv-compliance:I-SLLI-01_fpga_cw310_test_rom
    FuseSoC hw/build.verilator_real; 511s local, remote-cache
    Testing //third_party/riscv-compliance:I-SB-01_fpga_cw310_rom; 1s local
    Action sw/device/silicon_creator/lib/boot_data_functest_prog_sim_dv.fake_dev_key_0.signed.64.scr.vmem; 1s remote-cache, linux-sandbox
    Action sw/device/silicon_creator/lib/boot_data_functest_prog_sim_dv.logs.txt; 0s remote-cache, linux-sandbox
    [Sched] Testing //sw/device/silicon_creator/rom/e2e:e2e_bootstrap_entry_test_unlocked0_fpga_cw310_rom; 461s
    [Sched] Testing //third_party/riscv-compliance:I-LH-01_fpga_cw310_test_rom; 459s
    [Sched] Testing //sw/device/silicon_creator/rom/e2e:e2e_bootstrap_entry_rma_fpga_cw310_rom; 459s

Note verilator is building and FPGA tests are running

[drewmacrae]

Using the "cpu:" tag to control verilator makes it harder to control the number of threads verilator tests are allocated at the command line. The machinery to build verilator for n threads is still in place, we can still build for 4 by default and tag verilator tests with "cpu:4", but bazelbuild/bazel#16786 keeps us from limiting the total number of tests HOST_CPUs/verilator_threads when we change the number of threads verilator is built for.

It's a hack but I'm using:

#build verilator for 8 threads instead of 4 for a ~30% speedup
build:fast --//hw:verilator_options=--threads,8

#8 thread verilator gets a nice speedup, but test_jobs break resource management r/n and jobs slows things down pretty significantly. 
#build:fast --jobs=HOST_CPUS*0.11
#build:fast --local_test_jobs=HOST_CPUS*0.11
build:fast --local_cpu_resources=HOST_CPUS*0.5

[drewmacrae]

there are some rstmgr and alert_handler tests that are a bit flaky when tests aren't exclusive. I think there are some race conditions. I've seen some failures in:
//sw/device/tests:rstmgr_alert_info_test_fpga_cw310_test_rom
//sw/device/tests:alert_handler_reverse_ping_in_deep_sleep_test_fpga_cw310_test_rom
//sw/device/tests:alert_handler_ping_timeout_test_fpga_cw310_rom

[drewmacrae]

Decreases our test suite from 345m down to 188m, but running tests in parallel means we can catch new errors orders of magnitude faster.

[drewmacrae]

profile2.gz
This is a result of the --profile command and can be viewed with chrome://tracing in the chrome browser.
The action queue gets filled with FPGA jobs while verilator is building the model... Verilator jobs then get kicked off as FPGA jobs complete.