[BUG] Investigate compile time of stand-alone header include of ivf_flat, ivf_pq, and refine headers

[ahendriksen]

Describe the bug
Surfaced in PR #1437, discussion here.

A single file containing

#include <raft/neighbors/ivf_flat.cuh> // or:
#include <raft/neighbors/ivf_pq.cuh> // , or:
#include <raft/neighbors/refine.cuh>

takes ~2 minutes to compile.

Steps/Code to reproduce bug

The following tests should be cheap to compile, because they do not instantiate any templates. I checked the build metrics report at https://downloads.rapids.ai/ci/raft/pull-request/1439/b0b8fe5/cuda11_x86_64.tests_bench_prims.html. Surprisingly, some of the files are expensive to compile:

CMakeFiles/EXT_HEADERS_TEST_IMPLICIT.dir/test/ext_headers/raft_neighbors_ivf_pq.cu.o            2:21 min
CMakeFiles/EXT_HEADERS_TEST_COMPILED_IMPLICIT.dir/test/ext_headers/raft_neighbors_ivf_pq.cu.o   2:20 min
CMakeFiles/EXT_HEADERS_TEST_COMPILED_IMPLICIT.dir/test/ext_headers/raft_neighbors_ivf_flat.cu.o 2:10 min
CMakeFiles/EXT_HEADERS_TEST_IMPLICIT.dir/test/ext_headers/raft_neighbors_ivf_flat.cu.o          2:07 min
CMakeFiles/EXT_HEADERS_TEST_IMPLICIT.dir/test/ext_headers/raft_neighbors_refine.cu.o            2:04 min
CMakeFiles/EXT_HEADERS_TEST_COMPILED_IMPLICIT.dir/test/ext_headers/raft_neighbors_refine.cu.o   2:03 min

Expected behavior
Should compile in <20 seconds.

Additional context
Surfaced in PR #1437, discussion here.