feat: cache sources too #1700

Job	Run time
Build x86_64-unknown-linux-musl	3m 20s
Build aarch64-unknown-linux-musl	8m 2s
Build powerpc64le-unknown-linux-gnu	8m 26s
test (ubuntu-latest)	1m 51s
test (macos-latest)	3m 10s
Build x86_64-apple-darwin	12m 19s
test (windows-latest)	6m 8s
Build aarch64-apple-darwin	4m 22s
Build x86_64-pc-windows-gnu	15m 46s
Build x86_64-pc-windows-msvc	14m 12s
Post-process	2s
	1h 17m 38s

Provide feedback