Performance issues with swift.index_while_building

[jerrymarino]

Good morning! I'm again looking at how to use the feature swift.index_while_building with Bazel and found it to introduce significant build performance issues. The impact of using this flag is a 300% increase in swift compilation times as opposed to not using it.

From what I can tell, it adds multiple GB of output to the build - as there is a per swift_library index, all system modules and deps seem to be indexed O(N swift_library) times. rules_ios uses the program index-import which churns trying to import these indicies for longer than it'd takes Xcode to index the source code after build. Using a "global index" - a single index mitigates the perf impact, but I'm not sure how this would work with remote caching.

I was curious how "index while building" is working with rules_swift and Bazel in general for larger builds. XCHammer doesn't have this feature and works around it by letting Xcode index a focused subset after the build.

[brentleyjones]

At Target we used a global index, but only when running locally: 6b77a6b.

Remote cache hits would still download to each module folder, but post build we would copy those into the global index. Locally built modules would go directly into the global index. We then only index-import from the global index, also only importing units and records that changed: MobileNativeFoundation/index-import@6bb07ce

We personally found that the largest hit was the vast number of files and directories being made (and re-made, since declare_directory would delete the indexstore before re-populating it). I wrote up about it here: https://brentley.dev/xcodebuild-vs-bazel-incremental/#multiple-index-stores

---

I would love to see a first class solution for a global index store, but I think it would require changes to Bazel: bazelbuild/bazel#7527

[jerrymarino]

@brentleyjones great to hear that you've got this working! Also thanks for taking the time to write up how you're dealing with this, I'm going to give your blog post a read 👍

Remote cache hits would still download to each module folder, but post build we would copy those into the global index.

So you're writing a per-swift_library-index into the cache - then locally you pull from the cache; locally the build also uses a global index store. I assumed the file .."/rules_swift_global_index_enabled" is how you're able to get the worker to manage this.

I was thinking it'd be ideal to stripping out a minimal subset of records before writing to the cache - perhaps that's not 100% necessary if the artifacts are reasonably small. I was also thinking to leverage -index-ignore-system-modules to minimize the size here.

With the incremental index-import MobileNativeFoundation/index-import@6bb07ce you're able to get index-import running at a reasonable speed importing the out of band global index into DerivedData? I was hoping index-import would be fast enough and assumed it might have been if people didn't devise a way to patch Xcode to use bazel's index.

---

Yep it seems like Bazel needs some solution to bazelbuild/bazel#7527. Like the clang module cache this is basically a mutable tree artifact that many actions need to mutate 🤔

[amberdixon]

I was also thinking to leverage -index-ignore-system-modules to minimize the size here.

Looks like this feature was introduced here, yay! #563

[keith]

Definitely pass that to reduce extra work

[brentleyjones]

My analysis was with that flag on, btw.

[jerrymarino]

Nice, I think this is a sensible default! I tried passing -index-ignore-system-modules but it's still a large regression. I can post up stats.

Basically there is a O(N indexes * M imports) problem here, so repos with a ton of code will have same problem but with internal code: N*M indexing internal code anyhow. Making a per module index ( N * M ) is the real root of the issue that going with a global index will fix.

In addition to burning CPU and I/O, I get OOMs due to the amount of code it's loading in memory: specifically when --remote_cache=some and --disk_cache=.

I think to implement "Index while building" in Bazel there's some options.

Use the original architecture of a global index and implicit cache

Remote exec / caching

Workers use a global index internally, and then write records and units into the remote cache by copying material from the worker's global index into bazel-out. Perhaps using a tree artifact to obfuscate this to Bazel is the way to go.

Locally

Local machines w/o cache writes write to a global index directly. Ideally Xcode can just read from this index, possibly with a patch to Xcode or somehow preventing the re-mapping

Implement "per module" indexing

Possibility 1. Perhaps we can add a feature to clang and swift that only index files in the module. This could be nice for other reasons
Possibility 2. Make everything a system import to piggy back onto -index-ignore-system-modules. This has tradeoffs in clang for diagnostics.

[segiddins]

Use the original architecture of a global index and implicit cache

This potentially seems doable with the worker, similar to how it gives a global module cache? We'd just need to be able to copy out the relevant bits of the index to the desired output directory after the swiftc invocation

[jerrymarino]

Yep it seems totally reasonable and I think we'd just need a C++ program to extract the relevant units/records. The other piece to this is making it work with the swift and objc rules. In practice, this rules_swift feature doesn't work with a large majority of the objc code and there isn't yet a way to handle objc code and remote caching I know if.

The other simple/pragmatic option is to disable "Index while building" in the interim: driving this with Xcode, letting it manage the indexes incrementally as you open files after a build. With a focused project it will quickly index the subset of the code in the editor w/o paying to index the entire graph. Even a theoretical optimized approach that uses a global index will have a noticeable impact on build perf.

[keith]

Closed by #567