Fix Scala using add_dependencies_on_all_siblings=True

[stuhood]

After dependency inference was improved for Scala, add_dependencies_on_all_siblings was not removed:

pants/src/python/pants/backend/scala/target_types.py

Line 233 in c2f6404

add_dependencies_on_all_siblings=True,

This means that we are overly coarsening Scala (all BUILD targets end up compiled together).

We should fix this (or drive it via an option), but it will definitely impact compilation success rates, and might also impact performance. We should test out the impact in our testbed repositories.

[somdoron]

I've experimented with removing this flag on our codebase, which required some more improvements to scala parser, after those improvements (PR is coming), the number of coarsened targets increased from 80 to around 800. It also allowed us to reduce process_per_child_memory_usage and increase the parallelism.

The best benefit for us would be less cache invalidation of test results.

I would like to contribute a PR to address this. Should I remove the flag or allow it to be configurable?

[stuhood]

I would like to contribute a PR to address this. Should I remove the flag or allow it to be configurable?

Thanks a ton! It will need to be configurable for sure, in order to allow folks to migrate from the existing default. But I expect that we can begin deprecating the existing value immediately.

[Eric-Arellano]

Awesome! For other languages, we set it dynamically based on if dependency inference is enabled:

pants/src/python/pants/backend/python/target_types_rules.py

Lines 77 to 82 in c1b1eaa

	@rule
	def python_files_generator_settings(
	_: PythonFilesGeneratorSettingsRequest,
	python_infer: PythonInferSubsystem,
	) -> TargetFilesGeneratorSettings:
	return TargetFilesGeneratorSettings(add_dependencies_on_all_siblings=not python_infer.imports)

pants/src/python/pants/backend/python/target_types_rules.py

Line 556 in c1b1eaa

UnionRule(TargetFilesGeneratorSettingsRequest, PythonFilesGeneratorSettingsRequest),

We will want to preserve that. Without dependency inference, it is incredibly tedious to specify "sibling" dependencies and you have to do it via the overrides field.

So, I suspect we maybe also have a temporary option like [scala-infer].force_sibling_dependencies with a default to true, per our deprecation policy. We deprecate not setting the default, and then possibly deprecate the option entirely in the subsequent release.