Remove dependency file generator from checks

[vyasr]

Currently the checks shared workflow includes a job that runs dfg. As of version 1.2, dfg supports being run as a pre-commit hook, which is superior to this workflow because it can be run identically both locally and in CI. Moreover, all RAPIDS repos were migrated to using pre-commit for style checks during the GHA migration, so there's very little additional work required to add the new dfg hook. Once all repositories have been migrated to use the hook, the dfg support can be removed from the check here.

[vyasr]

CC @ajschmidt8

[ajschmidt8]

@vyasr, do we have plans to adapt this hook to support the commands below, which remove any existing generated files before re-running the dfg tool?

• https://github.com/rapidsai/gha-tools/blob/37f3946294647eadf5b60d8229ac396a4d913b59/tools/rapids-dependency-file-checker#L9-L15

These lines ensure that files that are no longer generated by the tool as cleaned up from the repository.

[vyasr]

Yeah I realized that was a missing bit. We could implement that as part of the Python package itself; I think it would be reasonable to do and better than relying on the action for this. The main question would be where we start searching for these files from. Since the location of dependencies.yaml is configurable, I assume that we don't want to search from the directory containing dependencies.yaml. The action assumes that it is running from the root of the repository. Do we want to bake the assumption that we are in a git repository into dfg? If so, we could use GitPython.

[ajschmidt8]

The action assumes that it is running from the root of the repository. Do we want to bake the assumption that we are in a git repository into dfg? If so, we could use GitPython.

I'd really like to avoid adding additional dependencies to dfg if possible.

How does pre-commit handle this?

Can pre-commit be configured to run it from the root repository without needing GitPython installed?

Or will it execute from the current working directory (which may not be the root of the repository)?

I think starting from the location of dependencies.yaml is probably a fine idea since the generated files are all written to paths relative to that source file (and presumably, though not guaranteed, not written to directories higher up in the file tree. e.g. ../../some/path).

[vyasr]

That's a good point, pre-commit will always run from the root of the repository so we should be able to just use that directly. I'll prototype that solution.

[vyasr]

rapidsai/dependency-file-generator#32 adds the necessary feature to dfg. Once that is in, we should be able to specify the --clean option to the pre-commit hooks in each repo and move forward with the removal proposed here.

[bdice]

Repos that still have this check enabled:

• dask-cuda
• cucim
• kvikio
• cuxfilter
• ucx-py
• rapids-cmake
• some other private repos

[bdice]

I cleaned this up a bit in #102 but we need to address the repos listed above.

[jakirkham]

Would it make sense to move this issue to build-planning?

[jameslamb]

Looked at all the non-archived repos in the rapidsai org with pre-commit configs today (GitHub search).

Of those, the following had a dependencies.yaml at the root of the repo but no pre-commit check running rapids-dependency-file-generator.

• cucim
  • add rapids-dependency-file-generator pre-commmit hook cucim#682
• rapids-cmake (repo)
  • add rapids-dependency-file-generator pre-commit hook rapids-cmake#531
• some other private repos

[vyasr]

Thanks for getting started with this. It would be nice to clean up the remainder of these when you have some bandwidth!

[vyasr]

Based on the comments above, I think we can disable the check now in shared-workflows but leave the option as a no-op while we go through every repo removing the option, then once we're confident everything has it removed we can remove the check entirely from here. Am I missing anything?

[jameslamb]

I agree with that @vyasr

[vyasr]

Removing in #204