exp: running experiments is very slow

[courentin]

Bug Report

Description

Hello!

I'm trying to test the suggestion made in #5557 but running dvc exp run export_corpus_test_fr --set-param debug=true is taking a lot of time (~8min) while reproducing the stage without experiment does not take so much time.
I tried to add as many directories in .dvcignore as I could but it doesn't change anything. So I suspect something in dvc.

Environment information

I'm using the dvc version from master (862e18) and dulwich from master too (bbcc4b).

DVC version: 2.0.5+862e18
---------------------------------
Platform: Python 3.7.6 on Darwin-20.2.0-x86_64-i386-64bit
Supports: http, https, s3
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s2s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s2s1
Repo: dvc, git

Additional Information:

Profiling information here: exp.prof.zip

[pmrowla]

The issue is with computing git status (via dulwich), not DVC status, so using .dvcignore won't address the issue.

[pmrowla]

@courentin @ettadar do you have large untracked directories in your workspaces?

I know @ettadar's issue mentioned .venv in particular. Is your venv directory gitignored? (if not, can you try adding it to your .gitignore and then see if the performance changes)

[shcheklein]

If it is the case with .venv not being git-ignored, it somewhat reminds me a discussion we had about DVC collecting stages and data files - should we ignore some common subset of directories (.env, node_modules, etc). I wonder if it makes sense in this kind of situations to at least write something if we some of those? Or may be mention in docs/faq?

[ettadar]

@courentin @ettadar do you have large untracked directories in your workspaces?

Nop.

I know @ettadar's issue mentioned .venv in particular. Is your venv directory gitignored? (if not, can you try adding it to your .gitignore and then see if the performance changes)

Yep, it is already gitignored. This is the output of git status:

On branch develop
Your branch is up to date with 'origin/develop'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   poetry.lock
        modified:   pyproject.toml

no changes added to commit (use "git add" and/or "git commit -a")

[pmrowla]

Can confirm with our example-get-started. If a large gitignored directory (i.e. .venv) is present, dvc exp run goes from taking 1.8s in cprofile to 11.7s.

It looks like dulwich status is walking the entire directory when checking for untracked files (instead of stopping after the top level directory is ignored). This is probably also the cause of the performance issue reverted in #5596

[courentin]

@courentin @ettadar do you have large untracked directories in your workspaces?

I have a .venv but it is gitignored

[pmrowla]

Should be resolved by jelmer/dulwich#853

[jdonzallaz]

Should this be resolved by the recent dvc updates ?

[pmrowla]

@jdonzallaz the upstream fix for dulwich has not been merged yet, so this issue will still affect the latest DVC release (2.0.15)

[pmrowla]

For reference, while we are waiting on a dulwich release, anyone experiencing this issue can install the latest dulwich from source to get the fix (this will only work for DVC installed via pip)

pip install -U --force-reinstall git+https://github.com/dulwich/dulwich.git