Some love for update

[mih]

Fixes #2858, and makes the state of update and particularly its tests a bit nicer.

[codecov]

Codecov Report

Merging #2859 into master will increase coverage by 0.07%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #2859      +/-   ##
==========================================
+ Coverage   90.19%   90.26%   +0.07%     
==========================================
  Files         246      246              
  Lines       31786    31828      +42     
==========================================
+ Hits        28668    28731      +63     
+ Misses       3118     3097      -21

Impacted Files	Coverage Δ
datalad/distribution/tests/test_update.py	100% <100%> (ø)	⬆️
datalad/distribution/update.py	98.13% <100%> (+20%)	⬆️
datalad/support/annexrepo.py	88.15% <0%> (ø)	⬆️
datalad/support/gitrepo.py	88.57% <0%> (+0.09%)	⬆️
datalad/support/repo.py	91.3% <0%> (+3.8%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3432419...aba7afc. Read the comment docs.

[mih]

@yarikoptic @bpoldrack @kyleam I would appreciate some input on how to deal with this situation:

It happens after a tiny modification of datalad/distribution/tests/test_update.py:test_update_volatile_subds

The starting point is a repo with a submodule, and the origin repo of the submodule has been removed from the origin repo. Now we are in the root of the repo clone that still has the submodule

# submodule known
mih@meiner ...temp_test_update_volatile_subdsEcqfyR (git)-[master] % git submodule
 c948beb9ec443add41ac4e26f54d30dc84ee41af subm 1 (heads/master)
# submodule has origin
mih@meiner ...temp_test_update_volatile_subdsEcqfyR (git)-[master] % git -C subm\ 1 remote -v
origin  /tmp/datalad_temp_test_update_volatile_subdsiLnX25/subm 1 (fetch)
origin  /tmp/datalad_temp_test_update_volatile_subdsiLnX25/subm 1 (push)
# but origin of the submodule is gone
mih@meiner ...temp_test_update_volatile_subdsEcqfyR (git)-[master] % ls /tmp/datalad_temp_test_update_volatile_subdsiLnX25/subm\ 1 
ls: cannot access '/tmp/datalad_temp_test_update_volatile_subdsiLnX25/subm 1': No such file or directory

# and now the bummer!
mih@meiner ...temp_test_update_volatile_subdsEcqfyR (git)-[master] % git fetch
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 5 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (5/5), done.
From /tmp/datalad_temp_test_update_volatile_subdsiLnX25
   d1f26da..ef6ee29  master     -> origin/master
Fetching submodule subm 1
fatal: '/tmp/datalad_temp_test_update_volatile_subdsiLnX25/subm 1' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
1 mih@meiner ...temp_test_update_volatile_subdsEcqfyR (git)-[master] % git fetch

So a plain git fetch fails, but only the first time, the second attempt is just fine. I am not even sure, why it would touch the submodule in the clone at all!? I also cannot see a difference in the repos before and after the fetch.

Moreover, if I do further changes to the origin repo, and fetch those, the problem doesn't reappear -- all is splendid.

Q is now how to deal with this situation? Going through a hierarchy of subdataset and disabling remotes just because they did not response at the time of a fetch is not really an option, but running things twice when they fail also doesn't sound good.

Any input is much appreciated!

[kyleam]

So a plain git fetch fails, but only the first time, the second attempt is just fine. I am not even sure, why it would touch the submodule in the clone at all!?

Explicitly disabling submodule recursion seems to fix it on my end.

diff --git a/datalad/distribution/update.py b/datalad/distribution/update.py
index 0f6eddab8..8abca67d3 100644
--- a/datalad/distribution/update.py
+++ b/datalad/distribution/update.py
@@ -168,6 +168,7 @@ def __call__(
                 # test against user-provided value!
                 remote=None if sibling is None else sibling_,
                 all_=sibling is None,
+                recurse_submodules="no",
                 prune=True)  # prune to not accumulate a mess over time
             repo.fetch(**fetch_kwargs)
             # NOTE if any further acces to `repo` is needed, reevaluate

[mih]

Wonderful, thx!

[mih]

Any further remarks?

[mih]

Alrighty...