install recursive into existing dataset

[bpoldrack]

install --recursive is broken. It works fine if just calling datalad install -r -s whatever to/whereever.
But it doesn't if you called with an additional -d some. First level is installed, then it fails with

[ERROR  ] get() got an unexpected keyword argument dataset [dataset.py:apply_func:486] (TypeError)

I didn't look deeply into it yet, but this suggests, that get is called with dataset=... while already being bound to a dataset instance (or sth similar). I guess, we never really considered this use case and when calling get there's a confusion of what's the reference dataset to operate on.

FTR: Of course the same thing can still be achieved. datalad install .d . -s toplevel and only afterwards calling datalad install toplevel/sub works fine.

[yarikoptic]

wow, that is quite a blow indeed, fwiw replicated

$> datalad create testds
[INFO   ] Creating a new annex repo at /home/yoh/.tmp/testds 
create(ok): /home/yoh/.tmp/testds (dataset)
$> datalad install -d testds -r ///labs/haxby    
[INFO   ] Cloning http://datasets.datalad.org/labs/haxby [1 other candidates] into '/home/yoh/.tmp/testds/haxby' 
install(ok): haxby (dataset)                                                                                                                                                                                                      
[ERROR  ] get() got an unexpected keyword argument dataset [dataset.py:apply_func:434] (TypeError) 

[yarikoptic]

this seems to fix it

$> git diff datalad/distribution/install.py
diff --git a/datalad/distribution/install.py b/datalad/distribution/install.py
index f867cbf5..8180f5df 100644
--- a/datalad/distribution/install.py
+++ b/datalad/distribution/install.py
@@ -362,6 +362,11 @@ class Install(Interface):
 
         # Now, recursive calls:
         if recursive or get_data:
+            if 'dataset' in common_kwargs:
+                common_kwargs_ = common_kwargs.copy()
+                common_kwargs_.pop('dataset')
+            else:
+                common_kwargs_ = common_kwargs
             for r in destination_dataset.get(
                     curdir,
                     description=description,
@@ -374,7 +379,7 @@ class Install(Interface):
                     on_failure='ignore',
                     return_type='generator',
                     result_xfm=None,
-                    **common_kwargs):
+                    **common_kwargs_):
                 yield r
         # at this point no futher post-processing should be necessary,
         # `clone` and `get` must have done that (incl. parent handling)

but I've not done yet full analysis is that is the correct thing to do or just resort to calling Get.__call__ there and not yet 100% sure why we need get instead of install if recursive at that point

[yarikoptic]

yeap, this seems to work as well... I will add a test and submit PR to see if anything barfs

$> git diff datalad/distribution/install.py
diff --git a/datalad/distribution/install.py b/datalad/distribution/install.py
index f867cbf5..f8e7b548 100644
--- a/datalad/distribution/install.py
+++ b/datalad/distribution/install.py
@@ -362,8 +362,8 @@ class Install(Interface):
 
         # Now, recursive calls:
         if recursive or get_data:
-            for r in destination_dataset.get(
-                    curdir,
+            for r in Get.__call__(
+                    destination_dataset.path,
                     description=description,
                     # TODO expose this
                     # yoh: exactly!

[yarikoptic]

eh, not that easy -- apparently breaks test(s) causing "double installation"

$> python -m nose -s -v datalad/distribution/tests/test_install.py:test_install_recursive
datalad.distribution.tests.test_install.test_install_recursive ... > /home/yoh/proj/datalad/datalad-master/datalad/distribution/tests/test_install.py(370)test_install_recursive()
-> eq_(len(ds_list), 3)
(Pdb) p ds_list
[<Dataset path=/home/yoh/.tmp/datalad_temp_test_install_recursiveC97OrY>, <Dataset path=/home/yoh/.tmp/datalad_temp_test_install_recursiveC97OrY>, <Dataset path=/home/yoh/.tmp/datalad_temp_test_install_recursiveC97OrY/2>, <Dataset path=/home/yoh/.tmp/datalad_temp_test_install_recursiveC97OrY/subm 1>]
*(Pdb) for ds in ds_list: print ds.path
/home/yoh/.tmp/datalad_temp_test_install_recursiveC97OrY
/home/yoh/.tmp/datalad_temp_test_install_recursiveC97OrY
/home/yoh/.tmp/datalad_temp_test_install_recursiveC97OrY/2
/home/yoh/.tmp/datalad_temp_test_install_recursiveC97OrY/subm 1

so either additional checking/filtering to not act is needed or smth else