Support for folder download in Jotta-download

[antonhagg]

I am not sure, but it seems that jotta-download doesn't support download of whole folders. There might be something else causing this error, below is the error I get.

Traceback (most recent call last):
  File "C:\Python27\Scripts\jotta-download-script.py", line 9, in <module>
    load_entry_point('jottalib==0.4.post1', 'console_scripts', 'jotta-download')()
  File "c:\python27\lib\site-packages\jottalib\cli.py", line 226, in download
    total_size = remote_file.size
AttributeError: 'JFSFolder' object has no attribute 'size'

[havardgulldahl]

Thanks for this, @antonhagg !

No-one has implemented support for folder downloads yet. If you want to have a go, the field is yours :)

[antonhagg]

Ok cool!

So I see there might be multiple ways of getting this.
One is to get the full tree list beforehand and add all download paths to all files to a single que list. This will probably take some time to request from the server if there is a big tree.

So therefor I was thinking to only get one folder at the time and those files in that particular folder and work deeper and deeper within the tree structure (the same way most ftp transfers works).

By looking in the JFS.py file it seems that its possible to get the structure of a single folder in the class JFSFolder(object):

Where I think I lack knowledge is how the program should "remember" which folder has already been downloaded and which have not. Any suggestions how this can be done in a smart way?

[antonhagg]

Haven't come to far but suggestions how to go forward could be nice (btw, the last bit doesn't work) =)

def download(argv=None):
    if argv is None:
        argv = sys.argv[1:]
    parser = argparse.ArgumentParser(description='Download a file or folder from Jottacloud.')
    parser.add_argument('remoteobject', help='The path to the file or folder that you want to download')
    parser.add_argument('-l', '--loglevel', help='Logging level. Default: %(default)s.',
        choices=('debug', 'info', 'warning', 'error'), default='warning')
    parser.add_argument('-c', '--checksum', help='Verify checksum of file after download', action='store_true' )
    args = parse_args_and_apply_logging_level(parser, argv)
    jfs = JFS.JFS()
    root_folder = get_root_dir(jfs)
    path_to_object = posixpath.join(root_folder.path, args.remoteobject)
    remote_object = jfs.getObject(path_to_object)
    if hasattr(remote_object, 'size'): #Check if it's a file that is downloaded ny checking if the attribute 'size' exist
        remote_file = remote_object
        total_size = remote_file.size
        with open(remote_file.name, 'wb') as fh:
            bytes_read = 0
            with ProgressBar(expected_size=total_size) as bar:
                for chunk_num, chunk in enumerate(remote_file.stream()):
                    fh.write(chunk)
                    bytes_read += len(chunk)
                    bar.show(bytes_read)
        if args.checksum:
            md5_lf = JFS.calculate_md5(open(remote_file.name, 'rb'))
            md5_jf = remote_file.md5
            if md5_lf != md5_jf:
                print('''MD5 hashes don't match!''')
                print('%s - Checksum for downloaded file' % md5_lf)
                print('%s - Checksum for server file' % md5_jf)
                print('%s was NOT downloaded successfully' % remote_file.name)
                exit(1)
            print('%s - Checksum for downloaded file' % md5_lf)
            print('%s - Checksum for server file' % md5_jf)
        print('%s downloaded successfully' % remote_file.name)
    else: #if it's not a file it has to be a folder
        print "its a folder"
        print (remote_object.filedirlist().treefile)
        #print('%s downloaded successfully' % args.remoteobject)

[antonhagg]

So I've done some progress (I ended up going for ?mode=list way since I couldn't figure out how to only get one folder at the time). Any suggestions?

Currently this is what I got, but I can't seem to access the information within .tree
If i print the the whole remote_object.filedirlist().tree i see information about the files as well, but I struggle to access it. Any suggestions how to do this?

    else: #if it's not a file it has to be a folder
        print "its a folder"     
        for folder in remote_object.filedirlist().tree:
            print folder

[antonhagg]

Ok, maybe not the best code but it works.

def download(argv=None):
    if argv is None:
        argv = sys.argv[1:]
    parser = argparse.ArgumentParser(description='Download a file or folder from Jottacloud.')
    parser.add_argument('remoteobject', help='The path to the file or folder that you want to download')
    parser.add_argument('-l', '--loglevel', help='Logging level. Default: %(default)s.',
        choices=('debug', 'info', 'warning', 'error'), default='warning')
    parser.add_argument('-c', '--checksum', help='Verify checksum of file after download', action='store_true' )
    args = parse_args_and_apply_logging_level(parser, argv)
    jfs = JFS.JFS()
    root_folder = get_root_dir(jfs)
    path_to_object = posixpath.join(root_folder.path, args.remoteobject)
    print root_folder.path
    print args.remoteobject
    print path_to_object
    remote_object = jfs.getObject(path_to_object)
    if hasattr(remote_object, 'size'): #Check if it's a file that is downloaded by checking if the attribute 'size' exist
        remote_file = remote_object
        total_size = remote_file.size
        with open(remote_file.name, 'wb') as fh:
            bytes_read = 0
            with ProgressBar(expected_size=total_size) as bar:
                for chunk_num, chunk in enumerate(remote_file.stream()):
                    fh.write(chunk)
                    bytes_read += len(chunk)
                    bar.show(bytes_read)
        if args.checksum:
            md5_lf = JFS.calculate_md5(open(remote_file.name, 'rb'))
            md5_jf = remote_file.md5
            if md5_lf != md5_jf:
                print('''MD5 hashes don't match!''')
                print('%s - Checksum for downloaded file' % md5_lf)
                print('%s - Checksum for server file' % md5_jf)
                print('%s was NOT downloaded successfully' % remote_file.name)
                exit(1)
            print('%s - Checksum for downloaded file' % md5_lf)
            print('%s - Checksum for server file' % md5_jf)
        print('%s downloaded successfully' % remote_file.name)
    else: #if it's not a file it has to be a folder
        print "Its a folder"     
        fileTree = remote_object.filedirlist().tree #Download the folder tree
        print len(fileTree)
        char_in_path_to_object = (posixpath.split(path_to_object)[0]) #Characters up to the folder that we want to download
        #print char_in_path_to_object
        for folder in fileTree:
            rel_path_to_object = folder.lstrip(char_in_path_to_object)
            if not os.path.exists(rel_path_to_object):
                os.makedirs(rel_path_to_object)
            print rel_path_to_object
            #print folder
            for _file in fileTree[folder]:
                #download the files in this folder
                abs_path_to_object = posixpath.join(root_folder.path, posixpath.join(rel_path_to_object, _file[0])) #This is the absolute path to the file that is going to be downloaded
                print abs_path_to_object
                remote_object = jfs.getObject(abs_path_to_object)
                remote_file = remote_object
                total_size = remote_file.size
                print posixpath.join(rel_path_to_object,remote_file.name)
                with open(posixpath.join(rel_path_to_object,remote_file.name), 'wb') as fh:
                    bytes_read = 0
                    with ProgressBar(expected_size=total_size) as bar:
                        for chunk_num, chunk in enumerate(remote_file.stream()):
                            fh.write(chunk)
                            bytes_read += len(chunk)
                            bar.show(bytes_read)
                if args.checksum:
                    md5_lf = JFS.calculate_md5(open(posixpath.join(rel_path_to_object,remote_file.name), 'rb'))
                    md5_jf = remote_file.md5
                    if md5_lf != md5_jf:
                        print('''MD5 hashes don't match!''')
                        print('%s - Checksum for downloaded file' % md5_lf)
                        print('%s - Checksum for server file' % md5_jf)
                        print('%s was NOT downloaded successfully' % remote_file.name)
                        exit(1)
                    print('%s - Checksum for downloaded file' % md5_lf)
                    print('%s - Checksum for server file' % md5_jf)
                print('%s downloaded successfully' % remote_file.name)

[antonhagg]

I think I need to rewrite some of the code that was proposed in the version i submitted since there has been quite a lot of changes and fixes since I wrote the code in the first place. Any help is appriciated.

[havardgulldahl]

Thanks for your effort on this. I pulled the gist of your code and simplified it a bit. Please reopen if there are any issues