Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jotta-scanner fail to upload file #138

Open
EigilSletbak opened this issue Jan 5, 2017 · 13 comments
Open

jotta-scanner fail to upload file #138

EigilSletbak opened this issue Jan 5, 2017 · 13 comments

Comments

@EigilSletbak
Copy link

PC folder content:
eisl@EISL01 ~ $ ls -l /home/eisl/Documents
total 8
drwx------ 2 eisl eisl 4096 Sep 22 11:32 NoMachine
-rw-r--r-- 1 eisl eisl 34 Jan 4 13:52 Test.txt

Run jotta-scanner
eisl@EISL01 ~ $ jotta-scanner /home/eisl/Documents /Jotta/Archive/EISL01
Entering dir: /home/eisl/Documents
uploading 1 new files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 1/1 - 00:00:01
Network upload speed 33.543B/sec
Entering dir: /home/eisl/Documents/NoMachine
Finished syncing 1 files to JottaCloud, no errors. yay!

Web Client show empty Docments folder
image

Folder checked as mounted using jotta-fuse, showing 0 size file
eisl@EISL01 ~ $ ls -l /home/eisl/jottafs/Jotta/Archive/EISL01/Documents
total 0
-rw-r--r-- 0 eisl eisl 0 Jan 5 14:26 Test.txt

Removes the Documents folder using the Web Interface.
eisl@EISL01 ~ $ ls -l /home/eisl/jottafs/Jotta/Archive/EISL01
total 0

Running with debug on
eisl@EISL01 ~ $ jotta-scanner -l debug /home/eisl/Documents /Jotta/Archive/EISL01
DEBUG:jottalib.JFS:getting url: 'https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com', extra_headers=None, params=None
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.jottacloud.com
send: 'GET /jfs/eigil.sletbak@gmail.com HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\nAuthorization: Basic XXXX\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Thu, 05 Jan 2017 13:28:55 GMT
header: X-ID: 400889492780
header: Content-Type: text/xml
header: Transfer-Encoding: chunked
header: Server: Jetty(8.1.21.v20160908)
header: Strict-Transport-Security: max-age= 31536000
DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "GET /jfs/eigil.sletbak@gmail.com HTTP/1.1" 200 None
INFO:root:args: topdir u'/home/eisl/Documents', jottapath u'/Jotta/Archive/EISL01'
DEBUG:jottalib.jottacloud:_encode_filename_to_filesystem(u'/home/eisl/Documents')
DEBUG:jottalib.jottacloud:_encode_filename_to_filesystem('/home/eisl/Documents')
DEBUG:jottalib.jottacloud:_decode_filename_to_unicode('/home/eisl/Documents')
DEBUG:jottalib.jottacloud:compare walk: u'/home/eisl/Documents' -> 1 files
DEBUG:jottalib.jottacloud:_decode_filename_to_unicode('Test.txt')
DEBUG:jottalib.jottacloud:_decode_filename_to_unicode('NoMachine')
DEBUG:jottalib.jottacloud:get_jottapath u'/home/eisl/Documents' u'/home/eisl/Documents' u'/Jotta/Archive/EISL01'
DEBUG:jottalib.jottacloud:_decode_filename_to_unicode(u'/Jotta/Archive/EISL01/Documents')
DEBUG:jottalib.jottacloud:compare jottapath: u'/Jotta/Archive/EISL01/Documents'
DEBUG:jottalib.jottacloud:filelist u'/Jotta/Archive/EISL01/Documents'
DEBUG:jottalib.JFS:getting url: 'https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents', extra_headers=None, params=None
send: 'GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\nAuthorization: Basic XXXX\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Thu, 05 Jan 2017 13:28:56 GMT
header: Accept-Ranges: bytes
header: X-ID: 120555229075
header: Content-Type: text/xml
header: Transfer-Encoding: chunked
header: Server: Jetty(8.1.21.v20160908)
header: Strict-Transport-Security: max-age= 31536000
DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1" 200 None
DEBUG:jottalib.JFS:path: u'/Jotta/Archive/EISL01' + u'Documents'
INFO:jottalib.JFS:syncing u'/Jotta/Archive/EISL01/Documents'
DEBUG:jottalib.JFS:path: u'/Jotta/Archive/EISL01' + u'Documents'
DEBUG:jottalib.JFS:getting url: 'https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents', extra_headers=None, params=None
send: 'GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\nAuthorization: Basic XXXX\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Thu, 05 Jan 2017 13:28:56 GMT
header: Accept-Ranges: bytes
header: X-ID: 921222432945
header: Content-Type: text/xml
header: Transfer-Encoding: chunked
header: Server: Jetty(8.1.21.v20160908)
header: Strict-Transport-Security: max-age= 31536000
DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1" 200 None
DEBUG:jottalib.JFS:path: u'/Jotta/Archive/EISL01' + u'Documents'
DEBUG:jottalib.JFS:path: u'/Jotta/Archive/EISL01' + u'Documents'
DEBUG:jottalib.JFS:path: u'/Jotta/Archive/EISL01' + u'Documents'
DEBUG:root:folderlist u'/Jotta/Archive/EISL01/Documents'
DEBUG:jottalib.JFS:getting url: 'https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents', extra_headers=None, params=None
send: 'GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\nAuthorization: Basic XXXX\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Thu, 05 Jan 2017 13:28:56 GMT
header: Accept-Ranges: bytes
header: X-ID: 460642854897
header: Content-Type: text/xml
header: Transfer-Encoding: chunked
header: Server: Jetty(8.1.21.v20160908)
header: Strict-Transport-Security: max-age= 31536000
DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1" 200 None
DEBUG:jottalib.JFS:path: u'/Jotta/Archive/EISL01' + u'Documents'
INFO:jottalib.JFS:syncing u'/Jotta/Archive/EISL01/Documents'
DEBUG:jottalib.JFS:path: u'/Jotta/Archive/EISL01' + u'Documents'
DEBUG:jottalib.JFS:getting url: 'https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents', extra_headers=None, params=None
send: 'GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\nAuthorization: Basic XXXX\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Thu, 05 Jan 2017 13:28:56 GMT
header: Accept-Ranges: bytes
header: X-ID: 694074858768
header: Content-Type: text/xml
header: Transfer-Encoding: chunked
header: Server: Jetty(8.1.21.v20160908)
header: Strict-Transport-Security: max-age= 31536000
DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents HTTP/1.1" 200 None
DEBUG:jottalib.jottacloud:--cloudfiles: set([])
DEBUG:jottalib.jottacloud:--localfiles: set(['Test.txt'])
DEBUG:jottalib.jottacloud:--cloudfolders: set([])
DEBUG:jottalib.jottacloud:Create SyncFile from 'Test.txt'
DEBUG:jottalib.jottacloud:_encode_filename_to_filesystem('Test.txt')
DEBUG:jottalib.jottacloud:Got encoded filename 'Test.txt', joining with dirpath '/home/eisl/Documents'
DEBUG:jottalib.jottacloud:_encode_filename_to_filesystem('Test.txt')
DEBUG:jottalib.jottacloud:_decode_filename_to_unicode(u'/Jotta/Archive/EISL01/Documents')
DEBUG:jottalib.jottacloud:_decode_filename_to_unicode('Test.txt')
Entering dir: /home/eisl/Documents
DEBUG:jottalib.scanner:uploading new file: SyncFile(localpath='/home/eisl/Documents/Test.txt', jottapath=u'/Jotta/Archive/EISL01/Documents/Test.txt')
DEBUG:jottalib.scanner:running <function new at 0x7f7d55467230> with args ('/home/eisl/Documents/Test.txt', u'/Jotta/Archive/EISL01/Documents/Test.txt', <jottalib.JFS.JFS object at 0x7f7d55486d50>)
DEBUG:jottalib.JFS:posting content (len 34, hash a8b3b313e7685d4c400c0816f3b4c8f4) to url u'/Jotta/Archive/EISL01/Documents/Test.txt'
DEBUG:jottalib.JFS:posting content (len ?) to url https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/Test.txt
send: 'POST /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/Test.txt?cphash=a8b3b313e7685d4c400c0816f3b4c8f4 HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\njx_lisence: \r\nJMd5: a8b3b313e7685d4c400c0816f3b4c8f4\r\nJModified: 2017-01-04T13:52:49.311838\r\nJSize: 34\r\nX-Jfs-DeviceName: Jotta\r\nJCreated: 2017-01-04T13:52:49.311838\r\ncontent-type: multipart/form-data; boundary=932bcbb48e2d421994d6b8d4ae39c9f9\r\njx_csid: \r\nContent-Length: 602\r\nAuthorization: Basic XXXX\r\n\r\n'
send: <MultipartEncoder: {'created': ('', '2017-01-04T13:52:49.311838'), 'modified': ('', '2017-01-04T13:52:49.311838'), 'file': (u'Test.txt', <open file '/home/eisl/Documents/Test.txt', mode 'r' at 0x7f7d5546b0c0>, 'application/octet-stream'), 'md5': ('', 'a8b3b313e7685d4c400c0816f3b4c8f4')}>
sendIng a read()able
reply: 'HTTP/1.1 201 Created\r\n'
header: Date: Thu, 05 Jan 2017 13:28:56 GMT
header: X-ID: 525790647848
header: Content-Type: text/xml
header: Transfer-Encoding: chunked
header: Server: Jetty(8.1.21.v20160908)
header: Strict-Transport-Security: max-age= 31536000
DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "POST /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/Test.txt?cphash=a8b3b313e7685d4c400c0816f3b4c8f4 HTTP/1.1" 201 None
uploading 1 new files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 1/1 - 00:00:00
Network upload speed 85.788B/sec
DEBUG:jottalib.jottacloud:_encode_filename_to_filesystem('/home/eisl/Documents/NoMachine')
DEBUG:jottalib.jottacloud:_decode_filename_to_unicode('/home/eisl/Documents/NoMachine')
DEBUG:jottalib.jottacloud:compare walk: u'/home/eisl/Documents/NoMachine' -> 0 files
DEBUG:jottalib.jottacloud:get_jottapath u'/home/eisl/Documents' u'/home/eisl/Documents/NoMachine' u'/Jotta/Archive/EISL01'
DEBUG:jottalib.jottacloud:_decode_filename_to_unicode(u'/Jotta/Archive/EISL01/Documents/NoMachine')
DEBUG:jottalib.jottacloud:compare jottapath: u'/Jotta/Archive/EISL01/Documents/NoMachine'
DEBUG:jottalib.jottacloud:filelist u'/Jotta/Archive/EISL01/Documents/NoMachine'
DEBUG:jottalib.JFS:getting url: 'https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/NoMachine', extra_headers=None, params=None
send: 'GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/NoMachine HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\nAuthorization: Basic XXXX\r\n\r\n'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Date: Thu, 05 Jan 2017 13:28:57 GMT
header: Accept-Ranges: bytes
header: X-ID: 774401734012
header: Content-Type: text/xml
header: Transfer-Encoding: chunked
header: Server: Jetty(8.1.21.v20160908)
header: Strict-Transport-Security: max-age= 31536000
DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/NoMachine HTTP/1.1" 404 None
DEBUG:root:folderlist u'/Jotta/Archive/EISL01/Documents/NoMachine'
DEBUG:jottalib.JFS:getting url: 'https://www.jottacloud.com/jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/NoMachine', extra_headers=None, params=None
send: 'GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/NoMachine HTTP/1.1\r\nHost: www.jottacloud.com\r\nAccept-Encoding: identity\r\nX-JottaAPIVersion: 2.2\r\nUser-Agent: jottalib 0.5.1 (https://github.com/havardgulldahl/jottalib)\r\nAuthorization: Basic XXXX\r\n\r\n'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Date: Thu, 05 Jan 2017 13:28:57 GMT
header: Accept-Ranges: bytes
header: X-ID: 666925415241
header: Content-Type: text/xml
header: Transfer-Encoding: chunked
header: Server: Jetty(8.1.21.v20160908)
header: Strict-Transport-Security: max-age= 31536000
DEBUG:requests.packages.urllib3.connectionpool:https://www.jottacloud.com:443 "GET /jfs/eigil.sletbak@gmail.com/Jotta/Archive/EISL01/Documents/NoMachine HTTP/1.1" 404 None
DEBUG:jottalib.jottacloud:--cloudfiles: set([])
DEBUG:jottalib.jottacloud:--localfiles: set([])
DEBUG:jottalib.jottacloud:--cloudfolders: set([])
Entering dir: /home/eisl/Documents/NoMachine
Finished syncing 1 files to JottaCloud, no errors. yay!

@sencer
Copy link

sencer commented Jan 6, 2017

Same here.

Linux Sencer 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

@sjurtf
Copy link

sjurtf commented Jan 9, 2017

What version of requests are you using?

It was recently descovered in #137 that newer versions of requests is breaking jottalib. The issue here looks like what was happening in #137.

@EigilSletbak
Copy link
Author

Seems to be relevant, this is my requests info:

eisl@EISL01 ~ $ pip show requests
Name: requests
Version: 2.12.4
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: me@kennethreitz.com
License: Apache 2.0
Location: /usr/local/lib/python2.7/dist-packages

@sjurtf
Copy link

sjurtf commented Jan 9, 2017

If you downgrade to requests 2.12.0 it should work again.

@EigilSletbak
Copy link
Author

Still fail with requests 2.12.0

eisl@EISL01 ~ $ pip show requests
Name: requests
Version: 2.12.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: me@kennethreitz.com
License: Apache 2.0
Location: /usr/local/lib/python2.7/dist-packages
Requires:

eisl@EISL01 ~ $ ls -l /home/eisl/Documents
total 8
drwx------ 2 eisl eisl 4096 Sep 22 11:32 NoMachine
-rw-r--r-- 1 eisl eisl 34 Jan 4 13:52 Test.txt

eisl@EISL01 ~ $ jotta-scanner /home/eisl/Documents /Jotta/Archive/EISL01
Entering dir: /home/eisl/Documents
uploading 1 new files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 1/1 - 00:00:00
Network upload speed 119.602B/sec
Entering dir: /home/eisl/Documents/NoMachine
Finished syncing 1 files to JottaCloud, no errors. yay!

eisl@EISL01 ~ $ ls -l /home/eisl/jottafs/Jotta/Archive/EISL01/Documents
total 0
-rw-r--r-- 0 eisl eisl 0 Jan 9 15:36 Test.txt

@sencer
Copy link

sencer commented Jan 9, 2017

I also tried downgrading requests to 2.9.1, but it didn't work.

@fholmer
Copy link

fholmer commented Jan 22, 2017

I have same issue too. jotta-upload and jotta-fuse is also affected. I inspected the files status reported by jotta api and all uploaded files have incomplete status. I also compared the md5sum reported by jotta API and it matches md5sum of local files. So content is uploaded but file is still incomplete. It is not possible to list, download, delete og restore these files.

@paaland
Copy link

paaland commented Jan 27, 2017

I too have this problem. Seems the files are there somewhere (lost in space), because running jotta-scanner multiple times it starts to compare existing files. So somehow jotta-scanner finds the previously uploaded files, but https://www.jottacloud.com does not. The folder structure gets created, but no files are visible on the web site.

$jotta-scanner --exclude .streams /mnt/ /QNAPNAS/Backup

Entering dir: /mnt/Documents
uploading 14 new files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 14/14 - 00:00:05
Network upload speed 650.720KiB/sec
comparing 31 existing files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 31/31 - 00:00:20
Entering dir: /mnt/Documents/Bilder
Entering dir: /mnt/Documents/Bilder/Polar campingvogn
uploading 2 new files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 2/2 - 00:00:00
Network upload speed 57.232KiB/sec
comparing 10 existing files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 10/10 - 00:00:05
comparing 2 existing files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 2/2 - 00:00:00
Entering dir: /mnt/Documents/Diverse
uploading 35 new files: [●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●] 9/35 - 00:00:04
Finished syncing 82 files to JottaCloud, no errors. yay!

@ghost
Copy link

ghost commented Feb 4, 2017

I have the same problem too:
$ pip show requests
Name: requests
Version: 2.13.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: me@kennethreitz.com
License: Apache 2.0
Location: /docker-data/_backup/pip/docker_backup/lib/python2.7/site-packages
Requires:

@sjurtf
Copy link

sjurtf commented Feb 4, 2017

We have found that at least these versions and combinations of libs are working together with jottalib. We have not had time to update one and one an check which is breaking it yet.

certifi==2016.2.28
clint==0.5.1
humanize==0.5.1
lxml==3.7.2
six==1.10.0
chardet==2.3.0
python-dateutil==2.5.3
requests-toolbelt==0.6.2
jottalib==0.5.1

@ghost
Copy link

ghost commented Feb 7, 2017

I tried using the versions listed above, but no luck. I have some other additional packages installed, not sure if that should make any difference:
appdirs (1.4.0)
args (0.1.0)
backports.ssl-match-hostname (3.5.0.1)
certifi (2016.2.28)
cffi (1.9.1)
chardet (2.3.0)
clint (0.5.1)
docker (2.0.2)
docker-pycreds (0.2.1)
humanize (0.5.1)
ipaddress (1.0.18)
jottalib (0.5.1)
lxml (3.7.2)
packaging (16.8)
pip (9.0.1)
pycparser (2.17)
pyparsing (2.1.10)
python-dateutil (2.5.3)
python-gnupg (0.4.0)
PyYAML (3.12)
requests (2.13.0)
requests-toolbelt (0.6.2)
setuptools (34.1.1)
six (1.10.0)
websocket-client (0.40.0)
wheel (0.29.0)
xattr (0.9.1)

@sjurtf
Copy link

sjurtf commented Feb 8, 2017

Try downgrading to requests 2.10.0.
You can also try using my fork - it should not really matter with this issue tho.
But Im using that one.

pip install git+https://github.com/sjurtf/jottalib.git

@ghost
Copy link

ghost commented Feb 10, 2017

Ok. I have now tested a bit more.
Downgrading worked. Yay!
I only have requests==2.10.0 in requirements to make upload work again. Thanks everybody!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants