碰到一连串20M左右文件时无故异常中断退出 #153
Comments
|
|
我是用的syncup，还有syncdown但是下载和上传都会出现这种被杀死的情况，所以你说的碰到这种情况后先compare再syncup或者syncdown的动作能否程序完成呢？因为人工操作有几个问题，首先是效率低，而且不可能一直守着，特别是大批量文件的时候，而且每次重新运行程序会重新load本地MD5，其实都是不需要的，浪费了很多时间。其次是貌似这样的情况经常碰到，特别是上传或者下载的文件连续很大的时候，有时候我传一个文件要手工运行好几次。另外网盘软件同步貌似没有出现过这种意外退出的问题，是不是在文件块的大小参数上还需要调整？多谢，程序还是帮了我很大忙的，我是一个 Nas, 备份了很多照片视频，我想做的就是放到 Nas上的文件能够自动备份到百度网盘，我可以写一个脚本定期运行你的程序，但是希望每次运行都能完整同步。当然除非网络不通或者服务器down了。 |
|
你是说程序会被直接杀掉而不是你手动停掉的？这我倒没遇见过。 |
|
什么运行环境？ |
|
是啊，那个已杀死不是我人为操作。而且有时候连已杀死都不显示程序就莫名其妙终止了。我是用setsid 启动的，开始pstree可以看到进程，打开—v参数也能看到进度，但是碰到连续大文件的时候，比如进度条报到20%、40%，然后就半天不动了，重新pstree查一下发现进程已经没有了，只能手工重新启动，然后到60%又出现类似情况，所以一个文件我要反复手工启动，非常慢，因为每次启动都会去重新读取本地和远程目录树。在多批量小文件的时候没有出现过这样的状况，你可以放几个500M以上的电影文件，弄个6、7个测试看看。 |
|
我的 python是2.7.3, requests是2.7。 uname -r: cat /etc/issue: cat /proc/cpuinfo: |
|
哦，这样有点怀疑是MyBookLive的问题——释放资源？我以前用raspberry pi上传过几天的电影（有些是10G左右的文件），中间出过Connection reset by peear，但没有发生过被killed的事情。有办法查到log或dmesg是因为什么被杀掉的吗？ |
|
你说的情况的确很有可能，我也曾经怀疑过。我一共出现过三种错误，一种就是被 kill, 第二就是进程在上传文件中无缘无故消失了，也就是我上面描述的情况。第三种就是你说的 reset by peer, 其中第二种情况最为常见，几乎一定能重现。不过我一直不知道该怎么去查这个系统的 log判断是什么原因被杀，或者无故消失，求教！多谢！ |
|
这个系统我也不清楚。。:( |
|
我的意思是我对Linux里面查log的命令和方法不熟悉，你要是对哪个发行版熟悉告诉我那个发行版你怎么查的，我再去自己摸索。 |
|
我一般查log要么dmesg，要么去/var/log下面翻一翻，其他的我也没弄过，我也不知道程序被killed是在哪里会有log。我猜被杀的可能是内核要释放内存。 |
我是有一连串照片文件，大概几千张。前面3/4都备份的很好，但是备份到中间某段的时候发现程序总是会莫名退出，不知何故，于是打开Debug。（该段目录里面有大概200个左右的大文件，每个文件23M左右，比如下面的IMG_0149.CR2，奇怪的是如果程序中断后重复运行，又可以完成1-2个文件然后再次中断，我已经反复这样手工运行几十次了，才完成50几个文件，能否帮忙看看到底是什么原因？多谢。）
日志文件如下：
Request all goes fine
local type: F
'2012_08_12_昆山/CR2/IMG_0149.CR2' is being RapidUploaded.
[09:14:55] Cache hit for file '2012_08_12_昆山/CR2/IMG_0149.CR2',
md5: 929250cc5e161a54417b4fcb56a34134
size: 22518803
mtime: 1344773327
[09:14:55] Cache miss for file '2012_08_12_昆山/CR2/IMG_0149.CR2',
slice_md5: 717e2baf51cdf6896162471c30a81ffc
size: 22518803
mtime: 1344773327
[09:14:55] Periodically saving Hash Cash
[09:14:55] Cache miss for file '2012_08_12_昆山/CR2/IMG_0149.CR2',
crc32: 241980540
size: 22518803
mtime: 1344773327
[09:14:55] Periodically saving Hash Cash
RapidUploading Length: 22518803 MD5: 929250cc5e161a54417b4fcb56a34134, Slice-MD5: 717e2baf51cdf6896162471c30a81ffc, CRC: 0xe6c547cL
POST https://pcs.baidu.com/rest/2.0/pcs/file
actargs: None
Params: {u'slice-md5': '717e2baf51cdf6896162471c30a81ffc', u'content-length': 22518803L, u'ondup': u'overwrite', u'content-md5': '929250cc5e161a54417b4fcb56a34134', u'content-crc32': '0xe6c547cL', u'path': u'/apps/bypy/2012_08_12_\u6606\u5c71/CR2/IMG_0149.CR2', u'method': u'rapidupload'}
HTTP Status Code: 404
MD5 not found, rapidupload failed
'2012_08_12_昆山/CR2/IMG_0149.CR2' can't be RapidUploaded, now trying normal uploading.
'2012_08_12_昆山/CR2/IMG_0149.CR2' is being slicing uploaded.
Slice size: 20971520, Pieces: 2
Uploading MD5 slice: 90755cf62fc130686759b556c1837406, #1 / 2
POST https://c.pcs.baidu.com/rest/2.0/pcs/file
actargs: /apps/bypy/2012_08_12_昆山/CR2/IMG_0149.CR2
Params: {u'type': u'tmpfile', u'method': u'upload'}
[09:14:57] Error accessing 'https://c.pcs.baidu.com/rest/2.0/pcs/file'
[09:14:57] Exception:
('Connection aborted.', error(104, 'Connection reset by peer'))
Traceback (most recent call last):
File "/opt/local/lib/python2.7/site-packages/bypy-1.0.17-py2.7.egg/EGG-INFO/scripts/bypy.py", line 1258, in __request_work
File "/opt/local/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/api.py", line 109, in post
return request('post', url, data=data, json=json, *_kwargs)
File "/opt/local/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/api.py", line 50, in request
response = session.request(method=method, url=url, *_kwargs)
File "/opt/local/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/sessions.py", line 465, in request
resp = self.send(prep, *_send_kwargs)
File "/opt/local/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/sessions.py", line 573, in send
r = adapter.send(request, *_kwargs)
File "/opt/local/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(104, 'Connection reset by peer'))
[09:14:57] Function: __upload_slice_act
[09:14:57] Website parameters: {u'type': u'tmpfile', u'method': u'upload'}
[09:14:57] Waiting 10 seconds before retrying...
[09:15:07] Request Try #2 / 5
POST https://c.pcs.baidu.com/rest/2.0/pcs/file
actargs: /apps/bypy/2012_08_12_昆山/CR2/IMG_0149.CR2
Params: {u'type': u'tmpfile', u'method': u'upload'}
[09:15:08] Error accessing 'https://c.pcs.baidu.com/rest/2.0/pcs/file'
[09:15:08] Exception:
('Connection aborted.', error(104, 'Connection reset by peer'))
Traceback (most recent call last):
File "/opt/local/lib/python2.7/site-packages/bypy-1.0.17-py2.7.egg/EGG-INFO/scripts/bypy.py", line 1258, in __request_work
File "/opt/local/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/api.py", line 109, in post
return request('post', url, data=data, json=json, *_kwargs)
File "/opt/local/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/api.py", line 50, in request
response = session.request(method=method, url=url, *_kwargs)
File "/opt/local/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/sessions.py", line 465, in request
resp = self.send(prep, *_send_kwargs)
File "/opt/local/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/sessions.py", line 573, in send
r = adapter.send(request, *_kwargs)
File "/opt/local/lib/python2.7/site-packages/requests-2.7.0-py2.7.egg/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(104, 'Connection reset by peer'))
[09:15:08] Function: __upload_slice_act
[09:15:08] Website parameters: {u'type': u'tmpfile', u'method': u'upload'}
[09:15:08] Waiting 20 seconds before retrying...
[09:15:28] Request Try #3 / 5
[1]+ 已杀死
The text was updated successfully, but these errors were encountered: