restoration from a snapshot is failing #15432

binishabraham · 2015-12-15T04:18:39Z

I am getting the following exception while doing a restoration

curl -XPOST "http://hostname1:9200/_snapshot/snapshot_restore_repo/snapshot_07_12_15_17_08_56/_restore"
{"error":"RemoteTransportException[[hostname2][inet[/ipaddress:9300]][cluster:admin/snapshot/restore]]; nested: ConcurrentSnapshotExecutionException[[snapshot_restore_repo:snapshot_07_12_15_17_08_56] Restore process is already running in this cluster]; ","status":503}-bash-4.1$

Cluster state is shown below

GET _cluster/state

"restore": {
"snapshots": [
{
"snapshot": "snapshot_07_12_15_17_08_56",
"repository": "snapshot_restore_repo",
"state": "STARTED",
"indices": [
"indexname"
],
"shards": [
{
"index": "indexname",
"shard": 0,
"state": "SUCCESS"
},
{
"index": "indexname",
"shard": 1,
"state": "SUCCESS"
},
{
"index": "indexname",
"shard": 2,
"state": "INIT"
},
{
"index": "indexname",
"shard": 3,
"state": "SUCCESS"
},
{
"index": "indexname",
"shard": 4,
"state": "SUCCESS"
}
]
}
]
}

How to come out of "INIT" and complete restoration

clintongormley · 2015-12-15T13:53:49Z

What version of Elasticsearch are you using?

binishabraham · 2015-12-15T16:06:22Z

I am using 1.7.0

clintongormley · 2015-12-16T10:32:34Z

@imotov @ywelsch any ideas?

imotov · 2015-12-16T11:19:30Z

@binishabraham anything in the log file on master?

binishabraham · 2015-12-16T20:37:17Z

@clintongormley ,
@imotov

Please find more details

I used to restore the snapshot from one cluster to three other clusters.

Saw the following error during restoration from that particulat snapshot to last cluster

{"error":"MasterNotDiscoveredException[]","status":503}

But even after this exception I was able to find all the documents in the last cluster also. Count of documents equal to other three clusters.

From the next day snapshot onwards, I am getting the following exception for the fourth cluster

{"error":"RemoteTransportException[[app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]][cluster:admin/repository/put]]; nested: ElasticsearchIllegalStateException[trying to modify or unregister repository that is currently used ]; ","status":500} % Total % Received % Xferd Average Speed Time Time Time Current
372 Dload Upload Total Spent Left Speed
373 ^M 0 0 0 0 0 60 0 49059 --:--:-- --:--:-- --:--:-- 49059^M 0 0 0 0 0 60 0 59 --:--:-- 0:00:01 --:--:-- 0^M 0 0 0 0 0 60 0 29 --:--:-- 0:00:02 --:--:-- 0^M 0 0 0 0 0 60 0 19 --:--:-- 0:00:03 --:--:-- 0^M 0 0 0 0 0 60 0 14 - -:--:-- 0:00:04 --:--:-- 0^M176 293 146 293 0 60 68 14 0:00:04 0:00:04 --:--:-- 54
374 {"error":"RemoteTransportException[[app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]][cluster:admin/snapshot/restore]]; nested: ConcurrentSnapshotExecutionException[[snapshot_restore_repo:snapshot_08_12_15_15_32_36] Restore process is already running in this cluster]; ","status":503}

But cluster-2 and cluster-3 restores it successfully from same snapshot.

Exception from same day's ES log.

-bash-4.1$ grep Exception es-prd-b-cluster.log.2015-12-07
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/172.28.74.67:9300]][internal:discovery/zen/unicast_gte_1_4] request_id [2412434] timed out after [3751ms]
[2015-12-07 23:06:19,862][INFO ][discovery.zen] [app-wc-b1p.sys.company.net] failed to send join request to master [[app-wc-b2p.sys.company.net][FJ3lMq59TKmnSfdPlJNm5g][app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]]{master=true}], reason [RemoteTransportException[[app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[app-wc-b2p.sys.company.net][FJ3lMq59TKmnSfdPlJNm5g][app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]]{master=true}] not master for join request from [[app-wc-b1p.sys.company.net][wVA3GQwHTW6MM4jHlubQqA][app-wc-b1p.sys.company.net][inet[/172.28.74.7:9300]]{master=true}]]; ], tried [3] times
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/172.28.74.67:9300]][internal:discovery/zen/unicast_gte_1_4] request_id [2412470] timed out after [3750ms]
org.elasticsearch.ElasticsearchIllegalStateException: cluster state from a different master then the current one, rejecting (received [app-wc-b3p.sys.company.net][QNbUU32-T3yYaVBGBf6z9A][app-wc-b3p.sys.company.net][inet[/172.28.74.67:9300]]{master=true}, current [app-wc-b2p.sys.company.net][FJ3lMq59TKmnSfdPlJNm5g][app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]]{master=true})

I have the following questions as part of immediate fix

Any fix to this instead of recreating the index and reloading the data
Is there any provision to restore all the snapshots present in my share path
These are created on daily basis and present in a NAS mount point as /NAS/esbackup and has the names as follows
snapshot-snapshot_08_12_15_15_32_36, snapshot-snapshot_10_12_15_17_13_05 etc.

imotov · 2015-12-17T10:15:10Z

@binishabraham could you post here or send me by email the output of the following command as well as complete log from the master node?

curl "localhost:9200/_cluster/state/routing_table,customs"

binishabraham · 2015-12-17T17:43:19Z

@imotov
Please find details

I am using 7 node cluster. 5 - data nodes eligible to become master. 2 - client nodes

Log from master
[2015-12-07 23:01:45,541][WARN ][transport ] [app-wc-b1p.sys.company.net] Received response for a request that has timed out, sent [72441ms] ago, timed out [42440ms] ago, action [internal:discovery/zen/fd/master_ping], node [[app-wc-b3p.sys.company.net][QNbUU32-T3yYaVBGBf6z9A][app-wc-b3p.sys.company.net][inet[/172.28.74.67:9300]]{master=true}], id [2412225]
[2015-12-07 23:01:45,543][WARN ][transport ] [app-wc-b1p.sys.company.net] Received response for a request that has timed out, sent [42442ms] ago, timed out [12442ms] ago, action [internal:discovery/zen/fd/master_ping], node [[app-wc-b3p.sys.company.net][QNbUU32-T3yYaVBGBf6z9A][app-wc-b3p.sys.company.net][inet[/172.28.74.67:9300]]{master=true}], id [2412226]
[2015-12-07 23:02:23,013][WARN ][monitor.jvm ] [app-wc-b1p.sys.company.net] [gc][young][2250943][49420] duration [1.4s], collections [1]/[1.7s], total [1.4s]/[32m], memory [7gb]->[6.8gb]/[9.9gb], all_pools {[young] [248.8mb]->[878.5kb]/[266.2mb]}{[survivor] [20.6mb]->[21.9mb]/[33.2mb]}{[old] [6.7gb]->[6.7gb]/[9.6gb]}
[2015-12-07 23:06:15,118][INFO ][discovery.zen ] [app-wc-b1p.sys.company.net] master_left [[app-wc-b3p.sys.company.net][QNbUU32-T3yYaVBGBf6z9A][app-wc-b3p.sys.company.net][inet[/172.28.74.67:9300]]{master=true}], reason [failed to ping, tried [3] times, each with maximum [30s] timeout]
[2015-12-07 23:06:15,119][WARN ][discovery.zen ] [app-wc-b1p.sys.company.net] master left (reason = failed to ping, tried [3] times, each with maximum [30s] timeout), current nodes: {[app-wc-b2p.sys.company.net][FJ3lMq59TKmnSfdPlJNm5g][app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]]{master=true},[app-wc-b5p.sys.company.net][fCe-HjVqSdSj8Opr0A0amQ][app-wc-b5p.sys.company.net][inet[/172.28.74.69:9300]]{master=true},[app-wc-b4p.sys.company.net][ctuotCZqRjGbsZ1LfOIa0A][app-wc-b4p.sys.company.net][inet[/172.28.74.68:9300]]{master=true},[app-wc-b1p.sys.company.net][wVA3GQwHTW6MM4jHlubQqA][app-wc-b1p.sys.company.net][inet[/172.28.74.7:9300]]{master=true},[cspesweb-wc-b2p.sys.company.net][MNtdFAsqRI6SryWUEawdqA][cspesweb-wc-b2p.sys.company.net][inet[/76.96.55.242:9300]]{data=false, master=false},[cspesweb-wc-b1p.sys.company.net][vcU4GzyDTMCrYPfN_dbYkw][cspesweb-wc-b1p.sys.company.net][inet[/76.96.55.241:9300]]{data=false, master=false},}
[2015-12-07 23:06:15,119][INFO ][cluster.service ] [app-wc-b1p.sys.company.net] removed {[app-wc-b3p.sys.company.net][QNbUU32-T3yYaVBGBf6z9A][app-wc-b3p.sys.company.net][inet[/172.28.74.67:9300]]{master=true},}, reason: zen-disco-master_failed ([app-wc-b3p.sys.company.net][QNbUU32-T3yYaVBGBf6z9A][app-wc-b3p.sys.company.net][inet[/172.28.74.67:9300]]{master=true})
[2015-12-07 23:06:18,876][WARN ][discovery.zen.ping.unicast] [app-wc-b1p.sys.company.net] failed to send ping to [[#zen_unicast_3#][app-wc-b1p.sys.company.net][inet[/172.28.74.67:9300]]]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/172.28.74.67:9300]][internal:discovery/zen/unicast_gte_1_4] request_id [2412434] timed out after [3751ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:529)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2015-12-07 23:06:19,862][INFO ][discovery.zen ] [app-wc-b1p.sys.company.net] failed to send join request to master [[app-wc-b2p.sys.company.net][FJ3lMq59TKmnSfdPlJNm5g][app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]]{master=true}], reason [RemoteTransportException[[app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]][internal:discovery/zen/join]]; nested: ElasticsearchIllegalStateException[Node [[app-wc-b2p.sys.company.net][FJ3lMq59TKmnSfdPlJNm5g][app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]]{master=true}] not master for join request from [[app-wc-b1p.sys.company.net][wVA3GQwHTW6MM4jHlubQqA][app-wc-b1p.sys.company.net][inet[/172.28.74.7:9300]]{master=true}]]; ], tried [3] times
[2015-12-07 23:06:21,235][INFO ][cluster.service ] [app-wc-b1p.sys.company.net] detected_master [app-wc-b2p.sys.company.net][FJ3lMq59TKmnSfdPlJNm5g][app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]]{master=true}, reason: zen-disco-receive(from master [[app-wc-b2p.sys.company.net][FJ3lMq59TKmnSfdPlJNm5g][app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]]{master=true}])
[2015-12-07 23:06:23,631][WARN ][discovery.zen.ping.unicast] [app-wc-b1p.sys.company.net] failed to send ping to [[#zen_unicast_3#][app-wc-b1p.sys.company.net][inet[/172.28.74.67:9300]]]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/172.28.74.67:9300]][internal:discovery/zen/unicast_gte_1_4] request_id [2412470] timed out after [3750ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:529)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2015-12-07 23:08:21,515][WARN ][discovery.zen ] [app-wc-b1p.sys.company.net] received a cluster state from a different master then the current one, rejecting (received [app-wc-b3p.sys.company.net][QNbUU32-T3yYaVBGBf6z9A][app-wc-b3p.sys.company.net][inet[/172.28.74.67:9300]]{master=true}, current [app-wc-b2p.sys.company.net][FJ3lMq59TKmnSfdPlJNm5g][app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]]{master=true})
[2015-12-07 23:08:21,516][ERROR][discovery.zen ] [app-wc-b1p.sys.company.net] unexpected failure during [zen-disco-receive(from master [[app-wc-b3p.sys.company.net][QNbUU32-T3yYaVBGBf6z9A][app-wc-b3p.sys.company.net][inet[/172.28.74.67:9300]]{master=true}])]
org.elasticsearch.ElasticsearchIllegalStateException: cluster state from a different master then the current one, rejecting (received [app-wc-b3p.sys.company.net][QNbUU32-T3yYaVBGBf6z9A][app-wc-b3p.sys.company.net][inet[/172.28.74.67:9300]]{master=true}, current [app-wc-b2p.sys.company.net][FJ3lMq59TKmnSfdPlJNm5g][app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]]{master=true})
at org.elasticsearch.discovery.zen.ZenDiscovery.shouldIgnoreOrRejectNewClusterState(ZenDiscovery.java:898)
at org.elasticsearch.discovery.zen.ZenDiscovery$10.execute(ZenDiscovery.java:778)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:374)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:188)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:158)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2015-12-07 23:08:24,929][INFO ][cluster.service ] [app-wc-b1p.sys.company.net] added {[app-wc-b3p.sys.company.net][QNbUU32-T3yYaVBGBf6z9A][app-wc-b3p.sys.company.net][inet[/172.28.74.67:9300]]{master=true},}, reason: zen-disco-receive(from master [[app-wc-b2p.sys.company.net][FJ3lMq59TKmnSfdPlJNm5g][app-wc-b2p.sys.company.net][inet[/172.28.74.70:9300]]{master=true}])

GET _cluster/state/routing_table,customs
{
"cluster_name": "es-prd-b-cluster",
"routing_table": {
"indices": {
"company_prd": {
"shards": {
"0": [
{
"state": "STARTED",
"primary": false,
"node": "R3BcxJx1SfiLhdvGr8A-5Q",
"relocating_node": null,
"shard": 0,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": true,
"node": "bufngu7lS9igVfz0G1xVsg",
"relocating_node": null,
"shard": 0,
"index": "company_prd"
}
],
"1": [
{
"state": "STARTED",
"primary": false,
"node": "T6DUea7CTmm8VnjEDXCvMg",
"relocating_node": null,
"shard": 1,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": true,
"node": "o32Diim1T6qQGKrZynjsXw",
"relocating_node": null,
"shard": 1,
"index": "company_prd"
}
],
"2": [
{
"state": "STARTED",
"primary": false,
"node": "o32Diim1T6qQGKrZynjsXw",
"relocating_node": null,
"shard": 2,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": true,
"node": "pWSBCl57Qqu3b5R5EF1-vg",
"relocating_node": null,
"shard": 2,
"index": "company_prd"
}
],
"3": [
{
"state": "STARTED",
"primary": false,
"node": "bufngu7lS9igVfz0G1xVsg",
"relocating_node": null,
"shard": 3,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": true,
"node": "pWSBCl57Qqu3b5R5EF1-vg",
"relocating_node": null,
"shard": 3,
"index": "company_prd"
}
],
"4": [
{
"state": "STARTED",
"primary": true,
"node": "T6DUea7CTmm8VnjEDXCvMg",
"relocating_node": null,
"shard": 4,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": false,
"node": "R3BcxJx1SfiLhdvGr8A-5Q",
"relocating_node": null,
"shard": 4,
"index": "company_prd"
}
]
}
},
"company": {
"shards": {
"0": [
{
"state": "STARTED",
"primary": true,
"node": "bufngu7lS9igVfz0G1xVsg",
"relocating_node": null,
"shard": 0,
"index": "company"
},
{
"state": "STARTED",
"primary": false,
"node": "o32Diim1T6qQGKrZynjsXw",
"relocating_node": null,
"shard": 0,
"index": "company"
}
],
"1": [
{
"state": "STARTED",
"primary": true,
"node": "bufngu7lS9igVfz0G1xVsg",
"relocating_node": null,
"shard": 1,
"index": "company"
},
{
"state": "STARTED",
"primary": false,
"node": "o32Diim1T6qQGKrZynjsXw",
"relocating_node": null,
"shard": 1,
"index": "company"
}
],
"2": [
{
"state": "STARTED",
"primary": true,
"node": "T6DUea7CTmm8VnjEDXCvMg",
"relocating_node": null,
"shard": 2,
"index": "company"
},
{
"state": "STARTED",
"primary": false,
"node": "R3BcxJx1SfiLhdvGr8A-5Q",
"relocating_node": null,
"shard": 2,
"index": "company"
}
],
"3": [
{
"state": "STARTED",
"primary": true,
"node": "R3BcxJx1SfiLhdvGr8A-5Q",
"relocating_node": null,
"shard": 3,
"index": "company"
},
{
"state": "STARTED",
"primary": false,
"node": "pWSBCl57Qqu3b5R5EF1-vg",
"relocating_node": null,
"shard": 3,
"index": "company"
}
],
"4": [
{
"state": "STARTED",
"primary": true,
"node": "T6DUea7CTmm8VnjEDXCvMg",
"relocating_node": null,
"shard": 4,
"index": "company"
},
{
"state": "STARTED",
"primary": false,
"node": "pWSBCl57Qqu3b5R5EF1-vg",
"relocating_node": null,
"shard": 4,
"index": "company"
}
]
}
}
}
},
"routing_nodes": {
"unassigned": [],
"nodes": {
"T6DUea7CTmm8VnjEDXCvMg": [
{
"state": "STARTED",
"primary": false,
"node": "T6DUea7CTmm8VnjEDXCvMg",
"relocating_node": null,
"shard": 1,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": true,
"node": "T6DUea7CTmm8VnjEDXCvMg",
"relocating_node": null,
"shard": 4,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": true,
"node": "T6DUea7CTmm8VnjEDXCvMg",
"relocating_node": null,
"shard": 2,
"index": "company"
},
{
"state": "STARTED",
"primary": true,
"node": "T6DUea7CTmm8VnjEDXCvMg",
"relocating_node": null,
"shard": 4,
"index": "company"
}
],
"R3BcxJx1SfiLhdvGr8A-5Q": [
{
"state": "STARTED",
"primary": false,
"node": "R3BcxJx1SfiLhdvGr8A-5Q",
"relocating_node": null,
"shard": 0,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": false,
"node": "R3BcxJx1SfiLhdvGr8A-5Q",
"relocating_node": null,
"shard": 4,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": false,
"node": "R3BcxJx1SfiLhdvGr8A-5Q",
"relocating_node": null,
"shard": 2,
"index": "company"
},
{
"state": "STARTED",
"primary": true,
"node": "R3BcxJx1SfiLhdvGr8A-5Q",
"relocating_node": null,
"shard": 3,
"index": "company"
}
],
"bufngu7lS9igVfz0G1xVsg": [
{
"state": "STARTED",
"primary": true,
"node": "bufngu7lS9igVfz0G1xVsg",
"relocating_node": null,
"shard": 0,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": false,
"node": "bufngu7lS9igVfz0G1xVsg",
"relocating_node": null,
"shard": 3,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": true,
"node": "bufngu7lS9igVfz0G1xVsg",
"relocating_node": null,
"shard": 0,
"index": "company"
},
{
"state": "STARTED",
"primary": true,
"node": "bufngu7lS9igVfz0G1xVsg",
"relocating_node": null,
"shard": 1,
"index": "company"
}
],
"o32Diim1T6qQGKrZynjsXw": [
{
"state": "STARTED",
"primary": false,
"node": "o32Diim1T6qQGKrZynjsXw",
"relocating_node": null,
"shard": 2,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": true,
"node": "o32Diim1T6qQGKrZynjsXw",
"relocating_node": null,
"shard": 1,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": false,
"node": "o32Diim1T6qQGKrZynjsXw",
"relocating_node": null,
"shard": 0,
"index": "company"
},
{
"state": "STARTED",
"primary": false,
"node": "o32Diim1T6qQGKrZynjsXw",
"relocating_node": null,
"shard": 1,
"index": "company"
}
],
"pWSBCl57Qqu3b5R5EF1-vg": [
{
"state": "STARTED",
"primary": true,
"node": "pWSBCl57Qqu3b5R5EF1-vg",
"relocating_node": null,
"shard": 2,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": true,
"node": "pWSBCl57Qqu3b5R5EF1-vg",
"relocating_node": null,
"shard": 3,
"index": "company_prd"
},
{
"state": "STARTED",
"primary": false,
"node": "pWSBCl57Qqu3b5R5EF1-vg",
"relocating_node": null,
"shard": 3,
"index": "company"
},
{
"state": "STARTED",
"primary": false,
"node": "pWSBCl57Qqu3b5R5EF1-vg",
"relocating_node": null,
"shard": 4,
"index": "company"
}
]
}
},
"allocations": []
}

imotov · 2015-12-17T23:36:44Z

@binishabraham What happened to the indexname index? Did you delete it?

binishabraham · 2015-12-18T19:26:56Z

@imotov It was a mistake in my replacement
Here indexname (previous post) = company (last post)

imotov · 2015-12-18T19:41:22Z

@binishabraham could you try closing and opening the company index to see if it will get recovery process unstuck?

imotov · 2016-01-04T19:51:37Z

@binishabraham any updates?

binishabraham · 2016-01-05T20:13:39Z

@imotov I tried, but didn't workout. So deleted the index 'indexname' and restored it using the snapshot from similar ES cluster from another Farm.

clintongormley · 2016-02-02T13:53:01Z

No further info. Closing

clintongormley added feedback_needed :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs labels Dec 15, 2015

ppf2 mentioned this issue Jan 29, 2016

Prevent index closing while snapshot is restoring the indices #16321

Closed

clintongormley closed this as completed Feb 2, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

restoration from a snapshot is failing #15432

restoration from a snapshot is failing #15432

binishabraham commented Dec 15, 2015

clintongormley commented Dec 15, 2015

binishabraham commented Dec 15, 2015

clintongormley commented Dec 16, 2015

imotov commented Dec 16, 2015

binishabraham commented Dec 16, 2015

imotov commented Dec 17, 2015

binishabraham commented Dec 17, 2015

imotov commented Dec 17, 2015

binishabraham commented Dec 18, 2015

imotov commented Dec 18, 2015

imotov commented Jan 4, 2016

binishabraham commented Jan 5, 2016

clintongormley commented Feb 2, 2016

restoration from a snapshot is failing #15432

restoration from a snapshot is failing #15432

Comments

binishabraham commented Dec 15, 2015

clintongormley commented Dec 15, 2015

binishabraham commented Dec 15, 2015

clintongormley commented Dec 16, 2015

imotov commented Dec 16, 2015

binishabraham commented Dec 16, 2015

imotov commented Dec 17, 2015

binishabraham commented Dec 17, 2015

imotov commented Dec 17, 2015

binishabraham commented Dec 18, 2015

imotov commented Dec 18, 2015

imotov commented Jan 4, 2016

binishabraham commented Jan 5, 2016

clintongormley commented Feb 2, 2016