-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
snapshot save error: hash check failed for "meta.json" #3933
Comments
@tkald I am on Ubuntu 16.04 as well and unable to reproduce this. Is it possible you have disk corruption? https://help.ubuntu.com/community/FilesystemTroubleshooting Please reopen if you continue to see this after ruling out disk problems. |
I tried to run this backup task on 3 different consul server nodes. |
Also consul internal/automatic snapshots are created just fine:
|
Can reproduce it with Consul 1.0.7 (the Debian SID package). consul --version Error verifying snapshot file: failed to read snapshot file: failed checking integrity of snapshot: hash check failed for "meta.json" Consistently, on all nodes, all on different hardware. |
Same issue with consul 1.1.0. `[root@i-03f7f697d05a6944b consul]# consul snapshot save -stale -http-addr=${leader}:8500 test [root@i-03f7f697d05a6944b consul]# tar -xvf test [root@i-03f7f697d05a6944b consul]# cat SHA256SUMS [root@i-03f7f697d05a6944b consul]# sha256sum meta.json [root@i-03f7f697d05a6944b consul]# consul version |
Was in strange state as 1 one the 4 servers was leaving ("http://${LOCAL_IP}:8500/v1/operator/autopilot/health"). Solved this and no more problem with snapshots. |
(note that I still can reproduce the issue without 'strange state's :) ) |
Hi, I encountered this issue with RHEL too, is there any work around with it? :) |
I have a cluster running 1.3.0 which constantly exhibits this problem. I am no golang expert but did some digging. If I unpack the snapshot file with gtar the meta.json file ends with a newline. But it seems that when consul calculates the checksum when inspecting the file internally the data returned by tar.NewReader does not include the newline. The checksum in SHA256SUMS is correct for the file with the newline. Extracting and then repacking the snapshot files with gtar does not fix the problem. |
I continued experimenting and I can repair a broken snapshot by doing this:
This repacked archive now passes the inspect test. The original meta.json (in the broken file) was exactly 513 bytes long. |
Description of the Issue (and unexpected/desired result)
When running from command line on server node consul snapshot save consul.bak error is returned:
Error verifying snapshot file: failed to read snapshot file: failed checking integrity of snapshot: hash check failed for "meta.json"
consul.bak file itself is created.
Reproduction steps
run consul snapshot save consul.bak on server
consul version
for both Client and ServerClient:
[client version here]
Server:
Consul v1.0.6 Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
consul info
for both Client and ServerClient:
Server:
Operating system and Environment details
Ubuntu 16.04
Log Fragments or Link to gist
TIP: Use
-log-level=TRACE
on the client and server to capture the maximum log detail.The text was updated successfully, but these errors were encountered: