"failed to clone and verify log" error while using sumdbclone example

[rikkuness]

Hey, first of all I appreciate this is example code and I shouldn't expect too much, so thank you for the examples provided here.

Every time I've tried to clone the sumdb log I've hit this same sort of error, I've tried to empty out the database tables and start fresh and eventually hit the same error. The initial clone seems to work fine and the leaves table seems to fill up ~7.5GB of content, but after the initial clone the verify stage always fails with errors like the below:

sumdbclone.go:88] Failed to clone log: failed to clone and verify log: computed root 7e3499466c71a429de6508bb87514f506e3e7a10d47fc2b172cc610aa2dd8cc0 != provided checkpoint aef05edcb9f3f3a7198f6e5e891f97c535d3ad2aa56ee4247614b1b5850bdd1d for tree size 28371487

Any pointers where I'm going wrong are appreciated 🙏🏻

[mhutchinson]

That's pretty concerning as one obvious interpretation is that the data being served doesn't match the Merkle tree. I'll run this tool locally and see if I can reproduce the same error.

[rikkuness]

Appreciate the quick response! If there's anything I can do to test/debug this further my end do let me know 🙏🏻

[mhutchinson]

I've reproduced this here. I'm now trying again with a build from prior to #1055 and if that fails I'll try one just before #1026 to rule out recent code changes here from having caused this.

Error from my run:

sumdbclone.go:88] Failed to clone log: failed to clone and verify log: computed root e94a4d42675415860c2b0f64ffa774fd31c330480809eba8c4195a88fc2ea535 != provided checkpoint 77711a211cfaee20bb84e4dfa8105db446e6bff4ddc6b14a15c155fe1b361a12 for tree size 28375746

[mhutchinson]

Also fails at commit 4ac1b81c. Trying b4c43db2 now. If this doesn't work then we'll need to start looking at the log in more detail. It can't be completely broken as it appears that the Merkle structure of the log is still able to convince witnesses that it append-only:

go run github.com/transparency-dev/distributor/cmd/client@main --n=5
...
Log "sum.golang.org" (a32d071739c062f4973f1db8cc1069f517428d77105962b285bbf918c4062591)
✅ Got checkpoint:

go.sum database tree
28377760
gjkc0MxcaxgKZgoVughj6da+Mh+Lz+VV9MS0vonwPG8=

— sum.golang.org Az3grn7R0Oyd23klufH4hN2ZlRGKJb3wxAfIpDLJUZMmaEhnkzsAicUIhB14YTJlOVw60lACT3Pe0wRMaoj4kdkoQAk=
— ArmoredWitness-throbbing-bird Te71T0drlmYAAAAANS1rGd8LiLSFxiPFvXPDAzMOTj0jd56N+CE8DvTSaC7EW3TC1PgjiRJPg3EWhbkWOlLc3plGEBYnp4NHpTJABg==
— ArmoredWitness-snowy-sound iEWJ7zhrlmYAAAAAYsyZVwdlgY8Fw0u8meDOhrFe18V0PCzhtq8QysCqy3b00L92ZuBqPMqhwpQLOH/skmSA+dwWkichAZPmJz+EAQ==
— ArmoredWitness-rough-wind q5exkjhrlmYAAAAAk+nmSy47Uygkf31O+odZQ0uYOX4BN+hL5etpnXIaFwslqSfbBEul1hLcQ9Zb6qLInIYOCFQarSxaSaNUwtkuAQ==
— ArmoredWitness-shy-wind wKnyACtrlmYAAAAA8Z2oLmMNPfwwuWLD2NCAv/nwCt1JyqVipPT4Hf4UhzaF4O2CK5zRMj2sqHUy5eq4aCrx363YrcJK/T1Orh6hCA==
— ArmoredWitness-hidden-river yB97bDZrlmYAAAAAHpiYxvScn3TAIGpJ/+Xcfx5xjM17Xm4jUB6LpUtcxruTyLseA15aYgm2m6+mQwfc9EDt5V6m1gryft56HYn+Dw==

Witness timestamps:
— ArmoredWitness-throbbing-bird: 10 minutes ago (2024-07-16T13:44:55+01:00)
— ArmoredWitness-snowy-sound: 10 minutes ago (2024-07-16T13:44:40+01:00)
— ArmoredWitness-rough-wind: 10 minutes ago (2024-07-16T13:44:40+01:00)
— ArmoredWitness-shy-wind: 10 minutes ago (2024-07-16T13:44:27+01:00)
— ArmoredWitness-hidden-river: 10 minutes ago (2024-07-16T13:44:38+01:00)
...

If we need to start looking at the log, I'd start with checking that all of the leaf data correctly hashes to the hashes reported in the hash tiles in the merkle tree.

[mhutchinson]

Thanks for reporting this @rikkuness. I've got no idea how this ever worked, but at least it's working now!