Raw leaves described by alg != sha2-256/32 do not work ( also generalized test matrix ) #3552

mib-kd743naq · 2016-12-30T12:30:02Z

Version information:

go-ipfs version: 0.4.5-dev-99d72c2

Type:

Bug

Priority:

P1

Description:

This is a set of 45 test cases. Table shows what currently works and what doesn't. Description at the end.

Framing Data\Link	v0pb-sha2-256-32	v1pb-sha2-256-24	v1pb-sha1-20	v1pb-sha2-512-48	v1pb-sha2-256-32
v0pb-sha2-256-32	✔ QmXSt...SZy45	✔ zrkJE...xS9EL	✔ z8bvj...Q2dPd	✔ zM1zo...wYaBs	✔ zdj7W...XkNuC
v1pb-sha2-256-24	✔ QmNu4...2Xhpa	✔ zrkJE...foNgz	✔ z8bvi...WUNQe	✔ zM1zo...cbmBL	✔ zdj7W...dGsJt
v1pb-sha1-20	✔ QmPCU...f6DZn	✔ zrkJE...W7gKr	✔ z8bvi...HFE3S	✔ zM1zo...6yrt2	✔ zdj7W...asVU6
v1pb-sha2-512-48	✔ Qmasu...R5tzS	✔ zrkJE...Tohnj	✔ z8bvi...zYvF8	✔ zM1zo...ixMy4	✔ zdj7W...fr3kd
v1pb-sha2-256-32	✔ QmYtF...yy61r	✔ zrkJE...18ARP	✔ z8bvi...QszAN	✔ zM1zo...bvmfF	✔ zdj7W...LxkeR
v1raw-sha2-256-24	✔ QmTZH...1rcBx	✔ zrkJE...Kehpz	✔ z8bvi...fTnTU	✔ zM1zo...ZYJRd	✔ zdj7W...ihtGG
v1raw-sha1-20	✔ QmQpU...WFdjr	✔ zrkJE...4zJTB	✔ z8bvi...BjrEv	✔ zM1zo...okdG8	✔ zdj7W...oXWAA
v1raw-sha2-512-48	✔ QmZf6...QiUfj	✔ zrkJE...ogVJr	✔ z8bvi...3aVtu	✔ zM1zo...riR8P	✔ zdj7W...HDepB
v1raw-sha2-256-32	✔ QmZr6...mJpNj	✔ zrkJE...YKfbK	✔ z8bvi...7CnXu	✔ zM1zo...4tAgP	✔ zdj7W...R2MyH

The entire set represents the same data:

Tested "file" is 7 bytes long, containing exactly:

123\n45\n
or
31 32 33 0A 34 35 0A
4 chunks ( 3 unique ) splitting on the newline ( 0A ):

123
\n
45
\n

Organized into a file via 3 links as follows (arrangement is arbitrarily chosen, mostly to hit as many corners as I can think of):

         ___________________
        |                   |
        | FINAL OBJECT LINK |
        |___________________|
         /         \
        /           \
 ===========    ___________________
 || "123" ||   |                   |
 ===========   | INTERMEDIATE LINK |
               |___________________|
                 /     |       \
                /      |        \
      ==========   ==========    ___________________
      || "\n" ||   || "45" ||   |                   |
      ==========   ==========   | INTERMEDIATE LINK |
                                |___________________|
                                       |
                                       |
                                  ==========
                                  || "\n" ||
                                  ==========

Each data/link framing is described by <format>-<hashalg>-<truncation>, with the names being self explanatory.
Every link omits the optional size field entirely. This is deliberate, ask me for more info if the reasoning is unclear.
The entire dataset is available at QmYbb74fCe7pom18nPRw2rKAPMukza63DfQAdHG53WywbW in a prototype of "efficient put format". The same directory contains a (massaged) protoc --decode_raw of each <name>.put for easier reading.

The text was updated successfully, but these errors were encountered:

mib-kd743naq · 2016-12-30T12:35:16Z

@jbenet as per your comment this revision of the ticket includes the exact bytes (*.put) that I fed to http://127.0.0.1:5001/api/v0/block/put?format={F}&mhtype={T}&mhlen={L}, including the exact id under which ipfs stored/indexed the block.

@diasdavid @dignifiedquire I have not tested this against js-ipfs, currently do not have it set up. Please feel free to reuse the dataset.

@Kubuxu @whyrusleeping Not sure if this is an 0.4.5 blocker or not...

Kubuxu · 2016-12-30T14:08:46Z

@mib-kd743naq I am unable to fetch the dataset, I have set up my server to try to fetch it all the time so when you go online it will be fetched.

mib-kd743naq · 2016-12-30T14:44:54Z

@Kubuxu sorry, only have my laptop node up, and running around the conference doesn't help
Should be avail. for the next ~40 mins

Kubuxu · 2016-12-30T14:59:40Z

Got it.

Kubuxu · 2016-12-30T15:03:39Z

Typo

Trying to pin it:

ipfs ping add  QmYbb74fCe7pom18nPRw2rKAPMukza63DfQAdHG53WywbW
Error: multihash length inconsistent: &{1  185 [240]}

mib-kd743naq · 2016-12-30T15:05:25Z

@Kubuxu works here... perhaps s/ping/pin/ ?

Kubuxu · 2016-12-30T15:07:51Z

Yeah. :|

mib-kd743naq · 2017-01-01T09:21:53Z

@Kubuxu actually I just realized that the individual test objects themselves are not pinned in any way besides my laptop. Could you please:

ipfs pin add QmXStmqqnaBrfP94XjDy6Y39ygYGMEu7wQY1zxACpSZy45
ipfs pin add zrkJEqVXFugrfTfnKouxvqhPYYCEZ8xR6xS9EL
ipfs pin add z8bvj1kMtvw2BBGQNmnrBSbe7rr9Q2dPd
ipfs pin add zM1zoqKTnRq1nDrvRvEtHyY5EVFtqtiU2aSNoEvJ5aBXSryYoqLgx2TRVf8fGb28zRwYaBs
ipfs pin add zdj7WensUcg8BTMCKWay3tqMRFX4EKp7vvG5NhsYPYb8XkNuC
ipfs pin add QmNu4Qr8fx6KGsquhoPxN2r7KhyiEZTDyNMM1Ege42Xhpa
ipfs pin add zrkJEkxWEb91Wqi5MEChEX9K7niexPvwrfoNgz
ipfs pin add z8bvizRrNP6TWeRVFjHj7ppjho5wWUNQe
ipfs pin add zM1zooTWWFb3Qz4o5UPxsTubHjjgUU6XRJ6FfW8oGbcomNHDttmNySGjRP3qa4FHqFcbmBL
ipfs pin add zdj7WjNKJAfstF8sZ24BrzXsATE2j6NJLLHj2dcfMf96dGsJt
ipfs pin add QmPCUSABGvLcbT5mEUke4xnbQtWyYEXFWiYfyVmg6f6DZn
ipfs pin add zrkJEttLb3ZS2ejiCVEABpUS12yALX3HtW7gKr
ipfs pin add z8bvizBNsLQxdjzPwX3diwDRRNuyHFE3S
ipfs pin add zM1zosy8kvPMN224UHPcHS1yG8rkjRrbE1jv7dFmEQd4hrKZyrEePk6pz2RmEP6FMo6yrt2
ipfs pin add zdj7WhJPhomVdpuQfbRXBgAh7fKX5QnPyf4Jx4SvdRbpasVU6
ipfs pin add QmasuD6U3KMh1xk9S7P31FhtmVscgPuLeS7y79fGhR5tzS
ipfs pin add zrkJEkcDgbr8QirPCe1u5NuqanA689TNwTohnj
ipfs pin add z8bvizWcf5ygDhcthcqDncxPcXghzYvF8
ipfs pin add zM1zokDJkyFWUYgHsK3w6gsYojiiVE59WiAwH4cVZ1A64j2G3ZtsPMXe1TovickXiAixMy4
ipfs pin add zdj7WZ26dLvNN8HDb9q97iccbUrpr1SY89cFLCYGye5nfr3kd
ipfs pin add QmYtF44QRxonUCmuTzXzzM6BvZhm31ScNRnAJW8eLyy61r
ipfs pin add zrkJEx6uA5fkVTRzceo8HAfFSkMKjacGD18ARP
ipfs pin add z8bvixjJ5T5i5AuVPvpuWxSdBhfyQszAN
ipfs pin add zM1zotfVBYeLLtZ6FequfE6mtFhpbTC9i1eU9bztGWebLG4CCHzyhVb9mPabJtT17ibvmfF
ipfs pin add zdj7Wf3akXA1wixavzCBuGScQJNDDwXDkFDrSWo76WqqLxkeR
ipfs pin add QmTZHvDKNcteezN4pFuYVxaGsBxdYA7db28y32Fk71rcBx
ipfs pin add zrkJEceiNTnEvAtT1DBB8YYBDUD9xDwD4Kehpz
ipfs pin add z8bvixsxdW29qk86DWuxM1EKqJFqfTnTU
ipfs pin add zM1zokWGSVw4uE7hdYNRJeR5Xuvz6ugCxJyGEu1TfaZLyTegr1hyRaWEZWvXCqwFepZYJRd
ipfs pin add zdj7WVpqrZCizahDsCLGHQ4PthaeURBksvQiLfrYnkJVihtGG
ipfs pin add QmQpUNgBY5zeaxhLJiZegBH5jG9SYn9eEMEQNCTiYWFdjr
ipfs pin add zrkJEyQniyjrFgwVgGeTq6ih6rbUGuZNV4zJTB
ipfs pin add z8bvixjWUf9CMrWPHEPnJfB5UjbjBjrEv
ipfs pin add zM1zomNrhXGxsZFHUsGU8GC8w735DnPj8qndPSF1ThEuwoA652J9SJ6RBRAwaavRjookdG8
ipfs pin add zdj7WfReDohbVfSjnE6b9fVryQCyMPpxX6XWgE9t9cMnoXWAA
ipfs pin add QmZf6rWFuu7yKSfAdRL4ujK82LUejmYVSvxo4n9LAQiUfj
ipfs pin add zrkJEid8DSyZWD8j4S4FEYJEh7D9Ptd92ogVJr
ipfs pin add z8bvizPQpcjusmzKaSZQbCqdTbHD3aVtu
ipfs pin add zM1zopDo3k2mmNi5BPZRBgtAYMpcT8nffmdtqghEV3n1mw3Ko2gGfc4HJoNxkxR7h7riR8P
ipfs pin add zdj7WkpPDQu8S3yVpsM64p9VtrTPrKaYsRkFkKcbpYmnHDepB
ipfs pin add QmZr6fFPus4Djf8fnKC7VdCw2kaeNnHLz7u6hTyNpmJpNj
ipfs pin add zrkJEvoQdqBmpTaZCp8W7CgE2W36BkDgeYKfbK
ipfs pin add z8bviyMwLb6oYLBjWsNTxaBCfwKz7CnXu
ipfs pin add zM1zooVrTECynpfkpXcP7KkJLVmx3PsYiedzap4uQ2zj9rjEkVMag1h7bWFxgkxPGt4tAgP
ipfs pin add zdj7Wdomcy7ZkDTYUSanfJnQ8hxoPiddHR7dmwaYtaZiR2MyH

whyrusleeping · 2017-01-01T19:55:13Z

@mib-kd743naq that looks to be because of this hardcoding: https://github.com/ipfs/go-ipfs/blob/master/merkledag/raw.go#L15

whyrusleeping · 2017-03-11T01:12:53Z

Lets revisit the items in the grid, Its definitely worth trying to make progress here.

whyrusleeping · 2017-03-24T18:27:24Z

I think with some of the changes we've been making for cidv1 support in ipfs add that these will be a lot better.

Lets try and make a script that permutes all of these and outputs success/failures

mib-kd743naq · 2017-06-19T08:03:38Z

w00t! It looks like all the ✘-es in the matrix now properly resolve!

@whyrusleeping / @Kubuxu please add the relevant "fixed in" and close the ticket

mib-kd743naq · 2017-06-19T08:04:38Z

I should probably look into the IPLD / CBOR based stuff, prepare a similar test-matrix...
Is the specification finalized?

whyrusleeping · 2017-06-19T15:51:23Z

@mib-kd743naq thats great!! Thanks for reporting back :)

The specification for the cbor stuff is finalized, and its merged into go-ipfs. definitely give all of that a good run over

mib-kd743naq · 2017-10-12T15:09:32Z

I re-verified that all the problems are solved, adjusted all the "X"es to ✔s in the original description, and am closing this ticket

mib-kd743naq mentioned this issue Jan 1, 2017

Request for review of structures generated by a custom block-writer ipfs/notes#189

Closed

whyrusleeping added this to the ipfs 0.4.6 milestone Jan 6, 2017

whyrusleeping added the kind/bug A bug in existing code (including security flaws) label Jan 6, 2017

mib-kd743naq mentioned this issue Feb 9, 2017

Add batch operation to ipfs block put #3667

Closed

whyrusleeping modified the milestones: Ipfs 0.4.7, ipfs 0.4.6 Feb 16, 2017

whyrusleeping modified the milestones: Ipfs 0.4.8, Ipfs 0.4.7 Mar 11, 2017

whyrusleeping modified the milestones: Ipfs 0.4.9, Ipfs 0.4.8 Mar 24, 2017

whyrusleeping modified the milestones: Ipfs 0.4.10, Ipfs 0.4.9 May 8, 2017

whyrusleeping added the help wanted Seeking public contribution on this issue label May 8, 2017

magik6k modified the milestones: Ipfs 0.4.10, Ipfs 0.4.11 Jul 28, 2017

magik6k removed this from the Ipfs 0.4.10 milestone Jul 28, 2017

whyrusleeping modified the milestones: Ipfs 0.4.12, Ipfs 0.4.11 Aug 28, 2017

mib-kd743naq mentioned this issue Oct 12, 2017

Seeker can't seek when referencing data-containing node #4286

Closed

mib-kd743naq closed this as completed Oct 12, 2017

mib-kd743naq mentioned this issue Feb 22, 2018

Forbid non-cryptographic hashes within /ipfs namespace #4371

Open

mib-kd743naq mentioned this issue Aug 13, 2019

Adding additional file metadata to UnixFSv1 ipfs/specs#217

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raw leaves described by alg != sha2-256/32 do not work ( also generalized test matrix ) #3552

Raw leaves described by alg != sha2-256/32 do not work ( also generalized test matrix ) #3552

mib-kd743naq commented Dec 30, 2016 •

edited

Loading

mib-kd743naq commented Dec 30, 2016

Kubuxu commented Dec 30, 2016

mib-kd743naq commented Dec 30, 2016

Kubuxu commented Dec 30, 2016

Kubuxu commented Dec 30, 2016 •

edited

Loading

mib-kd743naq commented Dec 30, 2016

Kubuxu commented Dec 30, 2016

mib-kd743naq commented Jan 1, 2017

whyrusleeping commented Jan 1, 2017

whyrusleeping commented Mar 11, 2017

whyrusleeping commented Mar 24, 2017

mib-kd743naq commented Jun 19, 2017

mib-kd743naq commented Jun 19, 2017

whyrusleeping commented Jun 19, 2017

mib-kd743naq commented Oct 12, 2017

Raw leaves described by alg != sha2-256/32 do not work ( also generalized test matrix ) #3552

Raw leaves described by alg != sha2-256/32 do not work ( also generalized test matrix ) #3552

Comments

mib-kd743naq commented Dec 30, 2016 • edited Loading

Version information:

Type:

Priority:

Description:

mib-kd743naq commented Dec 30, 2016

Kubuxu commented Dec 30, 2016

mib-kd743naq commented Dec 30, 2016

Kubuxu commented Dec 30, 2016

Kubuxu commented Dec 30, 2016 • edited Loading

mib-kd743naq commented Dec 30, 2016

Kubuxu commented Dec 30, 2016

mib-kd743naq commented Jan 1, 2017

whyrusleeping commented Jan 1, 2017

whyrusleeping commented Mar 11, 2017

whyrusleeping commented Mar 24, 2017

mib-kd743naq commented Jun 19, 2017

mib-kd743naq commented Jun 19, 2017

whyrusleeping commented Jun 19, 2017

mib-kd743naq commented Oct 12, 2017

mib-kd743naq commented Dec 30, 2016 •

edited

Loading

Kubuxu commented Dec 30, 2016 •

edited

Loading