Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

Add support for Rabin fingerprinting to js-ipfs #1283

Closed
daviddias opened this issue Mar 27, 2018 · 15 comments
Closed

Add support for Rabin fingerprinting to js-ipfs #1283

daviddias opened this issue Mar 27, 2018 · 15 comments
Labels
exp/wizard Extensive knowledge (implications, ramifications) required help wanted Seeking public contribution on this issue P2 Medium: Good to have, but can wait until someone steps up

Comments

@daviddias
Copy link
Member

For feature parity with go-ipfs

ipfs/notes#1
@daviddias daviddias changed the title Add support for Rabinfingerprinting to js-ipfs Add support for Rabin fingerprinting to js-ipfs Mar 27, 2018
@daviddias daviddias added exp/expert Having worked on the specific codebase is important help wanted Seeking public contribution on this issue P2 Medium: Good to have, but can wait until someone steps up labels May 29, 2018
@gitcoinbot
Copy link

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 0.4 ETH (247.87 USD @ $619.68/ETH) attached to it.

@vs77bb
Copy link

vs77bb commented Jun 4, 2018

Hi @diasdavid, Vivek from Gitcoin here. We posted a small bounty on this on Gitcoin to see if it brings some attention. If you'd like any other bounties, let us know - we'd love to support!

@daviddias daviddias added status/ready Ready to be worked exp/wizard Extensive knowledge (implications, ramifications) required and removed exp/expert Having worked on the specific codebase is important labels Jun 4, 2018
@gitcoinbot
Copy link

@rafaelboyero Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

  • warning (3 days)
  • escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

@timxor
Copy link

timxor commented Jun 28, 2018

Still need help on this?

@haraldrudell
Copy link

The ECMAScript client is in need of a rewrite: it uses some pre-2015 non-standard streams implementation, closures and dictionaries of functions when today's answer would be async iterators. Not a class in sight

with the go client, one can do
ipfs add --chunker=rabin-48-96-192 LICENSE
added QmaJfZamtgZAPgYfPe5HbjEzbgf3YHgCUCu7yfDadwNXSd LICENSE
1.06 KiB / 1.06 KiB [================================================]
ipfs object links QmaJfZamtgZAPgYfPe5HbjEzbgf3YHgCUCu7yfDadwNXSd
QmYcQE8seMKSqWq2GZfcpnfK3WgPXLqhtryNEfRX6TXqsy 185
QmdH3L1zu7hWu7nrpoCe33zfn3hDm6baYpTSENbRitpcFu 140
Qmaw7roFAKSj3V3QxmFUfpqU3dqAEMPMqHynqXxJmhhGhB 64
Qmewm77gB4V7cgnZLsuikXgrG8Q38M1TX8qPRBcKrw4yJ4 90
QmRd6EoLAyp6TXZmSRTByaAgEn5632J8u9NwFSiHfXurLx 95
QmeLhozoTP1z3RiLRBRUuLJ7xpmDh5oi1ShN91B4UPjRwA 172
QmdxhoXpK74gGbm2G8bAoCiuqDfbq44rN6RZKefCAFEhoZ 99
QmZNEhqU6uVjMc7p7xfqfwQK5YGxe1bdUNdG6Hq81149iw 139
Qmd1yFj2ZuNd28r5MaQaWkr3vcUwPF61i8oBdfRjaUC4V5 64
QmeJhcwipWbjXA5DdXVGe6Sh8kwBzADNnw3Zd3HKcPtPkm 82
Qme1G8Mk1PLThzCr11NzChQVy1FJoHCQkjK7rB2iTWKkaJ 58

@spm32
Copy link

spm32 commented Jul 13, 2018

Hey @tcsiwula - yes! If you'd still like to take it on let me know.

@mikeal
Copy link
Contributor

mikeal commented Jul 13, 2018 via email

@dordille
Copy link
Contributor

Looking for help on this? Took a look at the rabin module linked by @mikeal and was wondering if there is any concern about compatibility with the browserify build of js-ipfs, not too familiar with that process and whether modules that have c dependencies are included elsewhere.

@mikeal
Copy link
Contributor

mikeal commented Jul 24, 2018

While there isn't a browser version we can just set the browser field in package.json to{'rabin': null}. That way, in the browser, the require() statement will just return null and we can check for it and opt out of these feature later in the code.

@dordille
Copy link
Contributor

Submitted dat-ecosystem-archive/rabin/pull/24 to allow for configuring the window size and polynomial used by the chunker. This will allow chunking used by js-ipfs to match go-ipfs-chunker.

Will have PRs up shortly for ipfs/js-ipfs-unixfs-engine and ipfs/js-ipfs

@dordille dordille mentioned this issue Jul 25, 2018
Closed
@mikeal
Copy link
Contributor

mikeal commented Jul 25, 2018

Should we consider surfacing these chunker settings in unixfs so that, when files are changed, they can be re-chunked to more closely match the prior chunking?

@dordille
Copy link
Contributor

I'd have to defer to someone that knows more about how often settings like window and the polynomial will be changed. Looking at the equivalent go source, it doesn't seem like there is a practical way to alter the window size or the polynomial used when chunking.

Polynomial: https://github.com/ipfs/go-ipfs-chunker/blob/master/rabin.go#L11
Window: https://github.com/whyrusleeping/chunker/blob/master/chunker.go#L16

It would be pretty straightforward though to update the chunkers in unixfs to export default values used in addition to the stream handler.

@dordille dordille mentioned this issue Jul 25, 2018
Closed
dordille added a commit to dordille/js-ipfs-unixfs-engine that referenced this issue Jul 26, 2018
@dordille
feat(importer): Add rabin chunker to available importer chunker algor…
510b9a6 
…ithms

This is required to have feature parity with go-ipfs which supports rabin chunking algorithm. Rabin
chunker supports teh following `chunkerOptions`: `minChunkSize`, `avgChunkSize`, `window`, and
`polynomial`. Rabin chunker uses the same defaults specified by the go-ipfs-chunker.

Supports ipfs/js-ipfs#1283

License: MIT
Signed-off-by: Dan Ordille <dordille@gmail.com>
dordille added a commit to dordille/js-ipfs-unixfs-engine that referenced this issue Jul 26, 2018
@dordille
feat(importer): Add rabin chunker
8861e5f 
This is required to have feature parity with go-ipfs which supports rabin chunking algorithm.
Rabin chunker supports the following `chunkerOptions`
minChunkSize: {integer}
avgChunkSize: {integer}
maxChunkSize: {integer}
polynomial:   {string}
window:       {integer}
Rabin chunker uses the same defaults defined by go-ipfs-chunker.

Supports ipfs/js-ipfs#1283

License: MIT
Signed-off-by: Dan Ordille <dordille@gmail.com>
@dordille dordille mentioned this issue Jul 26, 2018
Closed
dordille added a commit to dordille/js-ipfs-unixfs-engine that referenced this issue Jul 26, 2018
@dordille
feat(importer): add rabin fingerprinting chunk algorithm
95bb9cc 
This is required to have feature parity with go-ipfs which supports rabin chunking algorithm.
Rabin chunker supports the following `chunkerOptions`
minChunkSize: {integer}
avgChunkSize: {integer}
maxChunkSize: {integer}
polynomial:   {string}
window:		  {integer}
Rabin chunker uses the same defaults defined by go-ipfs-chunker.

Supports ipfs/js-ipfs#1283

License: MIT
Signed-off-by: Dan Ordille <dordille@gmail.com>
dordille added a commit to dordille/js-ipfs-unixfs-engine that referenced this issue Jul 26, 2018
@dordille
feat(importer): add rabin fingerprinting chunk algorithm
83a5feb 
This is required to have feature parity with go-ipfs which supports rabin chunking algorithm.
Rabin chunker supports the following `chunkerOptions`
minChunkSize: {integer}
avgChunkSize: {integer}
maxChunkSize: {integer}
polynomial:   {string}
window:       {integer}
Rabin chunker uses the same defaults defined by go-ipfs-chunker.

Supports ipfs/js-ipfs#1283

License: MIT
Signed-off-by: Dan Ordille <dordille@gmail.com>
@dordille dordille mentioned this issue Jul 26, 2018
Merged
dordille added a commit to dordille/js-ipfs that referenced this issue Jul 26, 2018
@dordille
feat(core): add option to specify chunking algorithm
562e302 
This allows the chunking algorithm, and options to be specified when using the adding files.
Specifying chunker and options are identical to go-ipfs and support the following formats:
default, size-{size}, rabin, rabin-{avg}, rabin-{min}-{avg}-{max}
This is required to achieve parity with go-ipfs.

Fixes ipfs#1283

License: MIT
Signed-off-by: Dan Ordille <dordille@gmail.com>
@dordille dordille mentioned this issue Jul 26, 2018
Merged
dordille added a commit to dordille/js-ipfs that referenced this issue Jul 26, 2018
@dordille
feat(core): add option to specify chunking algorithm
8a15c0e 
This allows the chunking algorithm, and options to be specified when using the adding files.
Specifying chunker and options are identical to go-ipfs and support the following formats:
default, size-{size}, rabin, rabin-{avg}, rabin-{min}-{avg}-{max}
This is required to achieve parity with go-ipfs.

Fixes ipfs#1283

License: MIT
Signed-off-by: Dan Ordille <dordille@gmail.com>
alanshaw pushed a commit to dordille/js-ipfs that referenced this issue Aug 15, 2018
@dordille @alanshaw
feat(core): add option to specify chunking algorithm
1918f14 
This allows the chunking algorithm, and options to be specified when using the adding files.
Specifying chunker and options are identical to go-ipfs and support the following formats:
default, size-{size}, rabin, rabin-{avg}, rabin-{min}-{avg}-{max}
This is required to achieve parity with go-ipfs.

Fixes ipfs#1283

License: MIT
Signed-off-by: Dan Ordille <dordille@gmail.com>
alanshaw pushed a commit to dordille/js-ipfs that referenced this issue Aug 17, 2018
@dordille @alanshaw
feat(core): add option to specify chunking algorithm
84b0425 
This allows the chunking algorithm, and options to be specified when using the adding files.
Specifying chunker and options are identical to go-ipfs and support the following formats:
default, size-{size}, rabin, rabin-{avg}, rabin-{min}-{avg}-{max}
This is required to achieve parity with go-ipfs.

Fixes ipfs#1283

License: MIT
Signed-off-by: Dan Ordille <dordille@gmail.com>
@gitcoinbot
Copy link

⚡️ A tip worth 0.40000 ETH (106.31 USD @ $265.77/ETH) has been granted to @dordille for this issue from @vs77bb. ⚡️

Nice work @dordille! To redeem your tip, login to Gitcoin at https://gitcoin.co/explorer and select 'Claim Tip' from dropdown menu in the top right, or check your email for a link to the tip redemption page.

@gitcoinbot
Copy link

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

The funding of 0.4 ETH (106.31 USD @ $265.77/ETH) attached to this issue has been approved & issued.

@vs77bb
Copy link

vs77bb commented Aug 22, 2018

Hi @dordille great work here. We tipped you for your work via the Gitcoin bounty which was posted on the issue 🙂

alanshaw pushed a commit to dordille/js-ipfs that referenced this issue Aug 23, 2018
@dordille @alanshaw
feat(core): add option to specify chunking algorithm
261d6d2 
This allows the chunking algorithm, and options to be specified when using the adding files.
Specifying chunker and options are identical to go-ipfs and support the following formats:
default, size-{size}, rabin, rabin-{avg}, rabin-{min}-{avg}-{max}
This is required to achieve parity with go-ipfs.

Fixes ipfs#1283

License: MIT
Signed-off-by: Dan Ordille <dordille@gmail.com>
@ghost ghost removed the status/ready Ready to be worked label Aug 24, 2018
alanshaw pushed a commit that referenced this issue Aug 24, 2018
@dordille @alanshaw
feat: Add option to specify chunking algorithm when adding files (#1469)
4f805d3 
This allows the chunking algorithm, and options to be specified when using the adding files.
Specifying chunker and options are identical to go-ipfs and support the following formats:
default, size-{size}, rabin, rabin-{avg}, rabin-{min}-{avg}-{max}
This is required to achieve parity with go-ipfs.

Fixes #1283

License: MIT
Signed-off-by: Dan Ordille <dordille@gmail.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
exp/wizard Extensive knowledge (implications, ramifications) required help wanted Seeking public contribution on this issue P2 Medium: Good to have, but can wait until someone steps up
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@mikeal @dordille @haraldrudell @daviddias @timxor @spm32 @vs77bb @gitcoinbot