Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

introduce LinkSystem #143

Merged
merged 10 commits into from
Mar 12, 2021
Merged

introduce LinkSystem #143

merged 10 commits into from
Mar 12, 2021

Commits on Feb 25, 2021

  1. Introduce LinkSystem.

    This significantly reworks how linking is handled.
    
    All of the significant operations involved in storing and loading
    data are extracted into their own separate features, and the LinkSystem
    just composes them.  The big advantage of this is we can now add as
    many helper methods to the LinkSystem construct as we want -- whereas
    previously, adding methods to the Link interface was a difficult
    thing to do, because that interface shows up in a lot of places.
    
    Link is now *just* treated as a data holder -- it doesn't need logic
    attached to it directly.  This is much cleaner.
    
    The way we interact with the CID libraries is also different.
    We're doing multihash registries ourselves, and breaking our direct
    use of the go-multihash library.  The big upside is we're now using
    the familiar and standard hash.Hash interface from the golang stdlib.
    (And as a bonus, that actually works streamingly; go-mulithash didn't.)
    However, this also implies a really big change for downstream users:
    we're no longer baking as many hashes into the new multihash registry
    by default.
    warpfork committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    a1482fe View commit details
    Browse the repository at this point in the history
  2. Extract multi{codec,hash} registries better.

    And, make a package which can be imported to register "all" of the
    multihashes.  (Or at least all of them that you would've expected
    from go-multihash.)
    
    There are also packages that are split roughly per the transitive
    dependency it brings in, so you can pick and choose.
    
    This cascaded into more work than I might've expected.
    Turns out a handful of the things we have multihash identifiers for
    actually *do not* implement the standard hash.Hash contract at all.
    For these, I've made small shims.
    
    Test fixtures across the library switch to using sha2-512.
    Previously I had written a bunch of them to use sha3 variants,
    but since that is not in the standard library, I'm going to move away
    from that so as not to re-bloat the transitive dependency tree
    just for the tests and examples.
    warpfork committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    8fef531 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2021

  1. Configuration menu
    Copy the full SHA
    45b8e9c View commit details
    Browse the repository at this point in the history
  2. LinkSystem: detect direct byte access features on readers and attempt…

    … to use it.
    
    I'm still not fully convinced this is always a win for performance,
    but as discussed in
    #143 (comment) ,
    there was past evidence, and reproducing that takes work -- so, we're
    going to try to be relatively conservative here and keep this logical
    branch behavior in place for now.
    
    (The reason this may be worth re-examining is that the change to
    hashing interface probably gets rid of one big source of copies.
    That doens't answer the holistic performance question alone, though;
    it just reopens it.)
    warpfork committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    08f13e5 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8fcc676 View commit details
    Browse the repository at this point in the history
  4. Merge branch 'master' into linksystem

    Resolves conflicts in go.sum.
    (They were conflicts in a textual sense only, but semantically trivial;
    application of `go mod tidy` was sufficient.)
    warpfork committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    705307f View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    8a7497f View commit details
    Browse the repository at this point in the history
  6. Update some comments regarding multihash.

    (Some things got better.  Several others in the area still have not.)
    warpfork committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    6386588 View commit details
    Browse the repository at this point in the history
  7. Better document why some of the branches around direct byte slice acc…

    …ess matter in LinkSystem.
    
    (It can be hard to intuit this just by reading the code, because some
    of the key relevance is actually in *other* functions that might not
    be in this repo at all!)
    warpfork committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    d9d5682 View commit details
    Browse the repository at this point in the history
  8. Update changelog.

    warpfork committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    8ef5eab View commit details
    Browse the repository at this point in the history