Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement flat dir support #27

Merged
merged 24 commits into from
Jul 28, 2022
Merged

feat: implement flat dir support #27

merged 24 commits into from
Jul 28, 2022

Conversation

Gozala
Copy link
Collaborator

@Gozala Gozala commented Jul 21, 2022

Implements preliminary directory support (no sharding or blocksize checks yet). Also makes few API changes to support this:

  1. File.createImporter API is gone is removed. Instead one should do following:

    import * as UnixFS from "@ipld/unixfs"
    
    // Create filesystem writer and a blocks stream from which created blocks
    // can be read.
    const fs = UnixFS.create()
    
    // Create file writer that can be used to encode file and write
    // content into it.
    const file = UnixFS.createFileWriter(fs)
    file.write(new TextEncoder().encode('hello world'))
    const { cid } = await file.close()
    // eventually
    fs.close()
  2. Implementation had be updated to be based on WritableStream interface which seems to be more widely available now. This means you can simply create UnixFS writer from WritableStream directly:

    import * as UnixFS from "@ipld/unixfs"
    const { writable, readable } = new TransformStream()
    const fs = UnixFS.createWriter({ writable })
    const file = UnixFS.createFileWriter(fs)
    file.write(new TextEncoder().encode('hello world'))
    const { cid } = await file.close()
    // eventually
    fs.close()

Directory writer interface is intentionally very limited. You can only add entries that were created by writing a file or a directory in prior. This design choice guarantees that you can't have directories with partially written files which you'd need to somehow ensure get closed before you close the directory or have internals that would await for closures. I feel this is a better design as it makes certain bugs impossible (e.g. never resolving promise because some file was not closed).

In addition I have also added experimental .fork() method to directories which allows you to "fork" directory into a multiple divergent copies. This could be utilized for example to encode filesystem patches, but it's pretty rough.

⚠️ I had no time to write nearly enough tests here to make sure everything works as expected ⚠️

@Gozala Gozala requested a review from hugomrdias July 21, 2022 10:42
Copy link
Member

@hugomrdias hugomrdias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really like the api now

README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
src/directory/api.ts Outdated Show resolved Hide resolved
test/dir.spec.js Outdated Show resolved Hide resolved
src/directory.js Outdated Show resolved Hide resolved
Gozala and others added 5 commits July 27, 2022 05:25
Co-authored-by: Hugo Dias <hugomrdias@gmail.com>
Co-authored-by: Hugo Dias <hugomrdias@gmail.com>
@Gozala Gozala merged commit af49439 into main Jul 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants