Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for blobs #1

Closed
10 of 18 tasks
boreq opened this issue May 16, 2022 · 2 comments
Closed
10 of 18 tasks

Add support for blobs #1

boreq opened this issue May 16, 2022 · 2 comments
Assignees
Labels
enhancement New feature or request priority/high

Comments

@boreq
Copy link
Contributor

boreq commented May 16, 2022

Incoming messages should be automatically scanned for references to blobs and that information should be saved in the database. Blobs referenced in messages created by feeds up to n hops away should be automatically retrieved (probably just friends).

The following message content types should be scanned for blobs:

  • post
  • about

It is unclear if other messages reference blobs. A different approach should involve parsing the raw content bytes to find anything that matches the &<hash>.sha256 format.

We also need a mechanism to request blob retrieval on demand. This will be used for blobs further away than n hops.

Blobs have a wants mechanism which allow peers to forward wants from other nodes. It is unclear if this is mandatory to implement. I think this could be skipped for now.

When it comes to creating blobs: right now creating blobs is largely decoupled from creating messages that reference them. A blob is added which results in a ref and then that ref is included in a message. This means that if you give up on creating that message you are going to have an orphaned blob in your program. Maybe it makes sense to develop some new approach which forces the user to call some code to get a ref to a blob but that blob will not become live (or will get cleaned up after some time) unless the user actually embeds that blob in a message somehow.

Progress tracker (incomplete):

  • persist information about blobs
  • scan messages for blobs
  • replication manager? (prefetch some blobs, fetch others on demand)
  • persist blobs
  • forward remote wants (let's not do this now)
  • handle incoming blobs.get
  • handle incoming blobs.getSlice
  • handle incoming blobs.createWants
  • reply with "has" when "want" is received
  • cleanup wants processes after incoming and outgoing streams are disconnected
  • after replicating a blob check if someone would like to know about it (wants)
  • cleanup remote wants after all of our streams disconnect?
  • persist an "on demand" want list
  • revamp the want list so that the blobs that we retrieve are removed from it (probably just to the on-demand want list for now)
  • after UI asks for a blob on demand persist it in the want list for a specific amount of time
  • refresh want list when some events are emitted
  • redo on-demand want list cleanups (consult @czeslavo to figure out how cron+command?)
  • create blobs
@boreq boreq self-assigned this May 16, 2022
@boreq boreq added this to the needed-by-planetary milestone May 16, 2022
@boreq boreq moved this to Todo in Verse: Nos Projects May 16, 2022
@boreq boreq moved this from Todo to In Progress in Verse: Nos Projects May 16, 2022
@boreq boreq mentioned this issue May 17, 2022
Closed
@boreq boreq added the enhancement New feature or request label May 18, 2022
@boreq boreq moved this from In Progress to Todo in Verse: Nos Projects May 23, 2022
@boreq boreq moved this from Todo to In Progress in Verse: Nos Projects Jun 13, 2022
@boreq
Copy link
Contributor Author

boreq commented Jun 13, 2022

Related to planetary-social/planetary-ios#628.

@boreq boreq mentioned this issue Jun 20, 2022
Merged
@boreq boreq moved this from In Progress to Todo in Verse: Nos Projects Jun 21, 2022
@mplorentz mplorentz moved this from Todo to In Progress in Verse: Nos Projects Jun 27, 2022
@boreq boreq moved this from In Progress to Todo in Verse: Nos Projects Jul 13, 2022
@mplorentz mplorentz moved this from Todo to Backlog in Verse: Nos Projects Jul 18, 2022
@boreq boreq moved this from Backlog to Todo in Verse: Nos Projects Oct 24, 2022
@boreq boreq closed this as completed Oct 24, 2022
Repository owner moved this from Todo to Done in Verse: Nos Projects Oct 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request priority/high
Projects
Archived in project
Development

No branches or pull requests

1 participant