Skip to content
This repository has been archived by the owner on Feb 8, 2023. It is now read-only.

Sprint Nov 23 #56

Closed
24 of 42 tasks
daviddias opened this issue Nov 23, 2015 · 24 comments
Closed
24 of 42 tasks

Sprint Nov 23 #56

daviddias opened this issue Nov 23, 2015 · 24 comments

Comments

@daviddias
Copy link
Member

Sprint Nov 23

Sprint Goals

Sprint Discussions

Schedule

Please take notes in a separate pad, if you can, and link it here.

Endeavour Lead Pad Permanent Link
Apps on IPFS Juan Benet and Friedel Ziegelmayer https://public.etherpad-mozilla.org/p/nov-23-apps-on-ipfs
infrastructure Lars Gierth https://public.etherpad-mozilla.org/p/nov-23-infrastructure
libp2p David Dias https://public.etherpad-mozilla.org/p/nov-23-libp2p0-and-js-ipfs #56 (comment)
js-ipfs David Dias https://public.etherpad-mozilla.org/p/nov-23-libp2p0-and-js-ipfs #56 (comment)
go-ipfs Jeromy Johnson https://public.etherpad-mozilla.org/p/nov-23-go-ipfs
testing + ci Jeromy Johnson https://public.etherpad-mozilla.org/p/nov-23-testing-ci
protocol + specs Juan Benet https://public.etherpad-mozilla.org/p/nov-23-protocol-specs
bitswap ml Juan Benet https://public.etherpad-mozilla.org/p/nov-23-bitswap-ml
Data Structures Juan Benet https://public.etherpad-mozilla.org/p/nov-23-data-structures

Please add the Agenda to the Pad before the endeavour sprint starts.

Sprint Deliverables

  • Add yourself below, that way we only have people listed who are really in the sprint
  • Add links to issues down here. Only add things you can finish this sprint.

[<% name %>](@<% your_GitHub_handle %)

David

@lgierth
@dignifiedquire
Copy link
Member

IPFS Apps

Lead: Friedel Ziegelmayer

Agenda

IPFS (HTTP) API (SPEC)

Participants

Notes

IPFS (HTTP) API (SPEC)

Questions

@dignifiedquire: Who has implemented the original version? Is thera short breakdown on how this came to be?

@jbenet: The goal for the http API was for it to be able to be layered over any transport. Some people may want to do non-http transports, like UNIX main sockets. Sometimes yuo don't even have HTTP over the wire. Another example woudl be websockets; ie, piping all requests through amux over the daemon, etc. The goal was to design an API that was simply patterened so that it could be layered. We just implemented the HTTP one first. The commands library supports reading and writing with JSON on HTTP and also over the CL. That means all of the regular ways of specifying commands and arguments / options need to be have a ways to be transported to the command in question. So, flags, pathing, query strings, etc.

@dignifiedquire: Ok. There are some main issues. Richard, I know you've been trying to get some actual stuff done using http commands. We also have the js-ipfs-api docs, because that is kind of a documentation.
So, today, how does the API as it is now implemented? Ie, returning JSON, chunks, newline-delimitered, etc.

@jbenet: Yeah, we don't even have a thing where that is documented. Another thing is that the core API - what API means - also needs to be defined. API could be a process around the daemon, or actually an API for the libraries. What would be good would be to have the core as close as possible to the daemon, so that you coudl follow the same for all. We're a little bit far from that, but we're close.

@dignifiedquire: So the goal wouldbe to have a spec for the API commands and then a version of that which would structure how the API works.

@dignifiedquire: @diasdavid said earlier that it's not just about documenting, but we also need a Spec that the implementation follows. Ideally, a spec that we could generate a test feed from would be good. Rather than having to guess and think and hope, which is the current scope.

@jbenet: Are there any good versions of that?

@dignifiedquire: That's why I've been looking at apiary. They have two way support. If you write a spec in this format, there are tools for generating mock servers.

@jbenet: One constraint is that the spec will have to be more general than the HTTP. It should be idiomatic, but that should be optional.
@RichardLitt: We need to make sure that the main source of truth is the HTTP spec, so that each implementation should talk to that.

@jbenet: one possibility is to make a spec for the core API, and then just go from there.
...<@RichardLitt was talking>

@dignifiedquire: So, HTTP is just a subset of what we need. So, we should have a high level, and then use tooling to convert it into Swagger, or whatever.

@jbenet: https://github.com/ipfs/specs/blob/master/modules/modules.go. This shows off how go might be used as an interface langauge.

@jbenet: another approach is enumerate all of the functions in a file, and then enum. all of the ways that interfacing components may do that. We have the ability to: positional arguments, flags, mention files, and that's when we start hitting things which are less defined. When do we need streaming? When do we need APIs such that we are going to be able to stream the output out, like with huge dirs where you need to stream the output as you stream the input.
@dignifiedquire: yeah, we'll need that.

@jbenet: Goals!

  • Enumerate all of the functions that already exist
  • Enumerate all of the things that the functions need to do: flags, I/O

(ipfs-inactive/http-api-spec#4)

From that, we can figure out which language is best. (Go may not be best, because of streaming interfaces.). From there:

  • Pick one of these languages and go from there.

So, start with the core, and then go from there. init, for instance, isn't in the core. But maybe it should be. Maybe there's an argument for one api - client API, and library apis.

@dignifiedquire: I think it would be good to have them all in the same spec. Let's just make a huge list of markdown in ipfs/api right now. Also,

Talk in the https://github.com/ipfs/api/issues please!

Possible Solutions

Would be interesting if it was RESTful rather than RPC, but probably not worth the effort to design a whole new API. – ion

Thanks everyone and sorry @diasdavid you couldn't be here 😛

@whyrusleeping
Copy link
Member

go-ipfs

Lead: @whyrusleeping

Agenda

  • commands lib
    • augment argument types (path)
    • cleanup in http client
  • close-notify problems
    • try to get minimal test case
    • ask bradfitz? (last resort)
  • no-sync flag for flatfs
    • may help s3-fuse-thing for kyle
  • private networks:
    • implement shared-key swarms
  • pubsub next steps:
    • scour pub/sub + multicast literature
    • list out links to papers + implementations
    • pick a couple implementations we like
    • figure out hierarchical channels
    • figure out channel-joining protocol
    • figure out authenticated (signed) and encrypted channel
  • chunking + dag layouts
    • move things out into own repo first
    • many real workload dataset benchmarks
    • generate stats + graphs to pick algos
    • maybe: heuristic to pick algos eventually
    • come up with a naming convention for them
    • eventually, a heuristic
    • new defaults in 0.4.0?

Participants

potentially interested:

@daviddias
Copy link
Member Author

libp2p and js-ipfs

Lead: David Dias

Agenda

You have 30 minutes for this agenda, 5 minutes before the meeting ends, consider moving the remaining items to a github discussion thread so that all the other sprint meetings can start in time.

Participants (write your name on the list)

Notes

@jbenet: I pulled the libp2p slides out of my qconf talk
https://ipfs.io/ipfs/QmRthja4NnZEwXcUNgTKpX86dtGMQxstMbhgnTzmVhrW4D/libp2p-part/ipfs-018.libp2p-part.pdf

  • announce what the end UX is (or what we expect it to be) and get feedback
  • look at network simulators

should this be in the agenda, or is it for the github issue:
- which bits are already stable enough for other teams like py-ipfs to start on?
- which bits are still in flux so we (py-ipfs team) should approach with caution?
- which bits are out of sync with the spec so we (py-ipfs) can help write spec as we implement?

js-ipfs

js-ipfs-api

@RichardLitt
Copy link
Member

My Todos:

  • Enumerate all of the functions that already exist, in ipfs/api
  • Suggest community infrastructure for meetups
  • Make the documentation around Commit Sign-Offs more friendly
  • Add a PR to ipfs.io/docs/api about how the commands there are not all-inclusive.
  • Sprint management.
  • Remove video links from the readme. Remove video links from the readme  #57

@dignifiedquire
Copy link
Member

TODO(@dignifiedquire)

  • Spec all the APIs

I am not allowed to commit any code unrelated to spec writing and tooling around that.

@harlantwood
Copy link
Contributor

Testing + CI

Lead: @jbenet??? anyobdy else?

Agenda

  • ci dashboard
  • go-ipfs heisentests / unreliable test suite

Participants

@harlantwood, @lgierth, @jbenet

Notes

ci-status

Random topics

  • @lgierth will open a pathway to access the git hash that is deployed on an IPFS gateway

@harlantwood
Copy link
Contributor

@harlantwood

@jbenet
Copy link
Member

jbenet commented Nov 24, 2015

Frozen from: https://public.etherpad-mozilla.org/p/nov-23-data-structures

Data Structures

Lead: @jbenet

Agenda

  • short intro of POST
  • go through inidividual use cases to align perspectives
  • @fazo96 ipfs-boards
  • @nginnever swarmchan
  • https://activestrea.ms
  • other "questions" and datastructures
  • POST message format
  • Extension specs
  • Come up with strategy + todos

Participants

Notes

Does a POST have some sort of application ID (if there are multiple versions of an application)?

@fazo96 -- ipfs-boards:

  • https://github.com/fazo96/ipfs-boards
  • all the aggregation is done on the client
  • use ipns for all the heads
  • personal filters + config per "board"
  • similar to haadcode's
  • define things in markdown files
  • api over ipfs/api to easily grab the data
  • biggest challenge:
    • aggregation-- boards with 100s of users, speed - CRDT magic!
    • how do i get the posts? the latest? the most interesting?
    • biggest issue: performance.
      • pub/sub would help
      • ipns is slow-- make it faster (pub/sub)
    • privacy concerns
    • DOSing the network

@nginnever -- SwarmChan:

Keychain:

  • digital signatures of content
  • timestamping into secure time chains
  • UI will be paramount

Other Questions -- a set of other datastructures we can parametrize or stub-out. don't need to solve them in POST.

  • Keychain
  • Identity
  • blockchain-related (persona in ethereum)
  • http://weboftrust.info (+ DPKI)
  • Sharing
  • Groups
  • Capabilities (privacy)
  • Common Formats
  • Time + Secure Time
  • Timestamping
  • Notifications
  • Forks/branches of the conversation

Important requirements:

  • felxible formatting (body various fmts)
  • Search:
    • encourage indexing + post-processing that generates
      more data that links back to original data.
    • various other index trees/structs

IPLD

Large-scale processing

  • for search and other systems
  • Centralized processing WITHOUT Central Control
  • sub-networks like bitcoin, where users can participate
  • still "centralized", but without control over the data

yacy

Things to do:

  • outline all the abstractions (posts, boards, users, groups, ...)
  • create POST repo and open a issue for each of the Questions
  • figure out all the specs out there to review (W3C, WebDAV, XMPP, matrix.org, ..., etc)

Other organizations to involve

TODOs:

  • start core + extension spec skeletons:
    • core spec (@jbenet, @mekarpeles)
    • bb/forum ext spec (@fazo96)
    • blog ext spec
    • todo (@mekarpeles)
    • chat ext spec
    • annotation ext spec
    • micro-blog ext spec
    • status updates
    • ToDos
    • Calendar Event/Notification/Invitation
  • modalities
    • like / favorite
    • republishing / retweet / reblog
    • replying
    • embedding attachments
    • cite
    • quote
  • Note: store things like mentions, citations, etc as IDs so they can be fetched and ranked

stay close to reality: use the apps already being built to play with datastructs and make sure these things are actually good + fun to program with! if not, wont work.

  • (see notes from other doc)
  • References: map of merkle-links
  • link w/ (opt) type [cite | mentions | response | edit]
  • OpenAnnotations? See IIIF's spec

This was referenced Nov 24, 2015
Open
Closed
@masylum
Copy link

masylum commented Nov 24, 2015

TODO @masylum

  • Keep working on js-repo

@rht
Copy link

rht commented Nov 24, 2015

  • Fix go-ipfs mem leaks so that only 1 C-c is required to shutdown the daemon

@whyrusleeping
Copy link
Member

TODO:

  • ipfs-update 🚢
  • fs-repo-migrations (needs CR to merge)
  • commands lib path argument type
  • write layout and chunking algo to root block of created dags.
  • say "pub sub" out loud three times
  • eat turkey

@jbenet
Copy link
Member

jbenet commented Nov 25, 2015

@jbenet:

  • CR ipfs-update
  • CR fs-repo-migrations
  • CR @rht's stuff (sorry, my CR is lagging)
  • start draft of POST spec
  • list some pub/sub papers/specs

I'll add some more later today/tomorrow.

@daviddias
Copy link
Member Author

If you feel your task list isn't puffed enough, you can always add this one:

:D

@daviddias
Copy link
Member Author

On the continuous endeavour to improve the libp2p spec, I've summited some PR and categorised issues on the ipfs/specs repo, which would be great if people interested could review :) https://github.com/ipfs/specs/labels/libp2p

@jbenet
Copy link
Member

jbenet commented Nov 30, 2015

@jbenet's update

Hey everyone, not sure i can make irc sync this morning. Am pretty sick, and still recovering. I'll try to-- but stayed up late catching up with some stuff. (this week was hosed for me-- between family time, getting sick, and org todos, i didnt have much time for my dev todos)

  • out with family for thanksgiving 🍗
  • got very sick :sick: -- out for a few days

@dignifiedquire
Copy link
Member

@jbenet get well soon 😷

@dignifiedquire
Copy link
Member

@dignifiedquire update

@daviddias
Copy link
Member Author

@jbenet get better soon!

SPRINT UPDATE

@RichardLitt
Copy link
Member

Officially this week:

Extra

@whyrusleeping
Copy link
Member

  • ipfs-update 🚢
  • fs-repo-migrations (needs CR to merge)
  • commands lib path argument type
  • write layout and chunking algo to root block of created dags.
  • say "pub sub" out loud three times
  • eat turkey
  • provide-many mechanism, yay bandwidth savings!

@ghost
Copy link

ghost commented Nov 30, 2015

My week was kinda dominated by the NPM import (a.k.a. registry-mirror) and its performance issues. On the other hand got a couple of smaller things done. Also got p2p/discovery/cjdns.go to work, (it's lovely,) just need to add tests.

@ghost
Copy link

ghost commented Nov 30, 2015

Sorry for my lack of presence :( not cool to be absent without any notice

@rht
Copy link

rht commented Dec 1, 2015

Merging ipfs/kubo#1964 & ipfs/kubo#1973 are needed to unclog distributed archival of content larger than 50 GB (e.g. wikipedia), otherwise it takes ages for those who attempted.

@RichardLitt
Copy link
Member

thanks @lgierth: no worries. :)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@masylum @harlantwood @jbenet @rht @dignifiedquire @RichardLitt @daviddias @whyrusleeping