Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updated API spec #1697

Closed
wants to merge 1 commit into from
Closed

updated API spec #1697

wants to merge 1 commit into from

Conversation

b5
Copy link
Member

@b5 b5 commented Mar 9, 2021

Now that we have/soon will have a bunch of really big refactoring under our belts, I think we should be doing a massive overhaul of our API endpoint routing before cutting a release. This PR proposes a bunch of spec changes, while also aiming to explicitly document the connection between API endpoints & lib methods by way of the operationId field in the open api spec.

If we get this right & fill in the doc we'll leave with spec, accurate documentation, and punchiest for getting our API into release-ready shape.

Lots of work to do here to fill in possible responses & query params based on input param structs. I think we should start with a high-level audit of the proposed endpoint changes

@b5 b5 added API JSON API issues Breaking Change things that change existing API contracts. spec issues & changes that affect specification labels Mar 9, 2021
@b5 b5 self-assigned this Mar 9, 2021
@b5 b5 requested review from Arqu, dustmop and ramfox March 9, 2021 20:42
using operationId & http verbs to draw an explicit mapping between lib methods
and our HTTP api, with proposed changes to endpoints
@b5
Copy link
Member Author

b5 commented Mar 16, 2021

I think it's easier to look at this as a table. Here's a first unfinished pass. @ramfox I'd love it if you could take over

endpoint HTTP methods Return Type Lib Method Name
"/" Health?
"/health" Health?
"/qfs/ipfs/{path:.*}" *
"/qfs/s3/{path:.*}" *
"/webui" HTML
"/list" GET []VersionInfo
"/list/{peer}" GET []VersionInfo
"/sql" POST [][]any
"/diff" GET,POST Diff diff.diff
"/changes" GET,POST ChangeReport
"/access/token" POST JSON Web Token
Dataset Endpoints
"/ds/apply/{path:.*}" POST Dataset transform.apply
"/ds/componentstatus/{path:.*}" GET []Status dataset.componentstatus
"/ds/get/{path:.*}" GET Dataset
"/ds/log/{path:.*}" GET []VersionInfo
"/ds/rename" POST DSRef
"/ds/save" POST DSRef
"/ds/pull/{path:.*}" POST DSRef
"/ds/push/{path:.*}" POST DSRef
"/ds/render/{path:.*}" GET,POST []byte
"/ds/remove" POST,DELETE DSRef
"/ds/unpack/{path:.*}" POST Dataset
P2P Endpoints
"/p2p" GET,POST,DELETE Peer
"/p2p/peer/{path:.*}" GET Peer
"/p2p/peer/{path:.*}" GET Peer
"/p2p/peer/connect/{path:.}" POST,DELETE Peer
"/p2p/peer/connections" GET []Peer
"/profile/me"
"/profile/me/photo"
"/profile/me/poster"
"/remote/feeds"
"/remote/preview/{path:.*}"
"/remote/registry/profile/new"
"/remote/registry/profile/prove"
"/remote/search"
"/remote/dsync"
"/remote/logsync"
"/remote/refs"
Working Driectory Endpoints
"/wd/status/{path:.*}" GET
"/wd/init/{path:.*}" POST
"/wd/checkout/{path:.*}" POST
"/wd/restore/{path:.*}" POST
"/wd/write/{path:.*}" POST

@ramfox
Copy link
Member

ramfox commented Mar 16, 2021

Got it!

Going to do a pass specifically on aligning/finishing out the current table.

b5 and I spoke about an additional round of work that goes with this in parallel: documenting/examining the params of each endpoint
@dustmop or @Arqu who wants to be the lucky winner? Deadline on Friday when we are having a discussion about the API spec.

@b5 b5 assigned ramfox and unassigned b5 Mar 16, 2021
@b5
Copy link
Member Author

b5 commented Mar 17, 2021

Talked with @dustmop & he's agreed to take on an audit of params & param fields

@ramfox
Copy link
Member

ramfox commented Mar 19, 2021

endpoint HTTP methods Return Type Lib Method Name Notes
"/" GET Health? api.HealthCheckHandler
"/health" GET Health? api.HealthCheckHandler
"/qfs/ipfs/{path:.*}" GET * fs.Get
"/qfs/s3/{path:.*}" GET * fs.Get
"/webui" GET HTML api.WebuiHandler
"/list" GET []VersionInfo collection.List?
"/listraw" GET String collection.ListRaw?
"/list/{peer}" GET []VersionInfo collection.List?
"/sql" POST [][]any sql.Exec
"/diff" GET,POST Diff diff.Diff?
"/changes" GET,POST ChangeReport diff.Changes?
"/access/token" POST JSON Web Token access.Token?
Dataset Endpoints
"/ds/apply/{path:.*}" POST Dataset transform.apply
"/ds/componentstatus/{path:.*}" GET []Status dataset.componentstatus
"/ds/get/{path:.*}" GET Dataset dataset.Get
"/ds/history/{path:.*} GET []VersionInfo dataset.History
"/ds/log/{path:.*}" GET []VersionInfo dataset.Log
"/ds/rename" POST DSRef? dataset.Rename
"/ds/save" POST DSRef dataset.Save
"/ds/pull/{path:.*}" POST DSRef dataset.Pull
"/ds/push/{path:.*}" POST DSRef dataset.Push
"/ds/render/{path:.*}" GET,POST []byte dataset.Render
"/ds/remove" POST,DELETE DSRef dataset.Remove
"/ds/unpack/{path:.*}" POST Dataset dataset.Unpack
"/ds/validate/{path:.*}" GET, POST ValidateRes dataset.Validate
"/ds/manifest/{path:.*}" POST Manifest dataset.Manifest
"/ds/manifestmissing/{path:.*}" POST Manifest dataset.ManifestMissing
"/ds/daginfo/{path:.*}" POST DagInfo dataset.DagInfo
P2P Endpoints
"/p2p" GET,POST,DELETE Profile
"/p2p/peer/{path:.*}" GET Profile peers.Info
"/p2p/peer/connect/{path:.}" POST,DELETE Profile peers.ConnectToPeer
"/p2p/peer/connections" GET []Profile peers.ConnectedQriProfiles
"/profile/me" GET Profile profile.GetProfile
"/profile/me/photo" GET JPEG ([]byte) profile.ProfilePhoto
"/profile/me/poster" GET JPEG ([]byte) profile.PosterPhoto
RemoteEndpoints
"/remote/feeds" GET Feed remote.Feeds
"/remote/preview/{path:.*}" GET Dataset remote.Preview
"/remote/registry/profile/new" POST Profile registry.CreateProfile
"/remote/registry/profile/prove" POST Profile registry.ProveProfile
"/remote/search" GET SearchResult remote.Search
"/remote/dsync" GET *1 dag.Info remote.DsyncHTTPHandler
"/remote/dsync" GET *2 []byte remote.DsyncHTTPHandler
"/remote/dsync" POST dag.Manifest remote.DsyncHTTPHandler
"/remote/dsync" PUT -- remote.DsyncHTTPHandler
"/remote/dsync" PATCH stream? remote.DsyncHTTPHandler
"/remote/logsync" GET -- remote.LogbookHTTPHandlery
"/remote/logsync" PUT []byte remote.LogbookHTTPHandlery
"/remote/logsync" DELETE -- remote.LogbookHTTPHandlery
"/remote/refs" GET DSRef remote.RefsHTTPHandler
"/remote/refs" DELETE -- remote.RefsHTTPHandler
Working Driectory Endpoints
"/wd/status/{path:.*}" GET []StatusItem fsi.Status
"/wd/init/{path:.*}" POST fsi.InitParams fsi.Init
"/wd/checkout/{path:.*}" POST fsi.LinkParams fsi.Checkout
"/wd/restore/{path:.*}" POST fsi.RestoreParams fsi.Restore
"/wd/write/{path:.*}" POST fsi.WriteParams fsi.Write
Left out:
"/peers" GET []Profiles peers.Peers lists profiles of all peers every seen
"/connections" GET []string peers.ConnectedIPFSPeers lists peerIDs of connected IPFS peers
"/logs" GET PlainLog log.PlainLogsHandler returns entire logbook as json
"/logbook/summary GET string log.LogbookSummary returns a string overview of a logbook
"/caninitdatasetworkdir/{path:.*}" GET fsi.InitParams fsi.CanInitDatasetWorkDir returns whether a dataset can be initalized at the given path
"/fsi/createlink/{path:.*}" POST VersionInfo fsi.CreateLink creates an fsi link
"/fsi/unlink/{path:.*}" POST string fsi.Unlink removes the fsi link
"/whatchanged/{path:.*}" GET StatusItem fsi.WhatChanged returns what changed for a specific commit

Notes

  • Profile is an alias for config.ProfilePod

  • the remote endpoints are just a recording of what exists, not my opinion on how it should be structured. These are going to be the most complicated to get across to the user & probably needs a review from b5 to ensure I've recorded everything correctly

    1. "/remote/dsync" GET call w/ manifestID params returns a dag.Info
    1. "/remote/dsync" GET call w/ blockID (and no manifestID) in params returns the contents of the requested block

@ramfox
Copy link
Member

ramfox commented Mar 19, 2021

Table after our api spec discussion 03/19/2021

endpoint HTTP methods Return Type Lib Method Name Notes
"/" GET Health? api.HealthCheckHandler
"/health" GET Health? api.HealthCheckHandler
"/qfs/ipfs/{path:.*}" GET * fs.Get
"/qfs/s3/{path:.*}" GET * fs.Get
"/webui" GET HTML api.WebuiHandler
Aggregate Endpoints
"/list" GET []VersionInfo collection.List?
"/list/{peer}" GET []VersionInfo collection.List?
"/sql" POST [][]any sql.Exec
"/diff" GET,POST Diff diff.Diff?
"/changes" GET,POST ChangeReport diff.Changes?
Access Endpoints
"/access/token" POST JSON Web Token access.Token
Dataset Endpoints
"/ds/apply/{path:.*}" POST Dataset transform.apply
"/ds/componentstatus/{path:.*}" GET []Status dataset.componentstatus
"/ds/get/{path:.*}" GET Dataset dataset.Get
"/ds/log/{path:.*}" GET []VersionInfo dataset.Log
"/ds/rename" POST DSRef? dataset.Rename
"/ds/save" POST DSRef dataset.Save
"/ds/pull/{path:.*}" POST DSRef dataset.Pull
"/ds/push/{path:.*}" POST DSRef dataset.Push
"/ds/render/{path:.*}" GET,POST []byte dataset.Render
"/ds/remove" POST,DELETE DSRef dataset.Remove
"/ds/validate/{path:.*}" GET, POST ValidateRes dataset.Validate
Peer Endpoints
"/peer" GET,POST,DELETE Profile
"/peer/{path:.*}" GET Profile peers.Info
"/peer/connect/{path:.}" POST,DELETE Profile peers.ConnectToPeer
"/peer/list" GET []Profile peers.Profiles filter to get "connected" peers
Profile Endpoints
"/profile/list" GET Profile profile.GetProfile
"/profile/me" GET Profile profile.GetProfile
"/profile/me/photo" GET JPEG ([]byte) profile.ProfilePhoto
"/profile/me/poster" GET JPEG ([]byte) profile.PosterPhoto
Remote Endpoints
"/remote/feeds" GET Feed remote.Feeds
"/remote/preview/{path:.*}" GET Dataset remote.Preview
"/remote/registry/profile/new" POST Profile registry.CreateProfile
"/remote/registry/profile/prove" POST Profile registry.ProveProfile
"/remote/search" GET SearchResult remote.Search
Sync Endpoints
"/sync/dag" GET *1 dag.Info remote.DsyncHTTPHandler
"/sync/dag" GET *2 []byte remote.DsyncHTTPHandler
"/sync/dag" POST dag.Manifest remote.DsyncHTTPHandler
"/sync/dag" PUT -- remote.DsyncHTTPHandler
"/sync/dag" PATCH stream? remote.DsyncHTTPHandler
"/sync/logs" GET -- remote.LogbookHTTPHandlery
"/sync/logs" PUT []byte remote.LogbookHTTPHandlery
"/sync/logs" DELETE -- remote.LogbookHTTPHandlery
"/sync/resolve" GET DSRef remote.RefsHTTPHandler
"/sync/resolve" DELETE -- remote.RefsHTTPHandler
Working Directory Endpoints
"/wd/status/{path:.*}" GET []StatusItem fsi.Status
"/wd/init/{path:.*}" GET? -- fsi.Init
"/wd/init/{path:.*}" POST DSRef fsi.Init
"/wd/checkout/{path:.*}" POST -- fsi.Checkout
"/wd/restore/{path:.*}" POST -- fsi.Restore
"/wd/write/{path:.*}" POST []StatusItem fsi.Write
Plumbing:
"/plumbing/logs" GET PlainLog log.PlainLogsHandler returns entire logbook as json
"/plumbing/logsummary" GET string log.LogbookSummary returns a string overview of a logbook
"/plumbing/listraw" GET String collection.ListRaw?
"/plumbing/wd/createlink/{path:.*}" POST VersionInfo fsi.CreateLink creates an fsi link
"/plumbing/wd/unlink/{path:.*}" POST string fsi.Unlink removes the fsi link
"/plumbing/ipfspeers" GET []string peers.ConnectedIPFSPeers lists peerIDs of connected IPFS peers
"/plumbing/ds/unpack/{path:.*}" POST Dataset dataset.Unpack
"/plumbing/ds/manifest/{path:.*}" POST Manifest dataset.Manifest
"/plumbing/ds/manifestmissing/{path:.*}" POST Manifest dataset.ManifestMissing
"/plumbing/ds/daginfo/{path:.*}" POST DagInfo dataset.DagInfo

Notes

  • Profile is an alias for config.ProfilePod

    1. "/remote/dsync" GET call w/ manifestID params returns a dag.Info
    1. "/remote/dsync" GET call w/ blockID (and no manifestID) in params returns the contents of the requested block
  • instead of adding endpoints that return the same type, add parameters/filters

  • boo @ endpoints that return different types

  • more discussion around aggregate endpoints & method prefix/nesting

@Arqu
Copy link
Contributor

Arqu commented Mar 26, 2021

Latest API list

Sugar

The purpose of the API pacakge is to expose the lib.RPC api and add syntatic sugar for mapping RESTful HTTP requests to lib method calls

endpoint HTTP methods Lib Method Name
"/" GET api.HealthCheckHandler
"/health" GET api.HealthCheckHandler
"/qfs/ipfs/{path:.*}" GET qfs.Get
"/qfs/s3/{path:.*}" GET qfs.Get
"/webui" GET api.WebuiHandler
/ds/get/{useranme}/{name} GET api.GetHandler
/ds/get/{useranme}/{name}/at/{path} GET api.GetHandler
/ds/get/{useranme}/{name}/at/{path}/{component} GET api.GetHandler
/ds/get/{useranme}/{name}/at/{path}/{component}.{encoding} GET api.GetHandler

RPC

The purpose of the lib pacakge is to expose a uniform interface for interacting with a qri instance

endpoint Return Type Lib Method Name
Aggregate Endpoints
"/list" []VersionInfo collection.List?
"/sql" [][]any sql.Exec
"/diff" Diff diff.Diff
"/changes" ChangeReport diff.Changes
Access Endpoints
"/access/token" JSON Web Token access.Token
Automation Endpoints
"/auto/apply" Dataset automation.apply
Dataset Endpoints
"/ds/componentstatus" []Status dataset.componentstatus
"/ds/get Dataset dataset.Get
"/ds/log" []VersionInfo dataset.Log
"/ds/rename" DSRef? dataset.Rename
"/ds/save" DSRef dataset.Save
"/ds/pull" DSRef dataset.Pull
"/ds/push" DSRef dataset.Push
"/ds/render" []byte dataset.Render
"/ds/remove" DSRef dataset.Remove
"/ds/validate" ValidateRes dataset.Validate
"/ds/unpack" Dataset dataset.Unpack
"/ds/manifest" Manifest dataset.Manifest
"/ds/manifestmissing" Manifest dataset.ManifestMissing
"/ds/daginfo" DagInfo dataset.DagInfo
Peer Endpoints
"/peer" Profile peers.Info
"/peer/remove" -- peers.Remove
"/peer/connect" Profile peers.ConnectToPeer
"/peer/disconnect" Profile peers.DisconnectFromPeer
"/peer/list" []Profile peers.Profiles
Profile Endpoints
"/profile/list" Profile profile.GetProfile
"/profile/me" Profile profile.GetProfile
"/profile/me/photo" JPEG ([]byte) profile.ProfilePhoto
"/profile/me/poster" JPEG ([]byte) profile.PosterPhoto
Remote Endpoints
"/remote/feeds" Feed remote.Feeds
"/remote/preview" Dataset remote.Preview
"/remote/registry/profile/new" Profile registry.CreateProfile
"/remote/registry/profile/prove" Profile registry.ProveProfile
"/remote/search" SearchResult remote.Search
Working Directory Endpoints
"/wd/status" []StatusItem fsi.Status
"/wd/caninitworkdir" -- fsi.CanInitworkdir
"/wd/init" DSRef fsi.Init
"/wd/checkout" -- fsi.Checkout
"/wd/restore" -- fsi.Restore
"/wd/write" []StatusItem fsi.Write
"/wd/createlink" VersionInfo fsi.CreateLink
"/wd/unlink" string fsi.Unlink

"Protocol Endpoints"

We're going to ignore this work until later, but eventually we'd like to group these under a "sync" noun:

Sync Endpoints
"/sync/dag" GET *1 dag.Info remote.DsyncHTTPHandler
"/sync/dag" GET *2 []byte remote.DsyncHTTPHandler
"/sync/dag" POST dag.Manifest remote.DsyncHTTPHandler
"/sync/dag" PUT -- remote.DsyncHTTPHandler
"/sync/dag" PATCH stream? remote.DsyncHTTPHandler
"/sync/logs" GET -- remote.LogbookHTTPHandlery
"/sync/logs" PUT []byte remote.LogbookHTTPHandlery
"/sync/logs" DELETE -- remote.LogbookHTTPHandlery
"/sync/resolve" GET DSRef remote.RefsHTTPHandler
"/sync/resolve" DELETE -- remote.RefsHTTPHandler

Examples

# RPC-style:
$ curl -X POST -d '{"ref": "b5/world_bank_population", "selector": "body", "page": 5, "offset": 60 }' -H 'Content-Type: application/json' http://localhost:2503/ds/get


# REST-style:
/ds/get/{username}/{name}
$ curl http://localhost:2503/ds/get/b5/world_bank_population
$ curl http://localhost:2503/ds/get/b5/world_bank_population/body.csv?all=true
$ curl -H "Accept: text/csv" http://localhost:2503/ds/get/b5/world_bank_population/body

From the API pacakge it should be a one-liner to surface the RPC api from a lib.Instance, then add RESTful routes atop the returned RPC mux:

package api

func (s *Server) Serve() error {
  mux, _ := s.inst.GiveAPIServer()
  AddGetHandlers(mux)
}

@ramfox
Copy link
Member

ramfox commented Mar 31, 2021

need to do another pass & will either use this pr or cannibalize a tangential issue, but here is some tracking and unanswered questions around the api refactor
qri_api.txt

@b5 b5 added this to the v0.10.0 milestone Apr 13, 2021
@b5
Copy link
Member Author

b5 commented Apr 20, 2021

ok, so the actual code from this PR is superseded by #1767, the discussion of the API has moved to #1731

@b5 b5 closed this Apr 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API JSON API issues Breaking Change things that change existing API contracts. spec issues & changes that affect specification
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants