Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: access-api handles provider/add invocations #462

Merged
merged 30 commits into from
Mar 4, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
1c5d196
add voucher/claim test that invokes the delegation in email
gobengo Feb 28, 2023
679e3c2
fix lint
gobengo Feb 28, 2023
b8849a4
access/authorize confirmation stores ucan/attest not ./update
gobengo Feb 28, 2023
d15d0db
add provider add capability
gobengo Feb 28, 2023
6da1479
add provider/add capability parser
gobengo Feb 28, 2023
25c4e0e
start createProviderAddHandler
gobengo Feb 28, 2023
da6adb6
start testing providerAdd handler
gobengo Feb 28, 2023
2f64d27
test provider/add as did:mailto
gobengo Feb 28, 2023
624b355
lint
gobengo Feb 28, 2023
bd05cbd
typo fix
gobengo Feb 28, 2023
fbb0ce4
provider-add.test asserts storage provider using new testing/space-st…
gobengo Feb 28, 2023
27acf17
provider/add stores StorageProvision (in-memory only)
gobengo Mar 1, 2023
0c62dbc
fix caps test after provider/add nb.consumer parser change
gobengo Mar 1, 2023
e537211
reenable important assertion even tho failing
gobengo Mar 1, 2023
d9768ad
StorageProvision backed by kysely
gobengo Mar 2, 2023
6c1d1be
Merge branch 'main' into 459-register-space-provider-add
gobengo Mar 3, 2023
49004e8
Merge branch 'main' into 459-register-space-provider-add
gobengo Mar 3, 2023
e403bb3
Merge branch 'main' into 459-register-space-provider-add
gobengo Mar 3, 2023
237d188
fix capabilities test
gobengo Mar 3, 2023
e62faaa
rename StorageProvisions model to just Provisions
gobengo Mar 3, 2023
2e90305
mv HelperTestContext interface into new access-api/test/helpers/types…
gobengo Mar 3, 2023
cd7dc57
fix: tests for provider add (#477)
Gozala Mar 3, 2023
184c55a
add comment to id col of migration 0006
gobengo Mar 3, 2023
754463d
rename issuer to sponsor in provisions table
gobengo Mar 3, 2023
4a48928
Merge branch 'main' into 459-register-space-provider-add
gobengo Mar 3, 2023
a199c11
provider capabilities dont use derive
gobengo Mar 3, 2023
6903891
remove provider/add ./update test that wasnt useful
gobengo Mar 4, 2023
40a4f16
provisions table has cid column as pk
gobengo Mar 4, 2023
c1e37f9
rename StoreProvisionCreation per review
gobengo Mar 4, 2023
9b9a14d
rename Provisions ProvisionsStorage
gobengo Mar 4, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
-- Migration number: 0006 2023-03-02T16:40:04.407Z

/*
goal: add a table to keep track of the storage provider for a space.
to enable https://github.com/web3-storage/w3protocol/issues/459
*/

CREATE TABLE
-- provision: the action of providing or supplying something for use
-- use case: representing the registration of a storage provider to a space
IF NOT EXISTS provisions (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: In the past when I had to do more DB work using plural names for tables was considered a bad practice, stack overflow still seems to think the singular is a the way to go with bunch of reasons there.

I'm not going to get upset if we use plural, but might be worth considering singular names instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in general I don't mind singular at all, and maybe even slightly prefer it, but here I made it plural because delegations accounts spaces already was.

I don't have any objection to batch changing them to singular in one big other PR

-- cid of invocation that created this provision
cid TEXT NOT NULL PRIMARY KEY,
-- DID of the actor that is consuming the provider. e.g. a space DID
consumer TEXT NOT NULL,
-- DID of the provider e.g. a storage provider
provider TEXT NOT NULL,
-- DID of the actor that authorized this provision
sponsor TEXT NOT NULL,
inserted_at TEXT NOT NULL DEFAULT (strftime ('%Y-%m-%dT%H:%M:%fZ', 'now')),
updated_at TEXT NOT NULL DEFAULT (strftime ('%Y-%m-%dT%H:%M:%fZ', 'now'))
);
1 change: 1 addition & 0 deletions packages/access-api/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
"@web3-storage/worker-utils": "0.4.3-dev",
"kysely": "^0.23.4",
"kysely-d1": "^0.1.0",
"multiformats": "^11.0.1",
"p-retry": "^5.1.2",
"preact": "^10.11.3",
"preact-render-to-string": "^5.2.6",
Expand Down
6 changes: 4 additions & 2 deletions packages/access-api/src/bindings.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { loadConfig } from './config.js'
import { ConnectionView, Signer as EdSigner } from '@ucanto/principal/ed25519'
import { Accounts } from './models/accounts.js'
import { DelegationsStorage as Delegations } from './types/delegations.js'
import { ProvisionsStorage } from './types/provisions.js'

export {}

Expand Down Expand Up @@ -63,10 +64,11 @@ export interface RouteContext {
url: URL
email: Email
models: {
spaces: Spaces
validations: Validations
accounts: Accounts
delegations: Delegations
spaces: Spaces
provisions: ProvisionsStorage
validations: Validations
}
uploadApi: ConnectionView
}
Expand Down
114 changes: 114 additions & 0 deletions packages/access-api/src/models/provisions.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
/* eslint-disable no-void */

/**
* @typedef {import("../types/provisions").ProvisionsStorage} Provisions
*/

/**
* @param {Array<import("../types/provisions").Provision>} storage
* @returns {Provisions}
*/
export function createProvisions(storage = []) {
/** @type {Provisions['hasStorageProvider']} */
const hasStorageProvider = async (consumerId) => {
const hasRowWithSpace = storage.some(({ space }) => space === consumerId)
return hasRowWithSpace
}
/** @type {Provisions['putMany']} */
const putMany = async (...items) => {
storage.push(...items)
}
/** @type {Provisions['count']} */
const count = async () => {
return BigInt(storage.length)
}
return {
count,
putMany,
hasStorageProvider,
}
}

/**
* @typedef ProvsionsRow
* @property {string} cid
* @property {string} consumer
* @property {string} provider
* @property {string} sponsor - did of actor who authorized for this provision
*/

/**
* @typedef {import("../types/database").Database<{ provisions: ProvsionsRow }>} ProvisionsDatabase
*/

/**
* Provisions backed by a kyseli database (e.g. sqlite or cloudflare d1)
*/
export class DbProvisions {
/** @type {ProvisionsDatabase} */
#db

/**
* @param {ProvisionsDatabase} db
*/
constructor(db) {
this.#db = db
this.tableNames = {
provisions: /** @type {const} */ ('provisions'),
}
void (/** @type {Provisions} */ (this))
}

/** @type {Provisions['count']} */
async count(...items) {
const { size } = await this.#db
.selectFrom(this.tableNames.provisions)
.select((e) => e.fn.count('provider').as('size'))
.executeTakeFirstOrThrow()
return BigInt(size)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is it otherwise a string ? Can you plz add a comment explaining why this needs to be a BigInt.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made it a BigInt to match the interface. I used bigint in the interface for consistency with the count() method on delegations, which I made return a BigInt per @alanshaw review. #427 (comment)

}

/** @type {Provisions['putMany']} */
async putMany(...items) {
if (items.length === 0) {
return
}
/** @type {ProvsionsRow[]} */
const rows = items.map((item) => {
return {
cid: item.invocation.cid.toString(),
consumer: item.space,
provider: item.provider,
sponsor: item.account,
}
})
await this.#db
.insertInto(this.tableNames.provisions)
.values(rows)
.executeTakeFirstOrThrow()
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be nice to have method that gives you providers you have on space in shape of { [provider]: [subscriber, ...subscriber[]] }. Which is more or less what I was asking to expose from the space/info.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#480 (comment)

By subscriber I assume you mean the account DIDs?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By subscriber I assume you mean the account DIDs?

Yes

/** @type {Provisions['hasStorageProvider']} */
async hasStorageProvider(consumerDid) {
const { provisions } = this.tableNames
const { size } = await this.#db
.selectFrom(provisions)
.select((e) => e.fn.count('provider').as('size'))
.where(`${provisions}.consumer`, '=', consumerDid)
.executeTakeFirstOrThrow()
return size > 0
}

/**
* @param {import("@ucanto/interface").DID<'key'>} consumer
*/
async findForConsumer(consumer) {
const { provisions } = this.tableNames
const rows = await this.#db
.selectFrom(provisions)
.selectAll()
.where(`${provisions}.consumer`, '=', consumer.toString())
.execute()
return rows
}
}
35 changes: 32 additions & 3 deletions packages/access-api/src/service/index.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import * as ucanto from '@ucanto/core'
import * as Ucanto from '@ucanto/interface'
import * as Server from '@ucanto/server'
import { Failure } from '@ucanto/server'
import * as Space from '@web3-storage/capabilities/space'
Expand All @@ -13,6 +14,7 @@ import * as uploadApi from './upload-api-proxy.js'
import { accessAuthorizeProvider } from './access-authorize.js'
import { accessDelegateProvider } from './access-delegate.js'
import { accessClaimProvider } from './access-claim.js'
import { providerAddProvider } from './provider-add.js'

/**
* @param {import('../bindings').RouteContext} ctx
Expand All @@ -22,6 +24,12 @@ import { accessClaimProvider } from './access-claim.js'
* }
*/
export function service(ctx) {
/**
* @param {Ucanto.DID<'key'>} uri
*/
const hasStorageProvider = async (uri) => {
return Boolean(await ctx.models.spaces.get(uri))
}
return {
store: uploadApi.createStoreProxy(ctx),
upload: uploadApi.createUploadProxy(ctx),
Expand All @@ -45,12 +53,21 @@ export function service(ctx) {
}
return accessDelegateProvider({
delegations: ctx.models.delegations,
hasStorageProvider: async (uri) => {
return Boolean(await ctx.models.spaces.get(uri))
},
hasStorageProvider,
})(...args)
},
},

provider: {
add: (...args) => {
// disable until hardened in test/staging
if (ctx.config.ENV === 'production') {
throw new Error(`provider/add invocation handling is not enabled`)
}
return providerAddProvider(ctx)(...args)
},
},

voucher: {
claim: voucherClaimProvider(ctx),
redeem: voucherRedeemProvider(ctx),
Expand Down Expand Up @@ -181,6 +198,18 @@ export function service(ctx) {
fail() {
throw new Error('test fail')
},
/**
* @param {Ucanto.Invocation<Ucanto.Capability<'testing/space-storage', Ucanto.DID<'key'>, Ucanto.Failure>>} invocation
*/
'space-storage': async (invocation) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just amend space/info and add providers: DID[] to it ? We would need that for store/* capabilities anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah. I thought about it but didn't want to extend the public api without discussion.

In this case I am very supportive because that will also be helpful to upload-api and things like storacha/w3infra#134 (comment)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can I do it as a fast follow once I have verified that the space registration via provider/add works on staging at all?
#480

const spaceId = invocation.capabilities[0].with
const hasStorageProvider =
await ctx.models.provisions.hasStorageProvider(spaceId)
return {
hasStorageProvider,
foo: 'ben',
}
},
},
}
}
59 changes: 59 additions & 0 deletions packages/access-api/src/service/provider-add.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
import * as Ucanto from '@ucanto/interface'
import * as Server from '@ucanto/server'
import { Provider } from '@web3-storage/capabilities'
import * as validator from '@ucanto/validator'

/**
* @typedef {import('@web3-storage/capabilities/types').ProviderAdd} ProviderAdd
* @typedef {import('@web3-storage/capabilities/types').ProviderAddSuccess} ProviderAddSuccess
* @typedef {import('@web3-storage/capabilities/types').ProviderAddFailure} ProviderAddFailure
*/

/**
* @callback ProviderAddHandler
* @param {Ucanto.Invocation<import('@web3-storage/capabilities/types').ProviderAdd>} invocation
* @returns {Promise<Ucanto.Result<ProviderAddSuccess, ProviderAddFailure>>}
*/

/**
* @param {object} options
* @param {import('../types/provisions').ProvisionsStorage} options.provisions
* @returns {ProviderAddHandler}
*/
export function createProviderAddHandler(options) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: This obviously follows the style of code that is in the repo already, but I do find amount of indirection adds so much unnecessary complexity e.g. I would much rather have something like:

/**
 * @param {object} input
 * @param {Ucanto.Invocation<import('@web3-storage/capabilities/types').ProviderAdd>} input.invocation
 * @param {{ storageProvisions: import('../types/provisions').StorageProvisions }} input.context
 */
export const add ({ invocation, context: { storageProvisions } }) => {
   // ...
}

And in the service defs something like

Server.provide(Provider.add, ({ invocation }) => add({ invocation, context })

Than all this layers and closures all over.

Also for what it's worth I'm going to add context to the ucanto provider handles to remove the need for all those closures.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a good idea and I'll try it out across all the relevant things I wrote in that style to use the less-closures style you suggest. #481

/** @type {ProviderAddHandler} */
return async (invocation) => {
const [providerAddCap] = invocation.capabilities
const {
nb: { consumer, provider },
with: accountDID,
} = providerAddCap
if (!validator.DID.match({ method: 'mailto' }).is(accountDID)) {
return {
error: true,
name: 'Unauthorized',
message: 'Issuer must be a mailto DID',
}
}
await options.provisions.putMany({
invocation,
space: consumer,
provider,
account: accountDID,
})
return {}
}
}

/**
* @param {object} ctx
* @param {Pick<import('../bindings').RouteContext['models'], 'provisions'>} ctx.models
*/
export function providerAddProvider(ctx) {
return Server.provide(Provider.add, async ({ invocation }) => {
const handler = createProviderAddHandler({
provisions: ctx.models.provisions,
})
return handler(/** @type {Ucanto.Invocation<ProviderAdd>} */ (invocation))
})
}
32 changes: 32 additions & 0 deletions packages/access-api/src/types/provisions.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
import * as Ucanto from '@ucanto/interface'
import { ProviderAdd } from '@web3-storage/capabilities/src/types'

export type AlphaStorageProvider = 'did:web:web3.storage:providers:w3up-alpha'

/**
* action which results in provisionment of a space consuming a storage provider
*/
export interface Provision {
invocation: Ucanto.Invocation<ProviderAdd>
space: Ucanto.DID<'key'>
account: Ucanto.DID<'mailto'>
provider: AlphaStorageProvider
}

/**
* stores instances of a storage provider being consumed by a consumer
*/
export interface ProvisionsStorage {
hasStorageProvider: (consumer: Ucanto.DID<'key'>) => Promise<boolean>
/**
* write several items into storage
*
* @param items - provisions to store
*/
putMany: (...items: Provision[]) => Promise<void>

/**
* get number of stored items
*/
count: () => Promise<bigint>
}
2 changes: 2 additions & 0 deletions packages/access-api/src/utils/context.js
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import {
delegationsTableBytesToArrayBuffer,
} from '../models/delegations.js'
import { createD1Database } from './d1.js'
import { DbProvisions } from '../models/provisions.js'

/**
* Obtains a route context object.
Expand Down Expand Up @@ -77,6 +78,7 @@ export function getContext(request, env, ctx) {
spaces: new Spaces(config.DB),
validations: new Validations(config.VALIDATIONS),
accounts: new Accounts(config.DB),
provisions: new DbProvisions(createD1Database(config.DB)),
},
email,
uploadApi: createUploadApiConnection({
Expand Down
10 changes: 8 additions & 2 deletions packages/access-api/src/utils/email.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
export const debug = () => new DebugEmail()

/**
* @typedef ValidationEmailSend
* @property {string} to
* @property {string} url
*/

/**
* @param {{token:string, sender?:string}} opts
*/
Expand All @@ -23,7 +29,7 @@ export class Email {
/**
* Send validation email with ucan to register
*
* @param {{ to: string; url: string }} opts
* @param {ValidationEmailSend} opts
*/
async sendValidation(opts) {
const rsp = await fetch('https://api.postmarkapp.com/email/withTemplate', {
Expand Down Expand Up @@ -90,7 +96,7 @@ export class DebugEmail {
/**
* Send validation email with ucan to register
*
* @param {{ to: string; url: string }} opts
* @param {ValidationEmailSend} opts
*/
async sendValidation(opts) {
try {
Expand Down
Loading