Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: api waits for trigger filecoin pipeline from the client #1332

Merged
merged 2 commits into from
Mar 26, 2024
Merged

feat: api waits for trigger filecoin pipeline from the client #1332

merged 2 commits into from
Mar 26, 2024

Conversation

vasco-santos
Copy link
Contributor

@vasco-santos vasco-santos commented Mar 13, 2024 •
edited
Loading

This PR hooks up possibility to compute Piece CID when submit message is received.

For extra context, today w3infra sets skipFilecoinSubmitQueue on filecoin/offer from users, resulting filecoin/offer in a noop that simply links receipts for effects of filecoin/submit and filecoin/accept. This is due to the API to not care about what was submit by client as it is calculated in bucket event anyway. However, we want to change this behaviour to have filecoin pipeline only triggered on filecoin/offer, putting it in the submit queue where validation will now happen.

We can summarise these changes in two units:

Rollout plan:

  • w3infra swaps skipFilecoinSubmitQueue to False when releasing with this. For the old bucket we invoke filecoin/offer from bucket event, given current client does not do it

Follow ups:

  • filecoin/offer from service on old bucket event
  • hook up finding where data is with datastore
  • clean up skipFilecoinSubmitQueue option

@vasco-santos vasco-santos changed the title feat!: trigger filecoin pipeline from client feat!: api waits for trigger filecoin pipeline from the client Mar 13, 2024
@vasco-santos vasco-santos force-pushed the feat/trigger-filecoin-pipeline-from-client branch 5 times, most recently from 7b30bea to 9af56fd Compare March 15, 2024 16:43
@vasco-santos vasco-santos changed the title feat!: api waits for trigger filecoin pipeline from the client feat: api waits for trigger filecoin pipeline from the client Mar 15, 2024
@vasco-santos vasco-santos force-pushed the feat/trigger-filecoin-pipeline-from-client branch 2 times, most recently from 7ac7bbf to bfd8368 Compare March 18, 2024 08:17
@vasco-santos vasco-santos marked this pull request as ready for review March 18, 2024 08:19
@vasco-santos vasco-santos mentioned this pull request Mar 18, 2024
Merged
1 task
vasco-santos
vasco-santos commented Mar 18, 2024
View reviewed changes
packages/filecoin-api/src/storefront/events.js Outdated
// TODO: verify piece
// read and compute piece for content
// TODO: needs to be hooked with location claims
const contentGetRes = await context.dataStore.get(message.content)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here we actually need to decide this interface. Being a datastore would be nice to allow implementers to use R2/S3 client kind of things, which would decrease costs and enhance performance. However, probably we do not want the datastore to be hooked with the claims itself, so probably datastore should in the future receive claims for the content?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand - the datastore can use content claims under the hood no?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.get() implies to me it's retrieving the whole data, perhaps consider calling this .stream()? (similar to https://developer.mozilla.org/en-US/docs/Web/API/Blob/stream)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah good point, I will change to stream.

Not sure I understand - the datastore can use content claims under the hood no?

yes it can. But probably we would like it to be within the protocol implementation rather than next to the infra wiring? Let's discuss it when we get there

This was referenced Mar 18, 2024
Closed
Closed
@vasco-santos vasco-santos requested review from alanshaw and removed request for alanshaw March 26, 2024 09:24
alanshaw
alanshaw approved these changes Mar 26, 2024
View reviewed changes
packages/filecoin-api/src/storefront/api.ts Outdated
Comment on lines 126 to 127
// eslint-disable-next-line @typescript-eslint/no-explicit-any
connection: ConnectionView<any>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// eslint-disable-next-line @typescript-eslint/no-explicit-any
connection: ConnectionView<any>
connection: ConnectionView<import('@web3-storage/content-claims/server/service/api').Service>

packages/filecoin-api/src/storefront/events.js Outdated
// TODO: verify piece
// read and compute piece for content
// TODO: needs to be hooked with location claims
const contentGetRes = await context.dataStore.get(message.content)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand - the datastore can use content claims under the hood no?

packages/filecoin-api/src/storefront/events.js Outdated
// TODO: verify piece
// read and compute piece for content
// TODO: needs to be hooked with location claims
const contentGetRes = await context.dataStore.get(message.content)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.get() implies to me it's retrieving the whole data, perhaps consider calling this .stream()? (similar to https://developer.mozilla.org/en-US/docs/Web/API/Blob/stream)

@vasco-santos vasco-santos force-pushed the feat/trigger-filecoin-pipeline-from-client branch 3 times, most recently from 110630e to 4aa3d6f Compare March 26, 2024 11:42
@vasco-santos vasco-santos force-pushed the feat/trigger-filecoin-pipeline-from-client branch 2 times, most recently from 8305489 to cbd288e Compare March 26, 2024 11:44
@vasco-santos vasco-santos force-pushed the feat/trigger-filecoin-pipeline-from-client branch from cbd288e to 20f8bfd Compare March 26, 2024 11:48
@vasco-santos vasco-santos merged commit 421bacb into main Mar 26, 2024
14 checks passed
@vasco-santos vasco-santos deleted the feat/trigger-filecoin-pipeline-from-client branch March 26, 2024 11:51
@it-dag-house it-dag-house mentioned this pull request Mar 26, 2024
Merged
vasco-santos pushed a commit that referenced this pull request Mar 26, 2024
@it-dag-house
chore(main): release filecoin-api 4.6.0 (#1346)
4b166fd 
🤖 I have created a release *beep* *boop*
---


##
[4.6.0](filecoin-api-v4.5.0...filecoin-api-v4.6.0)
(2024-03-26)


### Features

* api waits for trigger filecoin pipeline from the client
([#1332](#1332))
([421bacb](421bacb))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
@vasco-santos vasco-santos mentioned this pull request Mar 26, 2024
Closed
vasco-santos added a commit to storacha/w3infra that referenced this pull request Apr 2, 2024
@vasco-santos
feat: upgrade storefront api with filecoin offer from bucket event in…
3fd05a8 
…stead of directly compute (#343)

This PR intends to
[upgrade](storacha/w3up#1332) storefront
package. This enables `filecoin/offer` invocations lead to messages to
the submit queue, where message consumer will compute Piece for the
content. The main goal of this is to have the client to compute Filecoin
Pieces and offer them, instead of pieces being directly computed from
bucket event and written into the DB (skipping the submit validation).

Note that:
- given we will need to deal with client upgrades and old clients still
running for the time being, I opted in this PR to still keep a version
of the bucket event in place. However, instead of it running computation
of piece, it also triggers `filecoin/offer` so that we can keep old
clients running where `filecoin/offer` is not invoked. This of course
means that we may need to run Piece computation twice for the old path.
However, looking at our AWS bill, we paid 200 USD in
February for lambda execution of whole w3infra. This assumes then that we will
not incur a lot of additional cost while making migration much simpler
- datastore currently implements a composed store with S3+R2, which
allows us to move on an iterative approach where we can deploy first
step without client upgrade, while we can iterate on writes to anywhere
with the overall picture.
This was referenced Apr 15, 2024
Merged
Closed
vasco-santos added a commit that referenced this pull request Apr 19, 2024
@vasco-santos
fix!: drop filecoin storefront skip submit queue option (#1371)
1114383 
This is not needed since #1332
got released and used in w3infra
storacha/w3infra#343 and has been deployed
in production without issues for the last few weeks

BREAKING CHANGE: not possible to skip submit queue on storefront service
anymore
Peeja added a commit to storacha/upload-service that referenced this pull request Dec 2, 2024
@Peeja
chore(main): release filecoin-api 1.0.0 (#93)
4e2de50 
🤖 I have created a release *beep* *boop*
---


## 1.0.0 (2024-12-02)


### ⚠ BREAKING CHANGES

* **upload-api:** integrate agent store for idempotence &
invocation/receipt persistence
([storacha#1444](https://github.com/storacha/upload-service/issues/1444))
* dataStore in storefront renamed to contentStore
* not possible to skip submit queue on storefront service anymore
* return allocated bytes in `store/add` receipt
([storacha#1213](https://github.com/storacha/upload-service/issues/1213))

### Features

* add blob protocol to upload-client
([storacha#1425](https://github.com/storacha/upload-service/issues/1425))
([49aef56](49aef56))
* add support to prepend pieces while buffering to aggregate
([storacha#1301](https://github.com/storacha/upload-service/issues/1301))
([dff1846](dff1846))
* add usage/report capability
([storacha#1079](https://github.com/storacha/upload-service/issues/1079))
([6418b4b](6418b4b))
* aggregator keeping oldest piece ts
([storacha#1188](https://github.com/storacha/upload-service/issues/1188))
([97a7def](97a7def))
* api waits for trigger filecoin pipeline from the client
([storacha#1332](https://github.com/storacha/upload-service/issues/1332))
([421bacb](421bacb))
* filecoin info
([storacha#1091](https://github.com/storacha/upload-service/issues/1091))
([adb2442](adb2442))
* **filecoin-api:** allow custom hashing function to be passed to
aggregate builder
([storacha#1553](https://github.com/storacha/upload-service/issues/1553))
([e2653d4](e2653d4))
* **filecoin-api:** paginated queries
([storacha#1521](https://github.com/storacha/upload-service/issues/1521))
([25ed7d7](25ed7d7))
* move aggregate information out of deals in filecoin/info
([storacha#1192](https://github.com/storacha/upload-service/issues/1192))
([18dc590](18dc590))
* return allocated bytes in `store/add` receipt
([storacha#1213](https://github.com/storacha/upload-service/issues/1213))
([5d52e44](5d52e44))
* router ([#11](#11))
([c810735](c810735))
* upgrade ucanto/transport to 9.1.0 in all packages to get more verbose
errors from HTTP transport on non-ok response
([storacha#1312](https://github.com/storacha/upload-service/issues/1312))
([d6978d7](d6978d7))
* **upload-api:** integrate agent store for idempotence &
invocation/receipt persistence
([storacha#1444](https://github.com/storacha/upload-service/issues/1444))
([c9bf33e](c9bf33e))
* upload-client uploadDirectory, by default, sorts the provided files by
file name to help the user call us in a way that is deterministic and
minimizes cost
([storacha#1173](https://github.com/storacha/upload-service/issues/1173))
([8cd2555](8cd2555))
* use digest in `blob/accept` location commitment
([storacha#1480](https://github.com/storacha/upload-service/issues/1480))
([ade45eb](ade45eb))


### Fixes

* aggregate offer invocation cid wrong
([storacha#1063](https://github.com/storacha/upload-service/issues/1063))
([90a5a4d](90a5a4d))
* avoid duplicates on aggregator buffer concat
([storacha#1259](https://github.com/storacha/upload-service/issues/1259))
([9e64bab](9e64bab))
* check service did in w3filecoin
([storacha#1476](https://github.com/storacha/upload-service/issues/1476))
([11b00bf](11b00bf))
* configure max pieces
([storacha#1566](https://github.com/storacha/upload-service/issues/1566))
([71674ed](71674ed))
* dealer offer store keys without space
([storacha#1066](https://github.com/storacha/upload-service/issues/1066))
([301f411](301f411))
* drop filecoin storefront skip submit queue option
([storacha#1371](https://github.com/storacha/upload-service/issues/1371))
([1114383](1114383))
* enable storefront signer to be different from main service signer
([storacha#1072](https://github.com/storacha/upload-service/issues/1072))
([21ded3c](21ded3c))
* filecoin test use blob
([storacha#1422](https://github.com/storacha/upload-service/issues/1422))
([359c0b7](359c0b7))
* **filecoin-api:** parallel put to piece accept queue
([storacha#1560](https://github.com/storacha/upload-service/issues/1560))
([e7cbb6d](e7cbb6d))
* issue where typedoc docs would only show full docs for w3up-client
([storacha#1141](https://github.com/storacha/upload-service/issues/1141))
([0b8d3f3](0b8d3f3))
* lint ([storacha#1095](https://github.com/storacha/upload-service/issues/1095))
([f9cc770](f9cc770))
* migrate repo
([storacha#1389](https://github.com/storacha/upload-service/issues/1389))
([475a287](475a287))
* package metadata
([storacha#1161](https://github.com/storacha/upload-service/issues/1161))
([b8a1cc2](b8a1cc2))
* receipt chain has wrong CID because no expiration is set
([storacha#1060](https://github.com/storacha/upload-service/issues/1060))
([dfb46d8](dfb46d8))
* rename blob and index client capabilities
([storacha#1478](https://github.com/storacha/upload-service/issues/1478))
([17e3a31](17e3a31))
* repo URLs
([storacha#1550](https://github.com/storacha/upload-service/issues/1550))
([e02ddf3](e02ddf3))
* return piece accept receipt error
([storacha#1512](https://github.com/storacha/upload-service/issues/1512))
([05283cf](05283cf))
* revert enable storefront signer to be different from main service
signer ([storacha#1075](https://github.com/storacha/upload-service/issues/1075))
([80cdde0](80cdde0))
* storefront content store rename and separation for test
([storacha#1409](https://github.com/storacha/upload-service/issues/1409))
([05e5db3](05e5db3))
* storefront events cron with max concurrency
([storacha#1191](https://github.com/storacha/upload-service/issues/1191))
([11010c9](11010c9))
* upgrade ucanto core
([storacha#1127](https://github.com/storacha/upload-service/issues/1127))
([5ce4d22](5ce4d22))
* upgrade ucanto libs and format filecoin api
([storacha#1359](https://github.com/storacha/upload-service/issues/1359))
([87ca098](87ca098))
* upload API test fixes
([6b0d72d](6b0d72d))
* use one-webcrypto from npm
([storacha#1525](https://github.com/storacha/upload-service/issues/1525))
([9345c54](9345c54))


### Other Changes

* Add `pnpm dev` to watch-build all packages
([storacha#1533](https://github.com/storacha/upload-service/issues/1533))
([07970ef](07970ef))
* appease linter
([782c6d0](782c6d0))
* **main:** release filecoin-api 4.0.2
([storacha#1061](https://github.com/storacha/upload-service/issues/1061))
([5918ff2](5918ff2))
* **main:** release filecoin-api 4.0.3
([storacha#1064](https://github.com/storacha/upload-service/issues/1064))
([ef4ce7a](ef4ce7a))
* **main:** release filecoin-api 4.0.4
([storacha#1067](https://github.com/storacha/upload-service/issues/1067))
([c3ed3b9](c3ed3b9))
* **main:** release filecoin-api 4.0.5
([storacha#1073](https://github.com/storacha/upload-service/issues/1073))
([9cf2fc0](9cf2fc0))
* **main:** release filecoin-api 4.0.6
([storacha#1077](https://github.com/storacha/upload-service/issues/1077))
([40d0e7a](40d0e7a))
* **main:** release filecoin-api 4.1.0
([storacha#1085](https://github.com/storacha/upload-service/issues/1085))
([9b752f9](9b752f9))
* **main:** release filecoin-api 4.1.1
([storacha#1133](https://github.com/storacha/upload-service/issues/1133))
([3621b42](3621b42))
* **main:** release filecoin-api 4.1.2
([storacha#1148](https://github.com/storacha/upload-service/issues/1148))
([fa6405d](fa6405d))
* **main:** release filecoin-api 4.2.0
([storacha#1164](https://github.com/storacha/upload-service/issues/1164))
([dfdf762](dfdf762))
* **main:** release filecoin-api 4.3.0
([storacha#1194](https://github.com/storacha/upload-service/issues/1194))
([a1aa231](a1aa231))
* **main:** release filecoin-api 4.3.1
([storacha#1205](https://github.com/storacha/upload-service/issues/1205))
([7f42f39](7f42f39))
* **main:** release filecoin-api 4.4.0
([storacha#1303](https://github.com/storacha/upload-service/issues/1303))
([bb74ffc](bb74ffc))
* **main:** release filecoin-api 4.5.0
([storacha#1314](https://github.com/storacha/upload-service/issues/1314))
([d31ff06](d31ff06))
* **main:** release filecoin-api 4.6.0
([storacha#1346](https://github.com/storacha/upload-service/issues/1346))
([4b166fd](4b166fd))
* **main:** release filecoin-api 4.6.1
([storacha#1363](https://github.com/storacha/upload-service/issues/1363))
([efcb912](efcb912))
* **main:** release filecoin-api 5.0.0
([storacha#1384](https://github.com/storacha/upload-service/issues/1384))
([cb5b628](cb5b628))
* **main:** release filecoin-api 5.0.1
([storacha#1398](https://github.com/storacha/upload-service/issues/1398))
([2e3ef61](2e3ef61))
* **main:** release filecoin-api 6.0.0
([storacha#1414](https://github.com/storacha/upload-service/issues/1414))
([939e839](939e839))
* **main:** release filecoin-api 6.0.1
([storacha#1424](https://github.com/storacha/upload-service/issues/1424))
([f8c6c1d](f8c6c1d))
* **main:** release filecoin-api 7.0.0
([storacha#1430](https://github.com/storacha/upload-service/issues/1430))
([604d300](604d300))
* **main:** release filecoin-api 7.1.0
([storacha#1481](https://github.com/storacha/upload-service/issues/1481))
([9d38080](9d38080))
* **main:** release filecoin-api 7.1.1
([storacha#1513](https://github.com/storacha/upload-service/issues/1513))
([d4f5f45](d4f5f45))
* **main:** release filecoin-api 7.2.0
([storacha#1522](https://github.com/storacha/upload-service/issues/1522))
([0aebf9f](0aebf9f))
* **main:** release filecoin-api 7.2.1
([storacha#1528](https://github.com/storacha/upload-service/issues/1528))
([5b8e148](5b8e148))
* **main:** release filecoin-api 7.3.0
([storacha#1540](https://github.com/storacha/upload-service/issues/1540))
([77b31e9](77b31e9))
* **main:** release filecoin-api 7.3.1
([storacha#1561](https://github.com/storacha/upload-service/issues/1561))
([61d408a](61d408a))
* **main:** release filecoin-api 7.3.2
([storacha#1567](https://github.com/storacha/upload-service/issues/1567))
([534d3be](534d3be))
* package renames
([0f797ed](0f797ed))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alanshaw alanshaw alanshaw approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vasco-santos @alanshaw