Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Autoretrieve Stats - Data Ingestion #872

Open
jcace opened this issue Jan 13, 2023 · 2 comments
Open

Autoretrieve Stats - Data Ingestion #872

jcace opened this issue Jan 13, 2023 · 2 comments
Labels
New Feature Issues that we will work on with people or ourselves

Comments

@jcace
Copy link
Contributor

jcace commented Jan 13, 2023

Modify Autoretrieve to collect an event stream of all retrievals, and write it to our retrieval event database.

This should be possible by implementing a Lassie RetrievalSubscriber in Autoretrieve, and then wiring it up so that every time a retrieval event fires we will save the data to the database.

This will depend on getting our database/schema figured out ref: #873

@jcace jcace added the New Feature Issues that we will work on with people or ourselves label Jan 13, 2023
@jcace jcace added this to the Incentivized Retrievals Q1 milestone Jan 13, 2023
@jcace jcace changed the title Autoretrieve Data Ingestion Autoretrieve Stats - Data Ingestion Jan 13, 2023
@jcace
Copy link
Contributor Author

jcace commented Jan 17, 2023

Updated task description:

  1. Add an endpoint to estuary-metrics to be POST'ed to with retrieval stats. It will follow the struct defined here:
    https://github.dev/filecoin-project/lassie/blob/main/pkg/eventrecorder/eventrecorder.go#L63

  2. Configure autoretrieve to use Lassie Event Publisher to publish events to estuary metrics API (it is configured by this line here)
    https://github.dev/application-research/autoretrieve/blob/master/autoretrieve.go#L197

@jcace
Copy link
Contributor Author

jcace commented Jan 17, 2023

estuary-metrics task: application-research/estuary-metrics#18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
New Feature Issues that we will work on with people or ourselves
Projects
Status: No status
Development

No branches or pull requests

1 participant