Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate a go-legs publisher endpoint to publish dealbot metadata #342

Closed
9 tasks done
masih opened this issue Nov 12, 2021 · 3 comments
Closed
9 tasks done

Integrate a go-legs publisher endpoint to publish dealbot metadata #342

masih opened this issue Nov 12, 2021 · 3 comments
Assignees
@masih
Labels
P1 P1 Issue workstream/repsys Issue blocks Reputation System MVP

Comments

@masih
Copy link
Member

masih commented Nov 12, 2021 •
edited
Loading

The metadata in dealbot is already represented as DAG and exposed via graphQL here. It would be useful to integrate a go-legs publisher such that the same DAG is retrievable via a go-legs subscriber.

For example, this would be useful as the way to ingest dealbot metadata into a side chain reputation system.

  • Integrate go-legs into updates persisted by state db.
    • Upgrade imld-prime.
    • Upgrade Filecoin API.
  • Bootstrap go-legs libp2p host with Pando.
  • Configure runtime parameters and re-deploy dealbot.
  • Set up external IPs for libp2p host on dealbot controller and configure as listen addresses
  • Register dealbot as a provider in Pando
  • Confirm data is flowing on Pando side.
  • Publish Pando-compatible metadata
@dkkapur dkkapur added P1 P1 Issue workstream/repsys Issue blocks Reputation System MVP labels Nov 23, 2021
@dkkapur
Copy link
Collaborator

dkkapur commented Nov 23, 2021

This represents the current plan to export dealbot data into pando

@masih
Copy link
Member Author

masih commented Nov 24, 2021

Requires IPLD dependency upgrade first, being worked on in #343

This was referenced Feb 15, 2022
Merged
Merged
Merged
@masih masih self-assigned this Feb 17, 2022
masih added a commit to filecoin-project/lotus-infra that referenced this issue Feb 17, 2022
@masih
Configure dealbot bootstrap address to pando service
fbd9606 
Configure the dealbot bootstrap addrinfo to Pando service on both
staging and production clusters. The bootrstrapping mechanism is needed
so that the go-lengs announcements are propagated to pando over
gossipsub.

Relates to:
 - filecoin-project/dealbot#342
 - http://pando-api.kencloud.com/pando/info
 - https://github.com/kenlabs/pando
@masih masih mentioned this issue Feb 17, 2022
Merged
masih added a commit to filecoin-project/lotus-infra that referenced this issue Feb 17, 2022
@masih
Configure dealbot bootstrap address to pando service
30efb95 
Configure the dealbot bootstrap addrinfo to Pando service on both
staging and production clusters. The bootrstrapping mechanism is needed
so that the go-lengs announcements are propagated to pando over
gossipsub.

Relates to:
 - filecoin-project/dealbot#342
 - http://pando-api.kencloud.com/pando/info
 - https://github.com/kenlabs/pando
masih added a commit to filecoin-project/lotus-infra that referenced this issue Feb 17, 2022
@masih
Configure dealbot bootstrap address to pando and set log level to info
00887eb 
Configure the dealbot bootstrap addrinfo to Pando service on both
staging and production clusters. The bootrstrapping mechanism is needed
so that the go-lengs announcements are propagated to pando over
gossipsub.

While at it, set the log level of dealbot controller to `INFO`.

Relates to:
 - filecoin-project/dealbot#342
 - http://pando-api.kencloud.com/pando/info
 - https://github.com/kenlabs/pando
masih added a commit to filecoin-project/lotus-infra that referenced this issue Feb 17, 2022
@masih
Configure dealbot bootstrap address to pando and set log level to info
a50c21a 
Configure the dealbot bootstrap addrinfo to Pando service on both
staging and production clusters. The bootrstrapping mechanism is needed
so that the go-lengs announcements are propagated to pando over
gossipsub.

While at it, set the log level of dealbot controller to `INFO`.

Relates to:
 - filecoin-project/dealbot#342
 - http://pando-api.kencloud.com/pando/info
 - https://github.com/kenlabs/pando
@masih masih mentioned this issue Feb 22, 2022
Merged
masih added a commit to filecoin-project/lotus-infra that referenced this issue Feb 22, 2022
@masih
Set loadbalancer type for dealbot controller libp2p service
733c893 
USe NLB as the type of loadbalacer for the libp2p node exposed on
dealbot controller added here:
 - filecoin-project/helm-charts#124

Relates to:
 - filecoin-project/dealbot#342
@masih masih mentioned this issue Feb 22, 2022
Merged
masih added a commit that referenced this issue Feb 22, 2022
@masih
Bump helm chart version number to include libp2p service
a445ade 
Bump the helm chart version number to create a service that exposes
libp2p node statrted up in controller used for syncing chain of updates
announced by go-legs. The K8S service is added here:
 - filecoin-project/helm-charts#124

Relates to:
 - #342
@masih masih mentioned this issue Feb 22, 2022
Merged
masih added a commit that referenced this issue Feb 22, 2022
@masih
Bump helm chart version number to include libp2p service
a1fd7a0 
Bump the helm chart version number to create a service that exposes
libp2p node statrted up in controller used for syncing chain of updates
announced by go-legs. The K8S service is added here:
 - filecoin-project/helm-charts#124

Relates to:
 - #342
@masih
Copy link
Member Author

masih commented Feb 23, 2022

We now have dealbot libp2p exposed and serving legs sync requests.

Dealbot is also registered as a provider on Pando side. The instance of dealbot registered is the one running on mainet production cluster.

masih added a commit that referenced this issue Feb 25, 2022
@masih
Publish Pando-compatible metadata over GraphSync
570cde7 
Implement a publisher specific to Pando metadata that uses the CID of
`RecordUpdate` as byte value of Pando metadata and exposes a GraphSync
server that can resolve both Pando metadata, and the `RecordUpdate`
CIDs.

Decouple `statedb` from publication by changing the state interface to
return the CID of `RecordUpdate`. This removes all libp2p and legs code
from the `state` package. Instead, the publication is done on the
controller itself upon `POST`s to `/complete/<workedby>` as a
non-critical operation. This makes a better structured code in the
context of Separation of Concerns.

Relates to:
 - #342
@masih masih mentioned this issue Feb 25, 2022
Merged
masih added a commit that referenced this issue Feb 25, 2022
@masih
Publish Pando-compatible metadata over GraphSync
0663ac7 
Implement a publisher specific to Pando metadata that uses the CID of
`RecordUpdate` as byte value of Pando metadata and exposes a GraphSync
server that can resolve both Pando metadata, and the `RecordUpdate`
CIDs.

Decouple `statedb` from publication by changing the state interface to
return the CID of `RecordUpdate`. This removes all libp2p and legs code
from the `state` package. Instead, the publication is done on the
controller itself upon `POST`s to `/complete/<workedby>` as a
non-critical operation. This makes a better structured code in the
context of Separation of Concerns.

Relates to:
 - #342
masih added a commit that referenced this issue Feb 25, 2022
@masih
Publish Pando-compatible metadata over GraphSync
a08d090 
Implement a publisher specific to Pando metadata that uses the CID of
`RecordUpdate` as byte value of Pando metadata and exposes a GraphSync
server that can resolve both Pando metadata, and the `RecordUpdate`
CIDs.

Decouple `statedb` from publication by changing the state interface to
return the CID of `RecordUpdate`. This removes all libp2p and legs code
from the `state` package. Instead, the publication is done on the
controller itself upon `POST`s to `/complete/<workedby>` as a
non-critical operation. This makes a better structured code in the
context of Separation of Concerns.

Relates to:
 - #342
masih added a commit that referenced this issue Feb 25, 2022
@masih
Publish Pando-compatible metadata over GraphSync
b57f2e2 
Implement a publisher specific to Pando metadata that uses the CID of
`RecordUpdate` as byte value of Pando metadata and exposes a GraphSync
server that can resolve both Pando metadata, and the `RecordUpdate`
CIDs.

Decouple `statedb` from publication by changing the state interface to
return the CID of `RecordUpdate`. This removes all libp2p and legs code
from the `state` package. Instead, the publication is done on the
controller itself upon `POST`s to `/complete/<workedby>` as a
non-critical operation. This makes a better structured code in the
context of Separation of Concerns.

Relates to:
 - #342
@masih masih closed this as completed Feb 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@masih masih

Labels
P1 P1 Issue workstream/repsys Issue blocks Reputation System MVP
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@masih @dkkapur