Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PoWeb: Subscribe to new parcel notifications with GCP PubSub #760

Closed
gnarea opened this issue Mar 8, 2022 · 1 comment
Closed

PoWeb: Subscribe to new parcel notifications with GCP PubSub #760

gnarea opened this issue Mar 8, 2022 · 1 comment
Labels
enhancement New feature or request

Comments

@gnarea
Copy link
Member

gnarea commented Mar 8, 2022

Describe the problem

We currently use NATS Streaming in PoWeb's collectParcels method to be notified about new parcels, but we're deprecating NATS Streaming in production (relaycorp/cloud-gateway#84).

Describe the solution you'd like

This messaging channel should continue to be powered by NATS Streaming in development, but it should support GCP PubSub as well.

We might want to factor out this broker-agnostic functionality into an external library for non-durable subscriptions (@relaycorp/cloud-notifications?).

Pay a lot of attention to the quotas and limits of PubSub; in particular:

  • Max subscriptions (10,000) per project and per topic. We might want to have one subscription per process, and use an EventEmitter to notify each WebSocket connection... Which would unfortunately mean that we could have fewer than 10k processes, unless we build a new internal service to fan out notifications.
  • Publisher throughput per region (200 MB/s in small regions). So we shouldn't be sending too much data (we won't be the case once Store incoming, unprocessed cargo in object store #746 is fixed).

Describe alternatives you've considered

None

Additional context

This isn't the only use of NATS Streaming. See #759

@gnarea
Copy link
Member Author

gnarea commented Dec 7, 2022

Superseded by #929.

@gnarea gnarea closed this as not planned Won't fix, can't repro, duplicate, stale Dec 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant