Skip to content
This repository has been archived by the owner on Mar 3, 2023. It is now read-only.

Support topology deactivation period before scaling #1390

Closed
billonahill opened this issue Sep 14, 2016 · 0 comments
Closed

Support topology deactivation period before scaling #1390

billonahill opened this issue Sep 14, 2016 · 0 comments
Assignees
Milestone

Comments

@billonahill
Copy link
Contributor

This is a simple way to ensure stateful data correctness for many topologies, via @pankajgupta.

The approach used for scaling at Twitter currently is the following:

  1. Deactivate the topology for X seconds. This halts new tuples from being emitted by spouts and causes time-based caches to be drained, where X > the cache interval.
  2. Redeploy the topology as activated. New tuples are emitted and processed by the correct instances when using fields grouping. Caches are initially empty (i.e. drained) and correct.

A similar feature should be built into the scaling event. The topology update command would take an optional parameter to indicate that before the scaling event occurs, deactivate the topology for X seconds. The default deactivation interval could also be provided in the topology config, which would allow frameworks built on top of Heron to inject this setting based on their configured cache settings. This would eliminate the need for the user to set this properly.

Related to #1292.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant