Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bitfield distribution design and research #4232

Closed
Tracked by #3599
kishansagathiya opened this issue Oct 7, 2024 · 2 comments
Closed
Tracked by #3599

bitfield distribution design and research #4232

kishansagathiya opened this issue Oct 7, 2024 · 2 comments
Assignees
@haikoschol
Labels
A-design issue requires design work to think about how it would best be accomplished. S-subsystems-availability issues related to polkadot host availability subsystem functionality. T-research this issue/pr is a research type issue.

Comments

@kishansagathiya
Copy link
Contributor

Issue summary

  • Go over the node implementors guide and the code base of polkadot-sdk as well.
  • And come up with a design that makes sense for golang.
  • This design version should lead to some sort of spec that other devs could read and implement the features simply by reading specs.
  • The spec could be written in the form of issues of as a notion page as well.

Other information and links
@haikoschol haikoschol added A-design issue requires design work to think about how it would best be accomplished. T-research this issue/pr is a research type issue. S-subsystems-availability issues related to polkadot host availability subsystem functionality. labels Oct 7, 2024
@haikoschol haikoschol mentioned this issue Oct 8, 2024
Open
12 tasks
haikoschol added a commit that referenced this issue Oct 14, 2024
@haikoschol
feat(dot/sync): improve worker pool
482000e 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
haikoschol added a commit that referenced this issue Oct 14, 2024
@haikoschol
feat(dot/sync): improve worker pool
f2c0411 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
haikoschol added a commit that referenced this issue Oct 15, 2024
@haikoschol
feat(dot/sync): improve worker pool
d0c660d 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
haikoschol added a commit that referenced this issue Oct 15, 2024
@haikoschol
feat(dot/sync): improve worker pool
f9d469c 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
haikoschol added a commit that referenced this issue Oct 15, 2024
@haikoschol
feat(dot/sync): improve worker pool
8437f37 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
@haikoschol haikoschol mentioned this issue Oct 15, 2024
Closed
haikoschol added a commit that referenced this issue Oct 15, 2024
@haikoschol
feat(dot/sync): improve worker pool
03def6b 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
haikoschol added a commit that referenced this issue Oct 15, 2024
@haikoschol
feat(dot/sync): improve worker pool
36a2aa2 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
haikoschol added a commit that referenced this issue Oct 15, 2024
@haikoschol
feat(dot/sync): improve worker pool
987cb6f 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
haikoschol added a commit that referenced this issue Oct 17, 2024
@haikoschol
feat(dot/sync): improve worker pool
4c0d5cb 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
haikoschol added a commit that referenced this issue Oct 18, 2024
@haikoschol
feat(dot/sync): improve worker pool
c875d08 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
haikoschol added a commit that referenced this issue Oct 24, 2024
@haikoschol
feat(dot/sync): improve worker pool
d2d1903 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
haikoschol added a commit that referenced this issue Oct 24, 2024
@haikoschol
feat(dot/sync): improve worker pool
d4ddd49 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
haikoschol added a commit that referenced this issue Oct 28, 2024
@haikoschol
feat(dot/sync): improve worker pool
e3f9eef 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
haikoschol added a commit that referenced this issue Nov 1, 2024
@haikoschol
feat(dot/sync): improve worker pool
3bb5500 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
haikoschol added a commit that referenced this issue Nov 6, 2024
@haikoschol
feat(dot/sync): improve worker pool
e32f5f7 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
haikoschol added a commit that referenced this issue Nov 7, 2024
@haikoschol
feat(dot/sync): improve worker pool
e2c9ffd 
The main difference in the worker pool API is that SubmitBatch() does
not block until the whole batch has been processed. Instead, it returns
an ID which can be used to retrieve the current state of the batch.
In addition, Results() returns a channel over which task results are
sent as they become available.

The main improvement this brings is increased concurrency, since results
can be processed before the whole batch has been completed.

What has not changed is the overall flow of the Strategy interface;
getting a new batch of tasks with NextActions() and processing the
results with Process().

Closes #4232
@haikoschol
Copy link
Contributor

Ignore the link to the worker pool PR. I put the wrong issue number in the commit message. 🤦

@haikoschol haikoschol mentioned this issue Nov 26, 2024
Merged
@haikoschol
Copy link
Contributor

All issues have been created and linked to the epic.

@haikoschol haikoschol mentioned this issue Dec 17, 2024
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@haikoschol haikoschol

Labels
A-design issue requires design work to think about how it would best be accomplished. S-subsystems-availability issues related to polkadot host availability subsystem functionality. T-research this issue/pr is a research type issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@haikoschol @kishansagathiya