Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add export action and action manager #807

Merged
merged 14 commits into from
Oct 21, 2020
Merged

add export action and action manager #807

merged 14 commits into from
Oct 21, 2020

Conversation

guy-har
Copy link
Contributor

@guy-har guy-har commented Oct 13, 2020

draft

and action manager
draft
@guy-har guy-har requested a review from arielshaqed October 13, 2020 07:47
add tests for delete and touch
Copy link
Contributor

@arielshaqed arielshaqed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Very nice stuff.

Some operational comments on the manager. Also I think the block adapter copy needs some operation: for export it is not writing to a destination namespace and hash, just to some path.

action/action_manager.go Outdated Show resolved Hide resolved
action/action_manager.go Outdated Show resolved Hide resolved
action/action_manager.go Outdated Show resolved Hide resolved
action/action_manager.go Outdated Show resolved Hide resolved
action/action_manager.go Outdated Show resolved Hide resolved
action/action_manager_test.go Outdated Show resolved Hide resolved
action/action_manager_test.go Outdated Show resolved Hide resolved
action/handler.go Outdated Show resolved Hide resolved
export/export_handler.go Show resolved Hide resolved
export/export_handler_test.go Outdated Show resolved Hide resolved
guy-har and others added 5 commits October 13, 2020 18:53
Co-authored-by: arielshaqed <ariels@treeverse.io>
Co-authored-by: arielshaqed <ariels@treeverse.io>
document action_manager
add data to logs
Copy link
Contributor

@arielshaqed arielshaqed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
I'm approving, but would really like to see improved numbers on the benchmark from using this! That can happen in a future PR, too.

Workers int // number of goroutines handling tasks
ChannelSize int // size of the channel containing tasks for workers
MaxTasks int // max tasks requested in every ownTasks request
WaitTime *time.Duration // time to wait if OwnTasks returned no tasks.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why these are pointers and the ints are values, identified by being zero.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arielshaqed Because 0 is a valid argument for the pointers.
And I didn't wan't to override it with a default value.

parade/action_manager.go Outdated Show resolved Hide resolved
ownedTasks, err := a.parade.OwnTasks(a.handler.Actor(), a.properties.MaxTasks, a.handler.Actions(), a.properties.MaxDuration)
if err != nil {
logging.Default().WithField("actor", a.handler.Actor()).Errorf("manager failed to receive tasks: %s", err)
time.Sleep(*a.properties.WaitTime)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
time.Sleep(*a.properties.WaitTime)
time.Sleep(*a.properties.ErrWaitTime)
continue

?

The first line is in order to wait the time I think you meant. The second line is in order not to wait WaitTime after waiting ErrWaitTime.

res := a.handler.Handle(task.Action, task.Body)
err := a.parade.ReturnTask(task.ID, task.Token, res.Status, res.StatusCode)
if err != nil {
logging.Default().WithFields(logging.Fields{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we maybe add these fields to a logger and use that on all the handlers?
(Something to consider, certainly not in any way required for this PR!)

guy-har and others added 3 commits October 18, 2020 12:21
Co-authored-by: arielshaqed <ariels@treeverse.io>
document actor
@guy-har guy-har marked this pull request as ready for review October 19, 2020 09:40
@guy-har guy-har merged commit 36a5a8d into master Oct 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants