Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Long running dump and restore capability #6975

Merged
merged 19 commits into from
Nov 15, 2023
Merged

Long running dump and restore capability #6975

merged 19 commits into from
Nov 15, 2023

Conversation

nopcoder
Copy link
Contributor

New dump and restore refs API

A new API for dumping and restoring refs. The API allows users to submit requests and then pull the status of the task and the complete information about the request once it is finished.

The new API is used by the lakectl refs-dump and lakectl refs-restore commands.

The new API offers a long running operations without the risk of timeout dropping the request.

Close #6527

@nopcoder nopcoder added area/API Improvements or additions to the API area/lakectl Issues related to lakeFS' command line interface (lakectl) include-changelog PR description should be included in next release changelog labels Nov 10, 2023
@nopcoder nopcoder self-assigned this Nov 10, 2023
Copy link

github-actions bot commented Nov 10, 2023

♻️ PR Preview 86ad171 has been successfully destroyed since this PR has been closed.

🤖 By surge-preview

@nopcoder nopcoder requested a review from johnnyaug November 12, 2023 08:28
Copy link
Contributor

@johnnyaug johnnyaug left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome. Not blocking, but please see comments.

api/swagger.yml Outdated Show resolved Hide resolved
api/swagger.yml Outdated Show resolved Hide resolved
api/swagger.yml Outdated Show resolved Hide resolved
api/swagger.yml Outdated
tags:
- repositories
operationId: restoreStatus
summary: Restore repository (tags, commits, branches) from object store
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
summary: Restore repository (tags, commits, branches) from object store
summary: Status of a restore request

?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not done?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

think I updated and override the doc version

pkg/api/controller.go Outdated Show resolved Hide resolved
// pullRestoreStatus polls the restore status endpoint until the restore is complete or times out.
// test will fail in case of error.
// will return nil in case of timeout.
func pullRestoreStatus(t *testing.T, clt apigen.ClientWithResponsesInterface, repo string, taskID string) *apigen.RepositoryRestoreStatus {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
func pullRestoreStatus(t *testing.T, clt apigen.ClientWithResponsesInterface, repo string, taskID string) *apigen.RepositoryRestoreStatus {
func pollRestoreStatus(t *testing.T, clt apigen.ClientWithResponsesInterface, repo string, taskID string) *apigen.RepositoryRestoreStatus {

(also in other places)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are still occurrences of "pull" instead of "poll".

pkg/catalog/catalog.proto Outdated Show resolved Hide resolved
cmd/lakectl/cmd/refs_dump.go Outdated Show resolved Hide resolved
cmd/lakectl/cmd/refs_dump.go Outdated Show resolved Hide resolved
cmd/lakefs/cmd/run.go Outdated Show resolved Hide resolved
@nopcoder nopcoder requested a review from johnnyaug November 14, 2023 14:17
@nopcoder nopcoder enabled auto-merge (squash) November 15, 2023 07:56
@nopcoder nopcoder merged commit 245d175 into master Nov 15, 2023
31 checks passed
@nopcoder nopcoder deleted the feature/dump-async branch November 15, 2023 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/API Improvements or additions to the API area/lakectl Issues related to lakeFS' command line interface (lakectl) include-changelog PR description should be included in next release changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

API cleanup: dump/restoreRef as long running operation
3 participants