Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Find expired and active commits #2069

Merged
merged 31 commits into from
Jun 21, 2021
Merged
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
d466eb4
find active commits
johnnyaug Jun 1, 2021
0abc5c6
wip
johnnyaug Jun 4, 2021
47f44f6
Merge branch 'master' into feature/find_active_commits
johnnyaug Jun 6, 2021
431c207
rename test
johnnyaug Jun 6, 2021
3763a54
linter fixes
johnnyaug Jun 6, 2021
3876eff
fix test
johnnyaug Jun 6, 2021
2ddf880
wip
johnnyaug Jun 8, 2021
40ee31f
Merge branch 'master' into feature/find_active_commits
johnnyaug Jun 9, 2021
d37ec3a
wip
johnnyaug Jun 10, 2021
de1eef4
Merge branch 'master' into feature/find_active_commits
johnnyaug Jun 13, 2021
bb5823b
wip
johnnyaug Jun 14, 2021
499667f
Merge branch 'master' into feature/find_active_commits
johnnyaug Jun 14, 2021
494239d
put everything in graveler
johnnyaug Jun 14, 2021
8cb41dd
linting
johnnyaug Jun 14, 2021
196d946
revert typo
johnnyaug Jun 14, 2021
edff49c
finish api wiring
johnnyaug Jun 14, 2021
0d1db93
fix sorting in test
johnnyaug Jun 14, 2021
8d1b96f
gen files
johnnyaug Jun 14, 2021
88edd93
rules api
johnnyaug Jun 15, 2021
c8de6a5
gen
johnnyaug Jun 15, 2021
6de0637
Merge branch 'master' into feature/find_active_commits
johnnyaug Jun 15, 2021
759f5fe
gen
johnnyaug Jun 15, 2021
b18aca6
use config
johnnyaug Jun 15, 2021
63ff3b0
Merge branch 'master' into feature/find_active_commits
johnnyaug Jun 16, 2021
3acef2f
CR fixes: fix abstraction leaks, naming changes
johnnyaug Jun 16, 2021
d02f0c7
CR fixes
johnnyaug Jun 20, 2021
7ce29bc
CR fixes
johnnyaug Jun 20, 2021
a7c306a
gen clients
johnnyaug Jun 20, 2021
b5e6447
cr fixes
johnnyaug Jun 20, 2021
111cf0c
Merge branch 'master' into feature/find_active_commits
johnnyaug Jun 20, 2021
ea99529
cr fix
johnnyaug Jun 20, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,8 @@ clean:
pkg/ddl/statik.go \
pkg/graveler/sstable/mock \
pkg/webui \
pkg/graveler/committed/mock
pkg/graveler/committed/mock \
pkg/graveler/mock

check-licenses: check-licenses-go-mod check-licenses-npm

Expand Down Expand Up @@ -137,6 +138,7 @@ gen-api: go-install ## Run the swagger code generator
gen-mockgen: go-install ## Run the generator for inline commands
$(GOGENERATE) ./pkg/graveler/sstable
$(GOGENERATE) ./pkg/graveler/committed
$(GOGENERATE) ./pkg/graveler
$(GOGENERATE) ./pkg/pyramid
$(GOGENERATE) ./pkg/onboard
$(GOGENERATE) ./pkg/actions
Expand Down
115 changes: 115 additions & 0 deletions api/swagger.yml
Original file line number Diff line number Diff line change
Expand Up @@ -817,6 +817,48 @@ components:
- checksum
- size_bytes

GarbageCollectionPrepareRequest:
type: object
properties:
previous_run_id:
type: string
description: run id of a previous successful GC job
example: 64eaa103-d726-4a33-bcb8-7c0b4abfe09e

GarbageCollectionPrepareResponse:
type: object
properties:
run_id:
type: string
description: a unique identifier generated for this GC job
example: 64eaa103-d726-4a33-bcb8-7c0b4abfe09e
required:
- run_id

GarbageCollectionRule:
type: object
properties:
branch_id:
type: string
retention_days:
type: integer
required:
- branch_id
- retention_days

GarbageCollectionRules:
type: object
properties:
default_retention_days:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the relation between a rule's retention_days and the global default_retention_days?

type: integer
branches:
type: array
items:
$ref: '#/components/schemas/GarbageCollectionRule'
required:
- default_retention_days
- branches

paths:
/setup_lakefs:
post:
Expand Down Expand Up @@ -2829,7 +2871,80 @@ paths:
$ref: "#/components/responses/NotFound"
default:
$ref: "#/components/responses/ServerError"
/repositories/{repository}/gc/rules:
parameters:
- in: path
name: repository
required: true
schema:
type: string
get:
tags:
- retention
operationId: getGarbageCollectionRules
responses:
nopcoder marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing 404 for missing repo.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

200:
description: gc rule list
content:
application/json:
schema:
$ref: "#/components/schemas/GarbageCollectionRules"
401:
$ref: "#/components/responses/Unauthorized"
404:
$ref: "#/components/responses/NotFound"
default:
$ref: "#/components/responses/ServerError"
post:
tags:
- retention
operationId: set garbage collection rules
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/GarbageCollectionRules"
responses:
nopcoder marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing 404.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

204:
description: set garbage collection rules successfully
401:
$ref: "#/components/responses/Unauthorized"
404:
$ref: "#/components/responses/NotFound"
default:
$ref: "#/components/responses/ServerError"

/repositories/{repository}/gc/prepare_commits:
parameters:
- in: path
name: repository
required: true
schema:
type: string
post:
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/GarbageCollectionPrepareRequest"
tags:
- retention
operationId: prepareGarbageCollectionCommits
summary: save lists of active and expired commits for garbage collection
responses:
201:
description: paths to commit dataset
content:
application/json:
schema:
$ref: "#/components/schemas/GarbageCollectionPrepareResponse"
401:
$ref: "#/components/responses/Unauthorized"
404:
$ref: "#/components/responses/NotFound"
default:
$ref: "#/components/responses/ServerError"
/healthcheck:
get:
operationId: healthCheck
Expand Down
7 changes: 7 additions & 0 deletions clients/java/README.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

178 changes: 178 additions & 0 deletions clients/java/api/openapi.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 13 additions & 0 deletions clients/java/docs/GarbageCollectionCommits.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 13 additions & 0 deletions clients/java/docs/GarbageCollectionPrepareRequest.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading