Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

local block adaptor import support #5277

Merged
merged 12 commits into from
Feb 21, 2023
Merged

local block adaptor import support #5277

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
2900462
local block adaptor import support
nopcoder Feb 19, 2023
cb9634f
apply code review changes
nopcoder Feb 19, 2023
0640c71
local cache information for import performance
nopcoder Feb 19, 2023
ba1d755
fix lint errors
nopcoder Feb 19, 2023
c49915b
integration test for default import
nopcoder Feb 19, 2023
17aad65
remove unused and reuse block type as prefix
nopcoder Feb 20, 2023
24c1706
Update docs/reference/configuration.md
nopcoder Feb 20, 2023
5af3cc7
calc hash as part of iteration
nopcoder Feb 20, 2023
a8e28dd
using constant for magic numbers
nopcoder Feb 20, 2023
c1c1904
fix lint err on md5
nopcoder Feb 20, 2023
d03e193
Skip hidden files by default
nopcoder Feb 21, 2023
0e5ff7f
Update import_hidden parameter
nopcoder Feb 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions api/swagger.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,12 @@ components:
application/json:
schema:
$ref: "#/components/schemas/Error"
Forbidden:
description: Access is forbidden
content:
application/json:
schema:
$ref: "#/components/schemas/Error"
NotFound:
description: Resource Not Found
content:
Expand Down Expand Up @@ -944,6 +950,7 @@ components:
- blockstore_namespace_example
- blockstore_namespace_ValidityRegex
- pre_sign_support
- import_support
properties:
blockstore_type:
type: string
Expand All @@ -955,6 +962,8 @@ components:
type: string
pre_sign_support:
type: boolean
import_support:
type: boolean

VersionConfig:
type: object
Expand Down Expand Up @@ -3172,6 +3181,8 @@ paths:
$ref: "#/components/responses/ValidationError"
401:
$ref: "#/components/responses/Unauthorized"
403:
$ref: "#/components/responses/Forbidden"
404:
$ref: "#/components/responses/NotFound"
default:
Expand Down
16 changes: 16 additions & 0 deletions clients/java/api/openapi.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions clients/java/docs/ImportApi.md
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions clients/java/docs/StorageConfig.md
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions clients/java/src/main/java/io/lakefs/clients/api/ImportApi.java
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

33 changes: 31 additions & 2 deletions clients/java/src/main/java/io/lakefs/clients/api/model/StorageConfig.java
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions clients/java/src/test/java/io/lakefs/clients/api/model/StorageConfigTest.java
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions clients/python/docs/ImportApi.md
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions clients/python/docs/StorageConfig.md
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 8 additions & 2 deletions clients/python/lakefs_client/model/storage_config.py
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions cmd/lakefs/cmd/run.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -458,6 +458,7 @@ func checkMetadataPrefix(ctx context.Context, repo *catalog.Repository, logger l
if _, err := adapter.Get(ctx, block.ObjectPointer{
StorageNamespace: repo.StorageNamespace,
Identifier: dummyFile,
IdentifierType: block.IdentifierTypeRelative,
}, -1); err != nil {
logger.WithFields(logging.Fields{
"path": dummyFile,
Expand Down
11 changes: 11 additions & 0 deletions docs/assets/js/swagger.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,12 @@ components:
application/json:
schema:
$ref: "#/components/schemas/Error"
Forbidden:
description: Access is forbidden
content:
application/json:
schema:
$ref: "#/components/schemas/Error"
NotFound:
description: Resource Not Found
content:
Expand Down Expand Up @@ -944,6 +950,7 @@ components:
- blockstore_namespace_example
- blockstore_namespace_ValidityRegex
- pre_sign_support
- import_support
properties:
blockstore_type:
type: string
Expand All @@ -955,6 +962,8 @@ components:
type: string
pre_sign_support:
type: boolean
import_support:
type: boolean

VersionConfig:
type: object
Expand Down Expand Up @@ -3172,6 +3181,8 @@ paths:
$ref: "#/components/responses/ValidationError"
401:
$ref: "#/components/responses/Unauthorized"
403:
$ref: "#/components/responses/Forbidden"
404:
$ref: "#/components/responses/NotFound"
default:
Expand Down
2 changes: 2 additions & 0 deletions docs/reference/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,8 @@ This reference uses `.` to denote the nesting of values.
If specified, the storage namespace will be filled with this default value as a prefix when creating a repository from the UI.
The user may still change it to something else.
* `blockstore.local.path` `(string: "~/lakefs/data")` - When using the local Block Adapter, which directory to store files in
* `blockstore.local.import_enabled` `(bool: false)` - Enable import for local Block Adapter, relevant if you are using shared location
nopcoder marked this conversation as resolved.
Show resolved Hide resolved
* `blockstore.local.allowed_external_prefixes` `([]string: [])` - List of prefixes used to match any access for external location used by local block adapter
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this interact with the storage namespaces? Should sn be included here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the description - let me know if it is clear now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you push?

* `blockstore.gs.credentials_file` `(string : )` - If specified will be used as a file path of the JSON file that contains your Google service account key
* `blockstore.gs.credentials_json` `(string : )` - If specified will be used as JSON string that contains your Google service account key (when credentials_file is not set)
* `blockstore.gs.pre_signed_expiry` `(time duration : "15m")` - Expiry of pre-signed URL.
Expand Down
Loading