Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update/clarify S3 gateway config notes #5390

Merged
merged 3 commits into from
Apr 17, 2023
Merged

Update/clarify S3 gateway config notes #5390

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
af6f042
Update/clarify S3 gateway config notes
Mar 6, 2023
612d94a
Move ssl note to new section
Mar 28, 2023
fd743d2
Refine presentation of s3_use_ssl
rmoff Apr 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/integrations/duckdb.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,34 @@ Querying data in lakeFS from DuckDB is similar to querying data in S3 from DuckD

### Configure the lakeFS S3 Gateway endpoint

If not loaded already, install and load the HTTPFS extension:

```sql
INSTALL httpfs;
LOAD httpfs;
```

Then run the following to configure the connection.

```sql
SET s3_region='us-east-1';
SET s3_endpoint='lakefs.example.com';
SET s3_access_key_id='AKIAIOSFODNN7EXAMPLE';
SET s3_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY';
SET s3_url_style='path';
```

* `s3_endpoint` is the host (and port, if necessary) of your lakeFS server
* `s3_access_key_id` and `s3_secret_access_key` are the access credentials for your lakeFS user
* `s3_url_style` needs to be set to `path`
* `s3_region` is the S3 region on which your bucket resides. If local storage, or not S3, then just set it to `us-east-1`.

If lakeFS is not running on an SSL-secured server (for example, you are just running it locally) then you need to also run

```sql
SET s3_use_ssl=false;
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion:
Can we move this one to the "configure the connection" section as a comment just as example on what to set in case we are working with a secure connect (same as your comment).
It just more for the structure as we have section with all the settings and the following explanation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nopcoder is this the kind of thing you had in mind? If not please can you explain further or suggest the change you have in mind as I'm not clear. Thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I wasn't clear. Ment something simplier - just having the following included with the above configuration code.
Something like you see in other configuration files where you have all the options with comments saying what is the default and if you need to enable something to change the default behaviour.

-- Uncomment in case the endpoint listen on non-secure, for example running lakeFS locally.
-- SET s3_use_ssl=false;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah ok, thanks. updated per your suggestion.


### Querying Data

Once configured, you can query data using the lakeFS S3 Gateway using the following URI pattern:
Expand Down