Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-7420][VL] Fix GCS configuration #7421

Merged
merged 1 commit into from
Oct 16, 2024
Merged

[GLUTEN-7420][VL] Fix GCS configuration #7421

merged 1 commit into from
Oct 16, 2024

Conversation

majetideepak
Copy link
Contributor

@majetideepak majetideepak commented Oct 7, 2024 •
edited
Loading

What changes were proposed in this pull request?

We must directly set the gcs credential path in the config.
Add GCS configs to the backend config if present.

Fixes: #7420

How was this patch tested?

Tested locally.

@github-actions github-actions bot added the VELOX label Oct 7, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 7, 2024

#7420

@PHILO-HE
Copy link
Contributor

PHILO-HE commented Oct 9, 2024

@pratham76

@majetideepak
Copy link
Contributor Author

@PHILO-HE, @zhouyuan I need your help to plumb the GCS configs. Velox is not seeing the GCS configs.
I see the GCS configs appear in the VeloxRuntime session config. Should these appear in the VeloxBackend config instead?
Should I be adding them in GlutenConfig.scala?

@zhouyuan
Copy link
Contributor

@majetideepak can you please have a quick try following S3 configuration? I assume it may need to be fetched from scala side
https://github.com/apache/incubator-gluten/blob/main/shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala#L739

@github-actions github-actions bot added the CORE works for Gluten Core label Oct 15, 2024
@majetideepak majetideepak changed the title [GLUTEN-7420][VL] Fix kGCSCredentialsPath configuration [GLUTEN-7420][VL] Fix GCS configuration Oct 15, 2024
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@majetideepak
Copy link
Contributor Author

@pratham76 Can you verify this fix again? It worked for me locally.

@pratham76
Copy link
Contributor

pratham76 commented Oct 15, 2024 •
edited
Loading

@pratham76 Can you verify this fix again? It worked for me locally.

Sure, will verify and update

@majetideepak majetideepak marked this pull request as ready for review October 15, 2024 11:30
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

zhouyuan
zhouyuan approved these changes Oct 16, 2024
View reviewed changes
Copy link
Contributor

@zhouyuan zhouyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@pratham76
Copy link
Contributor

pratham76 commented Oct 16, 2024 •
edited
Loading

Have verified this change, GCS reads are working. Thanks @majetideepak

PHILO-HE
PHILO-HE approved these changes Oct 16, 2024
View reviewed changes
Copy link
Contributor

@PHILO-HE PHILO-HE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@majetideepak, thanks for your fix!

@PHILO-HE PHILO-HE merged commit fd2805f into apache:main Oct 16, 2024
47 checks passed
@majetideepak majetideepak deleted the fix-gcs branch October 17, 2024 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PHILO-HE PHILO-HE PHILO-HE approved these changes

@zhouyuan zhouyuan zhouyuan approved these changes

Assignees
No one assigned
Labels
CORE works for Gluten Core VELOX
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Fix kGCSCredentialsPath configuration
4 participants
@majetideepak @PHILO-HE @zhouyuan @pratham76