Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Merged by Bors] - fix resource limit usage #166

Closed
wants to merge 22 commits into from
Closed

[Merged by Bors] - fix resource limit usage #166

wants to merge 22 commits into from

Conversation

adwk67
Copy link
Member

@adwk67 adwk67 commented Oct 19, 2022 •
edited
Loading

Description

  • Restores setting # of executor instances
  • Define spark conf settings as well as defining the pod template
    • set kuberenetes.driver/executor.* from CRD directly
    • use cpu limit to set driver/executor.cores
  • Integration test to test/illustrate
    • just resource limits
    • just spark.conf settings
    • just defaults

Fixes #161.
Jenkins: https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/spark-k8s-operator-it-custom/41/

Review Checklist

  • Code contains useful comments
  • CRD change approved (or not applicable)
  • (Integration-)Test cases added (or not applicable)
  • Documentation added (or not applicable)
  • Changelog updated (or not applicable)
  • Cargo.toml only contains references to git tags (not specific commits or branches)
  • Helm chart can be installed and deployed operator works (or not applicable)

Once the review is done, comment bors r+ (or bors merge) to merge. Further information

@adwk67 adwk67 marked this pull request as ready for review October 19, 2022 13:39
sbernauer
sbernauer reviewed Oct 20, 2022
View reviewed changes
Copy link
Member

@sbernauer sbernauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM overall
I'm going to give it a try by using it within the warehouse demo

rust/crd/src/lib.rs Show resolved Hide resolved
rust/crd/src/lib.rs Outdated Show resolved Hide resolved
rust/crd/src/lib.rs Outdated Show resolved Hide resolved
rust/crd/src/lib.rs Outdated Show resolved Hide resolved
rust/crd/src/lib.rs Outdated Show resolved Hide resolved
rust/crd/src/lib.rs Outdated Show resolved Hide resolved
rust/crd/src/lib.rs Show resolved Hide resolved
tests/templates/kuttl/resources/12-deploy-spark-app.yaml.j2 Show resolved Hide resolved
tests/templates/kuttl/resources/12-deploy-spark-app.yaml.j2 Show resolved Hide resolved
adwk67 and others added 9 commits October 20, 2022 09:21
@adwk67 @sbernauer
Update rust/crd/src/lib.rs
ae99ff6 
Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>
@adwk67 @sbernauer
Update rust/crd/src/lib.rs
adc6a96 
Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>
@adwk67 @sbernauer
Update rust/crd/src/lib.rs
8655a5d 
Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>
@adwk67 @sbernauer
Update rust/crd/src/lib.rs
ed1f74a 
Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>
@adwk67
automatic link
3e500db
@adwk67
Merge branch 'main' into 161_fix_resources
c90638f
@adwk67
renamed function
4e49926
@adwk67 @sbernauer
Update tests/templates/kuttl/resources/12-deploy-spark-app.yaml.j2
1414b9e 
Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>
@adwk67 @sbernauer
Update tests/templates/kuttl/resources/12-deploy-spark-app.yaml.j2
047b0bc 
Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>
@sbernauer
Copy link
Member

Tests it out and seemed to work fine 👍

sbernauer
sbernauer approved these changes Oct 20, 2022
View reviewed changes
Copy link
Member

@sbernauer sbernauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % remaining two comments. worked fine 👍

sbernauer
sbernauer approved these changes Oct 20, 2022
View reviewed changes
rust/crd/src/lib.rs Outdated Show resolved Hide resolved
@adwk67 @sbernauer
Update rust/crd/src/lib.rs
0a7de8c 
Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>
@adwk67
Copy link
Member Author

adwk67 commented Oct 20, 2022

bors merge

bors bot pushed a commit that referenced this pull request Oct 20, 2022
@adwk67
fix resource limit usage (#166)
3425391 
# Description

- Restores setting # of executor instances
- Define spark conf settings as well as defining the pod template
  - set kuberenetes.driver/executor.* from CRD directly
  - use cpu limit to set driver/executor.cores
- Integration test to test/illustrate
  - just resource limits
  - just spark.conf settings
  - just defaults
  
Fixes #161.
Jenkins: https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/spark-k8s-operator-it-custom/41/
@bors
Copy link

bors bot commented Oct 20, 2022

Pull request successfully merged into main.

Build succeeded:

@bors bors bot changed the title fix resource limit usage [Merged by Bors] - fix resource limit usage Oct 20, 2022
@bors bors bot closed this Oct 20, 2022
@bors bors bot deleted the 161_fix_resources branch October 20, 2022 09:55
@sbernauer sbernauer mentioned this pull request Nov 4, 2022
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sbernauer sbernauer sbernauer approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Compute resources and number of executors are not set
2 participants
@adwk67 @sbernauer