Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create Job template for DataChain workers #486

Draft
wants to merge 23 commits into
base: main
Choose a base branch
from
Draft

Create Job template for DataChain workers #486

wants to merge 23 commits into from

Conversation

0x2b3bfa0
Copy link
Member

@0x2b3bfa0 0x2b3bfa0 commented Oct 24, 2024 •
edited
Loading

Isolation To Do

  • Set runtimeClassName to gvisor
  • Review default Network Policy with gvisor isolation or create custom one
  • Review persistent volumes architecture (NFS + local)
  • Review access to secrets and config maps
  • Review cold startup times for new jobs & find way to improve
  • Consider pod-per-node performance mode in GKE

Studio To Do

  • Get rid of Celery and move job polling logic to workers

Distributed To Do

  • Run jobs up to quota for each submssions (stop on quota exception)
  • Set idle timeout to 1 hour
  • Set quota to 6 or 10
  • Replace all the nested queue nonsense with column.sys__rand % parallelism

@0x2b3bfa0 0x2b3bfa0 changed the title Create job-studio-datachain-worker.yaml Create Job template for DataChain workers Oct 24, 2024
0x2b3bfa0 and others added 22 commits October 24, 2024 04:52
@0x2b3bfa0
Update role-datachain-worker.yaml
4bb1843
@0x2b3bfa0
Update Chart.yaml
90ae9b8
@0x2b3bfa0
Merge remote-tracking branch 'origin/main' into 0x2b3bfa0-patch-2
dbd053d
@0x2b3bfa0
Rename Job ResourceQuota template
a4c4fb3
@0x2b3bfa0
Add studioDatachainWorkerJob section
f72c39a
@0x2b3bfa0
fixup! Add studioDatachainWorkerJob section
e93004d
@0x2b3bfa0
fixup! Add studioDatachainWorkerJob section
9143b05
@0x2b3bfa0
Add runtimeClassName for GKE Sandbox
210ec29
@0x2b3bfa0 @github-actions
Helm-Docs update
32e93a2
@0x2b3bfa0
fixup! Add runtimeClassName for GKE Sandbox
9d02dd9
@0x2b3bfa0
fixup! Add studioDatachainWorkerJob section
b6e007d
@0x2b3bfa0
fixup! Add studioDatachainWorkerJob section
90a9572
@0x2b3bfa0
Run only the worker code on Jobs
d493911
@0x2b3bfa0
Remove volumes from DataChain master
f65fc71
@0x2b3bfa0
fixup! Run only the worker code on Jobs
909f424
@0x2b3bfa0 @github-actions
Helm-Docs update
f741090
@0x2b3bfa0
Prune unused/wrong parts of the job template
f899478
@0x2b3bfa0 @github-actions
Helm-Docs update
bc9ca25
@0x2b3bfa0
Bump version on Chart.yaml
0bad6a7
@0x2b3bfa0 @github-actions
Helm-Docs update
2882cd5
@0x2b3bfa0
Fix comment spacing
344c565
@0x2b3bfa0
Delete unused/undesired blobvault mounts
8fe1980
@0x2b3bfa0 0x2b3bfa0 self-assigned this Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@0x2b3bfa0 0x2b3bfa0

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@0x2b3bfa0