Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pass dependencies in schedule jobs #2379

Merged
merged 7 commits into from
Aug 19, 2024
Merged

Pass dependencies in schedule jobs #2379

merged 7 commits into from
Aug 19, 2024

Conversation

kartik4949
Copy link
Collaborator

Description

fix ##2329

Related Issues

Checklist

  • Is this code covered by new or existing unit tests or integration tests?
  • Did you run make unit_testing and make integration-testing successfully?
  • Do new classes, functions, methods and parameters all have docstrings?
  • Were existing docstrings updated, if necessary?
  • Was external documentation updated, if necessary?

Additional Notes or Comments

@kartik4949 kartik4949 marked this pull request as draft August 9, 2024 06:56
@kartik4949 kartik4949 force-pushed the feat/component/startup_jobs branch 6 times, most recently from 7021ff4 to 3c225e9 Compare August 16, 2024 16:42
@kartik4949 kartik4949 marked this pull request as ready for review August 16, 2024 16:49
blythed
blythed requested changes Aug 16, 2024
View reviewed changes
superduper/components/vector_index.py Outdated Show resolved Hide resolved
@kartik4949 kartik4949 force-pushed the feat/component/startup_jobs branch from 3c225e9 to 5409cd7 Compare August 17, 2024 09:15
blythed
blythed requested changes Aug 18, 2024
View reviewed changes
Copy link
Collaborator

@blythed blythed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need a solution for initial outputs computed in Listener. I don't see evidence for that in your test-case.

test/unittest/component/test_component.py
)

db.apply(m)
assert m.upstream.triggered_schedule_jobs == True # noqa
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok nice test.

test/unittest/component/test_listener.py

db.execute(table.insert(data))

# Insert data
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add comments - why is this a useful test? Can we now solve the problem mentioned on the issue (https://github.com/orgs/superduper-io/projects/1?pane=issue&itemId=71398398)?

test/unittest/component/test_listener.py Outdated
deps, features = db.apply(features_listener)

trainable_model.trainer = MyTrainer(
'test', select=features_listener.outputs_select, key=features_listener.outputs
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not what we want to test. We need a single db.apply.

blythed
blythed reviewed Aug 18, 2024
View reviewed changes
test/unittest/component/test_listener.py
@@ -172,3 +189,53 @@ def test_listener_cleanup(db, data):

db.remove('listener', listener1.identifier, force=True)
assert not db.databackend.check_output_dest(listener1.predict_id)


def test_listener_chaining_with_trainer(db):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do db.apply for a single Application, not multiple db.apply calls.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

jieguangzhou
jieguangzhou reviewed Aug 19, 2024
View reviewed changes
superduper/jobs/queue.py
Comment on lines +197 to +207
if from_type == 'DB':
ids = [event.id for event in events]
else:
ids = []
for event in events:
ids += event.id
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the logic the same?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if from_type is not DB i.e COMPONENT
this means its a startup job and hence it will have all ids into one event.

test/unittest/component/test_component.py Show resolved Hide resolved
superduper/jobs/queue.py
Comment on lines +168 to +181
consumers = self._get_consumers(db, components)
# All consumer are executed one by one.
for consumer in consumers:
events = queue[consumer]
queue[consumer] = []
component = components[consumer]
# Consume
jobs = consume_events(db, component=component, events=events)
queue_jobs[consumer].extend(jobs)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m not sure if this will break the dependencies or the running order

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@kartik4949 kartik4949 force-pushed the feat/component/startup_jobs branch from 5409cd7 to a8e9335 Compare August 19, 2024 09:10
@kartik4949 kartik4949 force-pushed the feat/component/startup_jobs branch from a8e9335 to 7c459d3 Compare August 19, 2024 09:54
blythed
blythed reviewed Aug 19, 2024
View reviewed changes
test/unittest/component/test_listener.py Outdated
identifier="listener1",
uuid="listener1",
)
features_listener.db = db
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't be necessary with db.apply

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

features_listener.outputs_select
will fail

blythed
blythed approved these changes Aug 19, 2024
View reviewed changes
Copy link
Collaborator

@blythed blythed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the one comment before merging.

@kartik4949 kartik4949 force-pushed the feat/component/startup_jobs branch from 7c459d3 to d24d3df Compare August 19, 2024 10:52
@kartik4949 kartik4949 force-pushed the feat/component/startup_jobs branch from d24d3df to a1f61e4 Compare August 19, 2024 10:57
@kartik4949 kartik4949 force-pushed the feat/component/startup_jobs branch from ff1b95d to c04795c Compare August 19, 2024 11:03
@kartik4949 kartik4949 merged commit 1752078 into superduper-io:main Aug 19, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jieguangzhou jieguangzhou jieguangzhou approved these changes

@blythed blythed blythed approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kartik4949 @blythed @jieguangzhou