Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose job resource in batch API #329

Merged
merged 3 commits into from
Feb 22, 2024
Merged

Conversation

illusional
Copy link

@illusional illusional commented Feb 21, 2024

It's useful for us to be able to programmatically access the resource data from a job and batches.
This is stored as a dataframe, we'll send back as json with orient='split'.

I've tested this in a dev deploy, and worked fairly well.

In my own interest, here's how to convert it back into a dataframe:

import pandas as pd

response = {} # response from Hail Batch
dataframes = {
    key: pd.DataFrame(data=values['data'], columns=values['columns'])
    for key, values in response.items()
}

@illusional illusional marked this pull request as ready for review February 21, 2024 04:58
Copy link

@EddieLF EddieLF left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it

@illusional
Copy link
Author

I have suggested this upstream, as it reduces the amount of conflicts we might receive, and might be useful for them: hail-is#14328

Copy link

@jmarshall jmarshall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of very minor comments, otherwise looks good!

batch/batch/front_end/front_end.py Outdated Show resolved Hide resolved
batch/batch/front_end/front_end.py Outdated Show resolved Hide resolved
Co-authored-by: John Marshall <john.marshall@populationgenomics.org.au>
@illusional illusional merged commit 8f6797b into main Feb 22, 2024
5 checks passed
@illusional illusional deleted the expose-job-resource-usage branch February 22, 2024 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants