Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

⚖️📄 Compare with reproduction results, if available #642

Merged
merged 42 commits into from
Nov 24, 2021

Conversation

mberr
Copy link
Member

@mberr mberr commented Nov 17, 2021

This PR compares the results obtained by the reproducibility pipeline to those present in the configuration

  • extract metric names & values from configuration
  • PipelineResult.get_metric cannot resolve some metric names as present in the configuration, e.g., "hits_at_k.best.10". Update metric name resolution
  • create and store a dataframe with the original and reproduced metric values
  • summarize difference with significance check

@mberr
Copy link
Member Author

mberr commented Nov 17, 2021

@cthoyt any thoughts about the metric name resolution?

@mberr
Copy link
Member Author

mberr commented Nov 17, 2021

I made a new utility in b6132f3 (functional, i.e. we do not need to instantiate a RankBasedMetricResults)

src/pykeen/pipeline/api.py Outdated Show resolved Hide resolved
src/pykeen/pipeline/api.py Outdated Show resolved Hide resolved
@cthoyt
Copy link
Member

cthoyt commented Nov 17, 2021

I think I'd go for updating to be able to handle hits_at_k.best.10... not sure how those got out of sync

@cthoyt
Copy link
Member

cthoyt commented Nov 17, 2021

can you show what the output looks like now when this is run?

src/pykeen/pipeline/api.py Outdated Show resolved Hide resolved
Copy link
Member

@cthoyt cthoyt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs an emoji, then merge when ready

@mberr mberr changed the title Compare with reproduction results, if available ⚖️📄 Compare with reproduction results, if available Nov 18, 2021
@mberr mberr requested a review from cthoyt November 18, 2021 10:44
src/pykeen/pipeline/api.py Outdated Show resolved Hide resolved
Copy link
Member

@cthoyt cthoyt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LSTM!

@mberr mberr merged commit 8281200 into master Nov 24, 2021
@mberr mberr deleted the compare-results branch November 24, 2021 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants