Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SBC-ECDF interpretation guide #236

Closed
elseml opened this issue Nov 4, 2024 · 3 comments
Closed

Add SBC-ECDF interpretation guide #236

elseml opened this issue Nov 4, 2024 · 3 comments
Assignees
Labels
documentation Improvements or additions to documentation

Comments

@elseml
Copy link
Member

elseml commented Nov 4, 2024

Initial thoughts by @marvinschmitt:

Minimal content would be examples of ECDF plots for

  • bias too high
  • bias: too low
  • overconfident
  • underconfident

Alternatively, I suggest that we simply point to this excellent blog post (in tutorials and even in the documentation): https://hyunjimoon.github.io/SBC/articles/rank_visualizations.html
In my opinion, it covers all important cases and clearly illustrates the advantages of ECDF plots vs. rank histograms as well as ECDF diff plots vs. standard ECDF plots.

@elseml elseml converted this from a draft issue Nov 4, 2024
@marvinschmitt
Copy link
Member

Thanks for initiating the discussion, Lasse! I agree that we should link to this great blog post in multiple places. Yet, I think that users might appreciate a gist of how to interpret SBC(-ECDF) plots even directly in the bayesflow tutorial resources.

Do you think that there is some minimal viable SBC interpretation guide that would be appropriate within a bayesflow tutorial notebook, which would then link to the blog post for more details?

@marvinschmitt marvinschmitt added the documentation Improvements or additions to documentation label Nov 4, 2024
@elseml
Copy link
Member Author

elseml commented Nov 6, 2024

I agree that it would be cool to provide more guidance in the BayesFlow context. We could take a tutorial notebook that consistently generates a well-calibrated posterior with simple shape (e.g., Gaussian), apply perturbations similar to the blog post (e.g., see lines 160-168), show the resulting rank histograms + ECDF plots with the built-in BayesFlow diagnostics, and refer to the blog post for further reading. The issue I see is that we would do more or less the same as the blog post while running the risk of bloating the tutorial materials, so we can explore whether it is feasible with minimal code.

@paul-buerkner
Copy link
Contributor

I agree that pointing to the notebook may be sufficient. @elseml would you perhaps add this link at the right places in some of our basic example notebooks?

vpratz added a commit that referenced this issue Dec 14, 2024
Add reference to SBC interpretation guide in starter examples (#236)
@vpratz vpratz closed this as completed Dec 14, 2024
@github-project-automation github-project-automation bot moved this from Future to Done in bayesflow development Dec 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
Status: Done
Development

No branches or pull requests

4 participants