Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add method comparison in training script #65

Merged
merged 2 commits into from
Oct 9, 2024
Merged

Conversation

SCiarella
Copy link
Member

This PR address #63 by testing which loss function approach we should use between Lux and Optim.jl.

A visual comparison:
plot_153

Observations:

  • Lux is ~25% faster
  • Lux is slightly more accurate, however notice that both are bad so further testing of our overall a-posteriori approach is required (maybe this cnn is too weak?)
  • Optim can use fancy optimization algorithms like CMA or LBFGS

Conclusions:

  • Training with Lux should be our main route
  • We can test fancy optimization algorithms with Optim if we put this loss in the package
  • Overall, we have no example where the a-posteriori training shows any sensible improvement over random model weights

@SCiarella SCiarella requested a review from luisaforozco October 7, 2024 14:37
θ_posteriori_lux = tstate.parameters

# Function to test the results
function validate_results(p, dataloader, nuse = 100)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering if this validation an plotting should be in this file or in evaluate_posteriori.jl, have you thought about that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice, however for the comparison we need to access the full training routine, while the idea of evaluate_posteriori is to just load a trained model.

I think that this comparison should go to the explorations repo, but we include both loss functions to src and we show only the Lux one in train_posteriori. What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I like the idea:

  1. copy the script as it is to explorations (still with the loss defined in it for clarity).
  2. adding the code to src/loss/loss_posteriori.jl.

move comparison to explorations repo
@SCiarella SCiarella requested a review from luisaforozco October 8, 2024 12:15
@luisaforozco
Copy link
Contributor

I'm ok with this! I'm merging it.

@luisaforozco luisaforozco merged commit 9c3f4cc into main Oct 9, 2024
4 checks passed
@luisaforozco luisaforozco deleted the test_Lux_train branch October 9, 2024 14:12
v1kko pushed a commit that referenced this pull request Jan 16, 2025
Loss for Optim
* add method comparison in training script
* put optim loss in src
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants