Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs on freezing layers should be ported to the explicit syntax #2216

Open
CarloLucibello opened this issue Mar 20, 2023 · 3 comments
Open

Comments

@CarloLucibello
Copy link
Member

https://fluxml.ai/Flux.jl/stable/models/advanced/#Freezing-Layer-Parameters
should be reworded on the line of
https://fluxml.ai/Optimisers.jl/dev/#Frozen-Parameters

@ghost
Copy link

ghost commented Mar 23, 2023

Hi! i added some changes on the docs.
Here's the PR: FluxML/Optimisers.jl#133

@mcabbott
Copy link
Member

That file is a bit of a random grab-bag of things, I wonder how useful it is to anyone? It's headed "Customised Layers" and maybe there could be a nice description of how & why to make layers, somewhere.

Besides that, it mentions feezing, and links to the description at http://fluxml.ai/Flux.jl/stable/training/training/#Freezing-and-Schedules Maybe that section can just be deleted?

And it mentions trainable, which is a better fit (as it permanently customises layer definitions). This has a docstring at http://fluxml.ai/Flux.jl/stable/destructure/#Optimisers.trainable but isn't described elsewhere.

@mcabbott
Copy link
Member

Re freezing, one thing you might be tempted to use freeze! for is roughly transfer learning: Fixing all but the last few layers, to train only those. That will work but is quite inefficient (and the old Params equivalent was too). What you should do is only differentiate part of the model, something like this: https://github.com/FluxML/model-zoo/blob/master/tutorials/transfer_learning/transfer_learning.jl#L117-L120

We don't say that in the docs anywhere right now. And guessing it requires quite a bit of understanding of how Zygote works. So perhaps we should... but where I don't know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants