Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add weight sharing #169

Closed
seanmor5 opened this issue Nov 28, 2021 · 2 comments · Fixed by #524
Closed

Add weight sharing #169

seanmor5 opened this issue Nov 28, 2021 · 2 comments · Fixed by #524
Labels
area:layers Applies to layer API kind:feature New feature or request note:discussion Details or approval are up for discussion

Comments

@seanmor5
Copy link
Contributor

There is currently no explicit way to use the same weights/layer for separate inputs. For example, in Keras I could do:

dense = tf.keras.layers.Dense(32)

d_x1 = dense(x1)
d_x2 = dense(x2)

And it would use the same layer for both operations. I believe this could potentially work for now:

x1 = Axon.input({nil, 32})
x2 = Axon.input({nil, 32})

d_x1 = Axon.dense(x1, 64, name: "dense")
d_x2 = Axon.dense(x2, 64, name: "dense")

And the parameters would be shared by name. Perhaps there is a way to make this a bit more explicit. This somewhat relates to some of the issues that appeared earlier with GANs and the proposed Axon.function

@seanmor5 seanmor5 added area:layers Applies to layer API kind:feature New feature or request note:discussion Details or approval are up for discussion labels Nov 28, 2021
@josevalim
Copy link
Contributor

Maybe an alternative is to accept a params option with the name of a previous layer as a value.

@seanmor5
Copy link
Contributor Author

With the introduction of namespaces and the new custom layer API there are a few additional open questions:

  1. How do we deal with duplicate usage of the same param (it will have the same ID) in two separate custom layers. To me the parameters should be tied, it is the same parameter!
  2. How do we deal with duplicate usage of the same namespace? If I define two namespaces (with different layers), and then give them the same name, what should be the intended behavior?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:layers Applies to layer API kind:feature New feature or request note:discussion Details or approval are up for discussion
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants