-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Will the whole mixture activation require OP/Fl(linear, convolution, or attention)? #20
Comments
Perhaps I don't quite understand this Figure 3 and how to achieve activation interaction while reducing the computational load of each device. I hope to receive guidance. Thank you very much. |
This is an illustration for better understanding. Only a minor portion of stale activation gets involved in the computation. For example, if Fl is a convolutional layer, then only the boundaries of the stale activations get involved in the computation. The majority part does not even instantiate. You can refer to our code for more details. |
As the given Figure 3 in the paper shown, while through the Scatter operation, we will get the whole mixture activation including old and new activation of patch. And the Figure 3 show that the mixture activation will perform OP/Fl in the next step. If that's the case, then isn't it true that each device in each layer performs OP/Fl on the entire activation? Isn't this enough to reduce the computational load on each device? If the old activation of the patch does not participate in the OP/Fl computation, how can new and old activation be fused and learned?
The text was updated successfully, but these errors were encountered: