Skip to content

Releases: Rishit-dagli/Nystromformer

Nystromformer v0.1.0

20 Aug 18:09
Compare
Choose a tag to compare

This is the initial release of Nystromformer and implements Nystromformer as a subclassed TensorFlow model.

  • MoorePenrosePseudoinverse: Implement the Moore Penrose Inverse as a Keras layer
  • NystromAttention: Implment Nystrom Attention as a Keras layer
  • Nystromformer: Put together a complete Nystromformer model stacking Nystrom Attention.
  • PreNorm: Normalize the activations of the previous layer for each given example in a batch independently and apply some function to it, implemented as a TensorFlow Keras Layer.
  • FeedForward: Create a FeedForward neural net with two Dense layers and GELU activation, implemented as a TensorFlow Keras Layer.

Full Changelog: https://github.com/Rishit-dagli/Nystromformer/commits/v0.1.0