Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Flax NNX Randomness #4279

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update Flax NNX Randomness #4279

wants to merge 1 commit into from

Conversation

8bitmp3
Copy link
Collaborator

@8bitmp3 8bitmp3 commented Oct 10, 2024 •
edited
Loading

@review-notebook-app Review Notebook App
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@8bitmp3
Copy link
Collaborator Author

8bitmp3 commented Oct 10, 2024 •
edited
Loading

Also updating:

- Here is a list of the main PRNG-related types in Flax NNX:
+ Here are the main PRNG-related types in Flax NNX:

Added the JAX jax.random tutorial for new users:

> **Note:** To learn more about random number generation in JAX,
the `jax.random` API, and PRNG-generated sequences, check out
this [JAX PRNG tutorial](https://jax.readthedocs.io/en/latest/random-numbers.html).

Added a link to JAX Working with pytrees for new users:

`nnx.reseed` is a function that accepts an arbitrary graph node (this includes
[pytrees](https://jax.readthedocs.io/en/latest/working-with-pytrees.html#working-with-pytrees)
of `nnx.Module`s)...

Added nnx.:

...`nnx.Rngs` object..... `nnx.reseed`.... `nnx.Dropout`

@8bitmp3 8bitmp3 self-assigned this Oct 10, 2024
@8bitmp3
Copy link
Collaborator Author

8bitmp3 commented Oct 10, 2024

rebasing after #4281

@8bitmp3 8bitmp3 closed this Oct 10, 2024
@8bitmp3 8bitmp3 reopened this Oct 10, 2024
@8bitmp3
Copy link
Collaborator Author

8bitmp3 commented Oct 15, 2024

@cgarciae PTAL, thanks!

IvyZX
IvyZX approved these changes Nov 27, 2024
View reviewed changes
docs_nnx/guides/randomness.ipynb
@@ -318,8 +344,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Splitting Rngs\n",
"When interacting with transforms like `vmap` or `pmap` it is often necessary to split the random state such that each replica has its own unique state. This can be done in two way, either by manually splitting a key before passing it to one of the `Rngs` streams, or using the `nnx.split_rngs` decorator which will automatically split the random state of any `RngStream`s found in the inputs of the function and automatically \"lower\" them once the function call ends. `split_rngs` is more convenient as it works nicely with transforms so we'll show an example of that here:"
"## Splitting `nnx.Rngs`\n",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not splitting nnx.Rngs directly. Maybe say "Splitting RNG keys".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@IvyZX IvyZX IvyZX approved these changes

@cgarciae cgarciae Awaiting requested review from cgarciae

Assignees

@8bitmp3 8bitmp3

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@8bitmp3 @IvyZX