Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid maximum in softmax #450

Merged
merged 7 commits into from
Jan 2, 2023
Merged

Avoid maximum in softmax #450

merged 7 commits into from
Jan 2, 2023

Conversation

mcabbott
Copy link
Member

@mcabbott mcabbott commented Jan 2, 2023

I think max is very careful about NaN & signed zero, and hence maximum is surprisingly slow. This uses the @fastmath version:

julia> let x = randn(Float32, 100, 1000)
         @btime softmax($x)
         @btime _softmax($x)  # this PR
       end;
  min 1.344 ms, mean 1.422 ms (7 allocations, 398.91 KiB)
  min 647.875 μs, mean 707.579 μs (3 allocations, 394.73 KiB)

@CarloLucibello
Copy link
Member

We could remove the doctest

@CarloLucibello CarloLucibello merged commit 07f67c8 into master Jan 2, 2023
@mcabbott mcabbott deleted the mcabbott-patch-2 branch January 2, 2023 22:24
Sleort added a commit to Sleort/NNlib.jl that referenced this pull request Jan 4, 2023
@Sleort
Copy link
Contributor

Sleort commented Jan 4, 2023

The new fast_maximum isn't always faster than maximum, ref. #456 (comment) ...

CarloLucibello pushed a commit that referenced this pull request Jan 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants