Skip to content

add softshrink kernel #570

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Oct 7, 2019
Merged

add softshrink kernel #570

merged 2 commits into from
Oct 7, 2019

Conversation

WindQAQ
Copy link
Member

@WindQAQ WindQAQ commented Oct 5, 2019

@WindQAQ WindQAQ requested review from facaiy, seanpmorgan and a team as code owners October 5, 2019 03:04
seanpmorgan
seanpmorgan reviewed Oct 7, 2019
View reviewed changes
Copy link
Member

@seanpmorgan seanpmorgan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks great. Op looks great, just want to clarify the default lambda

tensorflow_addons/activations/softshrink.py

@keras_utils.register_keras_custom_object
@tf.function
def softshrink(x, lower=-1.0, upper=1.0):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any intuition on defaulting to 1.0? The link you posted (which references pytorch) has a default of 0.5

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also 0.5 default in PaddlePaddle:
https://github.com/PaddlePaddle/Paddle/blob/master/paddle/fluid/operators/activation_op.cc#L251

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably no... It's my mistake on both hardshrink and softshrink though I could not find any research paper about the value 0.5. Will change them to 0.5 later.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Also couldn't find any rationale for 0.5 other than framework defaults.

seanpmorgan
seanpmorgan approved these changes Oct 7, 2019
View reviewed changes
Copy link
Member

@seanpmorgan seanpmorgan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again!

@seanpmorgan seanpmorgan merged commit 396b4a5 into tensorflow:master Oct 7, 2019
@wangmorgan
Copy link

can not install tensorflow-addons
pls help

@wangmorgan
Copy link

ERROR: Could not find a version that satisfies the requirement tensorflow-addons (from versions: none)
ERROR: No matching distribution found for tensorflow-addons

@seanpmorgan
Copy link
Member

ERROR: Could not find a version that satisfies the requirement tensorflow-addons (from versions: none)
ERROR: No matching distribution found for tensorflow-addons

Hi @wangmorgan it's likely that your pip version is too old to find the manylinux2010 pkg. Try running pip install --upgrade pip prior to installing

@seanpmorgan seanpmorgan mentioned this pull request Oct 16, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@seanpmorgan seanpmorgan seanpmorgan approved these changes

@facaiy facaiy Awaiting requested review from facaiy

Assignees
No one assigned
Labels
activations cla: yes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@WindQAQ @wangmorgan @seanpmorgan @googlebot