create Losses module #1264

CarloLucibello · 2020-07-02T14:59:09Z

Continuation of #1150, grouping losses under a Losses module as discussed.

An alternative for the module name could be Loss, but Losses seemed more natural.

I also renamed bce back to binarycrossentropy, since now that the function is within a module I could provide I deprecation path without changing the function name with respect to last tagged release.

Some of the function contain the _loss suffix, (e.g. hinge_loss, poisson_loss). We could drop that now that we have a namespace disambiguating the meaning, but again it seems more natural to keep, closer to the way people referer to them when speaking

PR Checklist

Tests are added
Entry in NEWS.md
Documentation, if applicable
Final review from @MikeInnes or @dhairyagandhi96 (for API changes).

johnnychen94 · 2020-07-02T15:07:04Z

An alternative for the module name could be Loss, but Losses seemed more natural.

How about Metrics as it can be a broader concept than Losses?

CarloLucibello · 2020-07-02T16:07:01Z

How about Metrics as it can be a broader concept than Losses?

mhmh, don't really fancy that, I usually think of metrics as something you keep track of, not optimize

DhairyaLGandhi · 2020-07-03T09:56:23Z

Metrics sounds better to my ear as well, the metrics that you mentioned also just sounds like the tracking information that would just be used to see training progress,

johnnychen94 · 2020-07-03T10:07:50Z

Disclaimer: I don't have a preference here. I proposed Metrics because

it is not uncommon to track losses during training, using Metrics provides a place for other non-loss metrics, such as accuracy, psnr, ssim, etc.
Losses has three s and I sometimes incorrectly type it as 😢 loses.

CarloLucibello · 2020-07-03T12:38:50Z

Keras has two separate modules, Losses, and Metrics
https://keras.io/api/metrics/
"A metric is a function that is used to judge the performance of your model.
Metric functions are similar to loss functions, except that the results from evaluating a metric are not used when training the model. Note that you may use any loss functions as a metric function."

pytorch instead has only losses, with metrics typically hand-coded by users.

We currently only have only loss functions here, not arbitrary metrics, so I think Losses would be a more appropriate name, something which conveys a clear sense of the primary purpose. If the concern is about future expandability, we could add a Metrics module at any point, possibly re-exporting Losses as well.

CarloLucibello · 2020-07-05T08:54:03Z

@DhairyaLGandhi merge?

CarloLucibello · 2020-07-06T10:07:32Z

CI failure likely due to JuliaDiff/ChainRules.jl#227

DhairyaLGandhi · 2020-07-06T12:55:40Z

src/Flux.jl

 include("layers/basic.jl")
 include("layers/conv.jl")
 include("layers/recurrent.jl")
 include("layers/normalise.jl")

 include("data/Data.jl")

+include("losses/Losses.jl")
+using .Losses # TODO: stop importing Losses in Flux's namespace in v0.12 


Loss should export all the functions as before.

It is, that is no loss is exported but all live in the Flux namespace, as they currently are. The comment is about keep the losses in Flux.Losses namespace, something we may want to do in v0.12

DhairyaLGandhi · 2020-07-06T12:57:43Z

src/losses/Losses.jl

+
+    agg((ŷ .- y).^2)
+"""
+mse(ŷ, y; agg=mean) = agg((ŷ .- y).^2)


Since there is a regression wrt to performance, maybe we should consider removing the aggregation for now

if you refer to #1255, mean is not affected by it

I mean specifically in the backwards pass.

which regression?

This is with Flux@0.10.4. cmse is the one from master, mymse is a current implementation that I am working on.

ok. But we can't remove aggregation now and add it back later, that's a lot of breakage. We should just improve performances under the hood. If you have a better implementation for agg=mean we can add a branch for that. But really for such basic functions as mean((x .- x).^2) we should expect Zygote to be perfomant, and track down the issue if it is not

DhairyaLGandhi · 2020-07-06T12:58:51Z

test/cuda/cuda.jl


-CUDA.allowscalar(false)


Why is this PR moving around tests? The tests should be largely untouched and esp tests unrelated to losses should be untouched

I had to reorganize a bit the cuda tests, in order to reflect the presence of the new Losses module and the new test/cuda/losses.jl file

DhairyaLGandhi · 2020-07-06T13:00:17Z

The package shouldn't move the definition of losses to a module like so. See how Optimisers.jl handles the directory structure.

DhairyaLGandhi · 2020-07-06T13:01:06Z

I would also consider calling this package Loss.jl or similar otherwise it sounds weird to my ear

src/losses/Losses.jl

CarloLucibello · 2020-07-06T14:47:13Z

I would also consider calling this package Loss.jl or similar otherwise it sounds weird to my ear

I fear using the singular here may go against julia's guidelines for module naming. Any opinion from @MikeInnes @oxinabox ?

DhairyaLGandhi · 2020-07-06T14:49:09Z

We should also make sure that we don't message things incorrectly. Currently it seems that we are saying that these bit of code are the loss functions which are somehow separate from being some generic transform that we can differentiate through. Continuing calling them stateless layers makes that a little bit more explicit. This is an important distinction to make for a newcomer to our ecosystem.

DhairyaLGandhi · 2020-07-09T16:50:35Z

bors try

DhairyaLGandhi · 2020-07-09T16:50:50Z

Hopefully we haven't missed any tests

bors · 2020-07-09T17:09:37Z

try

Build succeeded:

ci/gitlab/gitlab.com

CarloLucibello · 2020-07-09T18:30:51Z

bors r+

bors · 2020-07-09T18:51:41Z

Build succeeded:

ci/gitlab/gitlab.com

1287: Add CTC loss to new Losses module r=CarloLucibello a=maetshju This is a redux of adding the connectionist temporal classification loss from #342, now that the Losses module has been merged in #1264. Discussion in #342 suggested that a new PR would be easier than rebasing. Since the last commit in #342, functions and data structures from `CUDAnative.jl` and `CuArrays.jl` have been updated to work with `CUDA.jl`. This is in addition to incorporating the loss function into the Losses module. ### PR Checklist - [X] Tests are added - [X] Entry in NEWS.md - [X] Documentation, if applicable - [ ] Final review from `@dhairyagandhi96` (for API changes). Co-authored-by: Matt Kelley <matthew.curtis.kelley@gmail.com> Co-authored-by: Matthew C. Kelley <matthew.curtis.kelley@gmail.com>

CarloLucibello added this to the v0.11 milestone Jul 2, 2020

This was referenced Jul 2, 2020

Adding CTC loss #342

Closed

Add CUDA tests for stateless layers #906

Closed

CarloLucibello added 4 commits July 4, 2020 10:24

create Losses module

1b61939

cleanup

dd9e316

cleanup

ecb23da

refactor CUDA tests

957e4f4

CarloLucibello force-pushed the cl/losses branch from 7aec35c to 957e4f4 Compare July 4, 2020 09:06

cleanup

dbf36c3

better deprecation for binarycorssentropy

6fd4109

CarloLucibello mentioned this pull request Jul 6, 2020

no method matching MersenneTwister(::typeof(abs2), ::Array{Complex{Float64},1}) JuliaDiff/ChainRules.jl#227

Closed

update manifest

f7796e1

DhairyaLGandhi reviewed Jul 6, 2020

View reviewed changes

factorize module Losses

7dab012

DhairyaLGandhi reviewed Jul 6, 2020

View reviewed changes

src/losses/Losses.jl Outdated Show resolved Hide resolved

CarloLucibello added 2 commits July 6, 2020 16:53

move ALL_LOSSES to tests

6af8641

fix

67c0608

bors bot added a commit that referenced this pull request Jul 9, 2020

Try #1264:

ea41d68

DhairyaLGandhi approved these changes Jul 9, 2020

View reviewed changes

bors bot merged commit 683d580 into master Jul 9, 2020

maetshju mentioned this pull request Jul 21, 2020

Add CTC loss to new Losses module #1287

Merged

4 tasks

CarloLucibello deleted the cl/losses branch January 7, 2021 08:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

create Losses module #1264

create Losses module #1264

CarloLucibello commented Jul 2, 2020

johnnychen94 commented Jul 2, 2020 •

edited

Loading

CarloLucibello commented Jul 2, 2020

DhairyaLGandhi commented Jul 3, 2020

johnnychen94 commented Jul 3, 2020 •

edited

Loading

CarloLucibello commented Jul 3, 2020 •

edited

Loading

CarloLucibello commented Jul 5, 2020

CarloLucibello commented Jul 6, 2020

DhairyaLGandhi Jul 6, 2020

CarloLucibello Jul 6, 2020

DhairyaLGandhi Jul 6, 2020

CarloLucibello Jul 6, 2020

DhairyaLGandhi Jul 6, 2020

CarloLucibello Jul 6, 2020

DhairyaLGandhi Jul 6, 2020

CarloLucibello Jul 6, 2020

DhairyaLGandhi Jul 6, 2020

CarloLucibello Jul 6, 2020

DhairyaLGandhi commented Jul 6, 2020

DhairyaLGandhi commented Jul 6, 2020 •

edited

Loading

CarloLucibello commented Jul 6, 2020

DhairyaLGandhi commented Jul 6, 2020

DhairyaLGandhi commented Jul 9, 2020

DhairyaLGandhi commented Jul 9, 2020

bors bot commented Jul 9, 2020

CarloLucibello commented Jul 9, 2020

bors bot commented Jul 9, 2020


		CUDA.allowscalar(false)

create Losses module #1264

create Losses module #1264

Conversation

CarloLucibello commented Jul 2, 2020

PR Checklist

johnnychen94 commented Jul 2, 2020 • edited Loading

CarloLucibello commented Jul 2, 2020

DhairyaLGandhi commented Jul 3, 2020

johnnychen94 commented Jul 3, 2020 • edited Loading

CarloLucibello commented Jul 3, 2020 • edited Loading

CarloLucibello commented Jul 5, 2020

CarloLucibello commented Jul 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DhairyaLGandhi commented Jul 6, 2020

DhairyaLGandhi commented Jul 6, 2020 • edited Loading

CarloLucibello commented Jul 6, 2020

DhairyaLGandhi commented Jul 6, 2020

DhairyaLGandhi commented Jul 9, 2020

DhairyaLGandhi commented Jul 9, 2020

bors bot commented Jul 9, 2020

try

CarloLucibello commented Jul 9, 2020

bors bot commented Jul 9, 2020

johnnychen94 commented Jul 2, 2020 •

edited

Loading

johnnychen94 commented Jul 3, 2020 •

edited

Loading

CarloLucibello commented Jul 3, 2020 •

edited

Loading

DhairyaLGandhi commented Jul 6, 2020 •

edited

Loading