implement layerNormalization #60

mei1127 · 2023-12-07T05:42:17Z

@BruceDai @huningxin @fdwr @shiyi9801 .PTAL, thanks!

BruceDai

Thanks @mei1127, LGTM with some nits.

BruceDai · 2023-12-07T06:01:08Z

src/layer_normalization.js

+
+/**
+ * Normalize the tensor values of input features using
+ * [layer-Normalization]


Please update [layer-Normalization] with link, and combine two lines into just one line

* Normalize the tensor values of input features using * [layer-Normalization]

to

* Normalize the tensor values of input features using [layer-Normalization](https://arxiv.org/abs/1607.06450)

ok，thanks

BruceDai · 2023-12-07T06:02:36Z

src/layer_normalization.js

+ */
+export function layerNormalization(input, {scale, bias, axes, epsilon=1e-5}) {
+ validateLayerNormalizationParams(...arguments);
+ console.log('axes :', axes);


Please remove these debugging console.log code.

BruceDai · 2023-12-07T06:05:12Z

src/layer_normalization.js

+export function layerNormalization(input, {scale, bias, axes, epsilon=1e-5}) {
+ validateLayerNormalizationParams(...arguments);
+ console.log('axes :', axes);
+ if (axes === undefined) {


Synced with @shiyi9801, here default axes should be [1, ..., N-1] which N means the rank of input.

BruceDai · 2023-12-07T06:10:17Z

src/layer_normalization.js

+ const mean = reduceMean(input, reduceOptions);
+ const variance = reduceMean(pow(sub(input, mean), new Scalar(2)), reduceOptions);
+ output = div(sub(input, mean),
+ pow(add(variance, new Scalar(epsilon)), new Scalar(0.5)));


suggestion: sqrt op was added, so here we could invoke sqrt for simple usage.

BruceDai · 2023-12-07T06:28:46Z

src/lib/validate-input.js

+export function validateLayerNormalizationParams(input, {axes, scale, bias} = {}) {
+ if (scale && axes ) {
+ if (scale.rank !== axes.length) {
+ throw new Error('DataError: the rank of scale is not equal to the size of axes.');


These error messages are totally following Algorithm part of layerNormalization op, it would be a todo enhancement (low priority) to update validation checking for others ops. @huningxin WDYT, thanks.

sounds good to me, thanks!

shiyi9801 · 2023-12-07T06:39:06Z

src/layer_normalization.js

+ output = div(sub(input, mean),
+ pow(add(variance, new Scalar(epsilon)), new Scalar(0.5)));
+ if (scale) {
+ output = mul(output, reshape(scale, shape));


Here, scale and bias should not be directly reshape to shape.

The reason is that axes can be out-of-order.
Let's say an input shape = [1, 2, 3, 4], with axes = [1, 3], scale shape = [2, 4], then scale can be directly reshaped to [1, 2, 1, 4], and it's broadcast compatible to [1, 2, 3, 4].
But with axes = [3, 1], scale shape = [4, 2], if it's directly reshaped to [1, 2, 1, 4], then the broadcast to [1, 2, 3, 4] will work incorrectly.

So we should transpose the scale and bias from [4, 2] to [2, 4] before reshape them, and the transpose should follow how the axes being sorted to ascending order.

For example, with axes = [2, 0, 1] and scale shape = [4, 1, 2], we should transpose the scale following the permutation [1, 2, 0], so scale shape will be transposed to [1, 2, 4], and then we can reshape it to [1, 2, 1, 4].

thanks, let me think about it

fdwr

One important comment and some nits, but otherwise LGTM.

fdwr · 2023-12-07T06:30:41Z

src/layer_normalization.js

+ // The output tensor has the same shape as the input tensor.
+ let output = new Tensor(input.shape);
+ const inputShape = input.shape;
+ const shape = new Array(input.rank).fill(1);


How about shape -> compatibleShape or broadcastableShape?

fdwr · 2023-12-07T06:46:33Z

src/layer_normalization.js

+ const mean = reduceMean(input, reduceOptions);
+ const variance = reduceMean(pow(sub(input, mean), new Scalar(2)), reduceOptions);
+ output = div(sub(input, mean),
+ pow(add(variance, new Scalar(epsilon)), new Scalar(0.5)));


Can we just use the sqrt operator now? I think it's implemented now.

fdwr · 2023-12-07T06:53:51Z

src/lib/validate-input.js

+ throw new Error('DataError: the rank of scale is not equal to the size of axes.');
+ }
+ }
+ if (bias && axes ) {


Suggested change

if (bias && axes ) {

if (bias && axes) {

fdwr · 2023-12-07T06:54:59Z

src/lib/validate-input.js

+ for (let i = 0; i < axes.length; i++) {
+ const axis = axes[i];
+ if (axis >= input.rank) {
+ throw new Error('DataError:the value of axis in axes should be smaller than input.rank');


Suggested change

throw new Error('DataError:the value of axis in axes should be smaller than input.rank');

throw new Error('DataError: the value of axis in axes should be smaller than input.rank');

fdwr · 2023-12-07T06:55:53Z

src/lib/validate-input.js

+ }
+ const dim = input.shape[axis];
+ if (scale) {
+ if (scale.shape[i] == !dim) {


Suggested change

if (scale.shape[i] == !dim) {

if (scale.shape[i] !== dim) {

fdwr · 2023-12-07T06:58:00Z

test/layer_normalization_test.js

+ shape: [2, 2, 3],
+ value: [1, 2, 3, 6, 5, 4, 3, 6, 24, -10, 0, 5],
+ },
+ [-2.4494713718167804, 0, 4.898942743633561,


Suggested change

[-2.4494713718167804, 0, 4.898942743633561,

[

-2.4494713718167804, 0, 4.898942743633561,

New line for consistency with elsewhere, like below:

[ -1.4494713718167804, 2, 7.898942743633561, 3.4494713718167804,

fdwr · 2023-12-08T01:00:57Z

test/layer_normalization_test.js

- 1.4638475999719223, 0.8783085599831534, 0.29276951999438444,
- -0.1645769966453613, 0.131661597316289, 1.9090931610861905,
- -1.4482775704791793, -0.46081559060701155, 0.032915399329072226,
+ [-1.4638475999719223,


Oh, I wasn't talking about the wrapping of the numbers. Was just requesting a new line, like other places :b.

[ <--- -1.4638475999719223, 0.8783085599831534,

ok，thanks！ I got it

BruceDai · 2023-12-08T05:55:16Z

src/layer_normalization.js

+ const reduceOptions = {axes, keepDimensions: true};
+ const mean = reduceMean(input, reduceOptions);
+ const variance = reduceMean(pow(sub(input, mean), new Scalar(2)), reduceOptions);
+ output = div(sub(input, mean),


nit: these two lines can be in one .

BruceDai · 2023-12-08T06:30:01Z

src/layer_normalization.js

+ * @param {MLBatchNormalizationOptions} [options]
+ * @return {Tensor}
+ */
+export function sortByValue(axes) {


The function description isn't align with the below sortByValue function. Please add each description for each function.

And I suggest renaming sortByValue by getIndexOfSortedValue or getOriginalIndexOfSortedValue, it would be understandable.

BruceDai · 2023-12-08T07:10:25Z

src/layer_normalization.js

+ * Normalize the tensor values of input features using
+ * [layer-Normalization](https://arxiv.org/abs/1607.06450)
+ * @param {Tensor} input
+ * @param {Array} axes


Please remove this line description

BruceDai · 2023-12-08T07:13:43Z

src/layer_normalization.js

+ * [layer-Normalization](https://arxiv.org/abs/1607.06450)
+ * @param {Tensor} input
+ * @param {Array} axes
+ * @param {MLBatchNormalizationOptions} [options]


Need modify MLBatchNormalizationOptions to MLLayerNormalizationOptions

BruceDai · 2023-12-08T08:01:00Z

Thanks for addressing these review comments.
Maybe combine these 9 commits into new one, then it would be clean.

BruceDai · 2023-12-08T09:28:20Z

test/layer_normalization_test.js

+ );
+ });
+
+ it('layerNormalization Ascending order axis', function() {


nit: lowercase Ascending to ascending

fdwr · 2023-12-08T10:04:14Z

Maybe combine these 9 commits into new one, then it would be clean.

Don't you just squash merge the end result anyway, which results in a clean merge history to the target branch? Unfortunately forced pushes onto the active CR branch breaks this very useful functionality where reviewers can quickly see what's new since the previous time they reviewed it. 😢

BruceDai reviewed Dec 7, 2023

View reviewed changes

shiyi9801 reviewed Dec 7, 2023

View reviewed changes

fdwr reviewed Dec 7, 2023

View reviewed changes

BruceDai mentioned this pull request Dec 7, 2023

Implement webnn-baseline and WPT for operations needed for well-known transformers webmachinelearning/webnn#485

Closed

fdwr reviewed Dec 8, 2023

View reviewed changes

fdwr approved these changes Dec 8, 2023

View reviewed changes

BruceDai reviewed Dec 8, 2023

View reviewed changes

mei1127 changed the title ~~completed layer_normalization, revised validate-input.js~~ implement layerNormalization Dec 8, 2023

compeleted layer_normalization

5e5f729

mei1127 force-pushed the add_layernormalization branch from e6b9d71 to 5e5f729 Compare December 8, 2023 09:22

BruceDai reviewed Dec 8, 2023

View reviewed changes

fdwr approved these changes Dec 8, 2023

View reviewed changes

huningxin approved these changes Dec 12, 2023

View reviewed changes

huningxin merged commit cadafae into webmachinelearning:main Dec 12, 2023
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement layerNormalization #60

implement layerNormalization #60

mei1127 commented Dec 7, 2023 •

edited

Loading

BruceDai left a comment

BruceDai Dec 7, 2023

mei1127 Dec 7, 2023

BruceDai Dec 7, 2023

mei1127 Dec 7, 2023

BruceDai Dec 7, 2023

BruceDai Dec 7, 2023

mei1127 Dec 7, 2023

BruceDai Dec 7, 2023 •

edited

Loading

huningxin Dec 12, 2023

shiyi9801 Dec 7, 2023

mei1127 Dec 7, 2023

fdwr left a comment

fdwr Dec 7, 2023

mei1127 Dec 7, 2023

fdwr Dec 7, 2023

mei1127 Dec 7, 2023

fdwr Dec 7, 2023

mei1127 Dec 7, 2023

fdwr Dec 7, 2023

mei1127 Dec 7, 2023

fdwr Dec 7, 2023

mei1127 Dec 7, 2023

fdwr Dec 7, 2023

mei1127 Dec 7, 2023

fdwr Dec 8, 2023 •

edited

Loading

mei1127 Dec 8, 2023

BruceDai Dec 8, 2023

BruceDai Dec 8, 2023 •

edited

Loading

BruceDai Dec 8, 2023

BruceDai Dec 8, 2023

BruceDai commented Dec 8, 2023

BruceDai Dec 8, 2023

fdwr commented Dec 8, 2023

	throw new Error('DataError:the value of axis in axes should be smaller than input.rank');
	throw new Error('DataError: the value of axis in axes should be smaller than input.rank');

	[-2.4494713718167804, 0, 4.898942743633561,
	[
	-2.4494713718167804, 0, 4.898942743633561,

implement layerNormalization #60

implement layerNormalization #60

Conversation

mei1127 commented Dec 7, 2023 • edited Loading

BruceDai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BruceDai Dec 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fdwr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fdwr Dec 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BruceDai Dec 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BruceDai commented Dec 8, 2023

Choose a reason for hiding this comment

fdwr commented Dec 8, 2023

mei1127 commented Dec 7, 2023 •

edited

Loading

BruceDai Dec 7, 2023 •

edited

Loading

fdwr Dec 8, 2023 •

edited

Loading

BruceDai Dec 8, 2023 •

edited

Loading