[Initializers] Refactor random and variance-scaling initializers

[jon-tow]

Summary

Random Initializers

• Adds tests, testRandomUniform(), testRandomNormal(), and testTruncatedNormal().

Variance-Scaling Initializers

• Refactor computation of fans, fanIn and fanOut.
• Add paper references to glorot initializers.
• Update glorotNormal to use truncatedNormal initialization.
• Add tests, testGlorotUniform() and testGlorotNormal().

[Shashi456]

The error here is

/swift-apis/Tests/TensorFlowTests/LayerTests.swift:366: error: LayerTests.testRNN : XCTAssertEqual failed:
 ("[[[ 0.20775774,  0.20080023, -0.13768704, -0.18534681]], [[ 0.22666009,  0.30019346, -0.19720285,   
-0.146838]], [[ 0.23758982,  0.32101023, -0.20359214, -0.17870961]], [[ 0.24337786,  0.33891943, 
-0.21143381, -0.16750808]]]") is not equal to ("[[[ 0.20775771,  0.20080023, -0.13768704, -0.18534681]], [[ 
0.22666009,  0.30019346, -0.19720285, -0.14683801]], [[ 0.23758979,  0.32101023, -0.20359215,  
-0.1787096]], [[ 0.24337786,   0.3389194, -0.21143384,  -0.1675081]]]") -

But i thought we commented out the RNN tests due to variation of outputs.
cc: @eaplatanios

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[marcrasi]

Hi, looks like there was a CI failure on this one. Could you take a look? Once we get CI passing, this should be ready to merge!

[jon-tow]

Hey @marcrasi! Everything should be good to go. Let me know if anything needs to be addressed. :)

[marcrasi]

It looks like this has broken LayerTests.testRNN, LayerTests.testLSTM, and SequentialTests.testSequential because they depend on specific initialization values that have changed. Is it expected that the values change? If so, updating the constants in those tests should be good.

[jon-tow]

It is expected. The previous implementations of Tensor(glorotUniform:) and Tensor(glorotNormal:) scaled and shifted drawn samples prior to variance scaling. I'm not sure what the reasoning was behind this and haven't seen it mentioned in any of the initialization literature.

I've updated the hardcoded values in the tests you've mentioned. :)

[jon-tow]

This passes locally but the kokoro build results show a discrepancy between values and expected-values in only the least significant digits. This is probably just a difference in float precision round-off. I'm going to add a bit of accuracy tolerance in the equality assertions to remediate this for now.

[marcrasi]

Yay, it passed, thanks! Merging.

[jon-tow]

No problem! Thanks for the help :)