Initialization imprvements

[rnett]

Makes three simple improvements to initialization:

• Makes Ops.initAdd no-op in eager mode, since the op has already been ran.
• Adds a runInit() method to Session that runs the graph's initializers.
• Adds a doInitialization() method to Session.Runner to add the initializers as targets.

There's no changes that conflict w/ the type refactor, so I'll just wait till it's merged and rebase onto master.

[karllessard]

I remember that we we've added the Init ops, we discussed about having an specialized endpoint like this to initialize the variables but then ended up deciding that we should have something more generic that allows you to run any single op in a session.

try (Graph g = new Graph()) {
  Variable<TInt32> x = tf.variable(tf.constant(10));
  Variable<TInt32> y = tf.variable(tf.constant(20));

  try (Session s = new Session(g)) {
    s.run(tf.init());
    ...
   }
}

Just double checking, is there any specific reason to go back to a specialized endpoint for variable initialization only? @Craigacp , any comment on this?

[rnett]

Just double checking, is there any specific reason to go back to a specialized endpoint for variable initialization only?

Two reasons: discoverability that initialization is a thing that needs to be ran (i.e. if you depend on a network that uses init that you didn't write, you may not even know it exists. I got burnt by forgetting to run it a few times), and doing s.run(tf.init()) creates a new init op each time, where runInit() just runs each initialization op w/o creating anything.

[Craigacp]

I think I'm fine with adding this back in, the previous discussion about it happened here - #36 (comment)

[Craigacp]

Just double checking, is there any specific reason to go back to a specialized endpoint for variable initialization only?

Two reasons: discoverability that initialization is a thing that needs to be ran (i.e. if you depend on a network that uses init that you didn't write, you may not even know it exists. I got burnt by forgetting to run it a few times), and doing s.run(tf.init()) creates a new init op each time, where runInit() just runs each initialization op w/o creating anything.

All TF 1 networks require the init op to be executed, it's been dropped with TF 2 in Python, but that's because the variables changed and things are implicitly eager whereas we still have Graph & Eager modes which are quite different. Until we paper over that more (and it might not be possible to do so without more improvements in the C API) then TF Java will still always need the init operator.

I guess that means we should document that initialization is still required somewhere.

[karllessard]

Again, if we focus on the functional API instead, we can probably call session.run(tf.init()) implicitly when function.call is invoked, or something like that?

[rnett]

Depends on how you store the session in the ConcreteFunction, but yeah probably.

Idk about Python TF2, but I'm still using it for variable initialization in my PR for variables, and I don't see a way around it. Python uses init_scope() (as a context manager) which I think is the same or similar, I don't know if we have an equivilant.

[karllessard]

Thanks @rnett , I like this simplified version and I personally like to make it easier to initialize the variables. I've left a few minor comments, if you want to take a look before I merge.

Still to be discuss, I would like to see at some point the Functional API taking care of initializing the variables once before calling the graph without the need for the user to do it.

[rnett]

I'll change those.

I agree about variable initialization, we could have the session track whether it's initialized, and run it on the first run call if not? We'd probably want to prevent the manual usage of the init op then.