[GPU] Fix dynamic loop's not matched issue during multiple shapes are inferenced

[kelvinchoi-intel]

Details:

• Fix the issue which second infer with updated shape in dynamic loop doesn't update sliced layout.
• Fix the issue that the optimized reshape doesn't reinterpret output memory in update_output_layout()

Tickets:

• 122739
• 131544

[yeonbok]

What is the target case? Seems that outer mem is allocated, which means it is either of static shape or "bounded dynamic shape". Which case?

[yeonbok]

I am curious because why we need this dyn_sliced_layout mapping. Seems that it just holds original dynamic shape. Which means that it can be pulled from layout of program_node.

[kelvinchoi-intel]

The target case:
-shape "chunks[-1, 1, -1]" -data_shape "chunks[32, 1, 80000][19, 1, 80000]"
Multiple infer with different shape is required for loop operation

In case sliced layout of loop is dynamic, the static shape is decided during first infer execution. But during 2nd infer, the sliced layout of loop is not updated. To update the sliced layout, the original dynamic shape info is needed at loop_inst::create_concat_memory_map().

[ahnyoung-paul]

I agree with @yeonbok 's comment.
I think if you get the output layout from the program node, you don't need to use dynamic_sliced_layout_mappings to handle the original output pshape.

[kelvinchoi-intel]

Updated to get the output layout from program node instead of saving dynamic shape

[ahnyoung-paul]

Why don't you use the following sample code to get the original input partial shape without creating the additional get_input_node_output_layout?

Suggested change

	auto origin_input_pshape = body_network->get_input_node_output_layout(internal_id.pid).get_partial_shape();
	auto origin_input_pshape = body_network->get_primitive(internal_id.pid)->get_node_output_layout();

[kelvinchoi-intel]

updated

[yeonbok]

Is your case for optimized <gather|permute|reorder|strided_slice> + loop ?
In that case, what does it mean ignoring the below codes?

[kelvinchoi-intel]

In case input node of loop is reshape(or gather/permute/reorder/strided_slice) and 2nd inference has smaller output memory of the input node than 1st infer, the output mem is not updated at this time. This causes not-matched shape issue at the inner network of the loop.

[yeonbok]

This is relevant to the "memory"s shape not the primitive's shape.
From your comment, loop is using input "memory" shape instead of input's primitive_impl shape. If that is the case, it is the problem. memory should be handled in the current way.

[kelvinchoi-intel]

Yes, this was the WA code for the case.
I will find the way for loop to use input's primitive_impl shape instead of input "memory" shape as you pointed.

[kelvinchoi-intel]

Deleted WA code and updated by reinterpreting with impl_shape.

[yeonbok]

minor, but I think you can just check the layout.is_dynamic(), no need to get partial shape.

[kelvinchoi-intel]

Updated

[yeonbok]

To me, this is not intuitive updating "initial memory". As from the original intention initial memory is a memory to initialize. Seems that this change is "if initial memory layout is not same as input layout (backedge.to layout, update initial memory". I am not understainding why we need to update "initial" memory.....

@ahnyoung-paul You approved this change, could you answer my question too?

[kelvinchoi-intel]

The initial_mem of loop is set during first inference. If 2nd inference changes layout, layout of outer input node is updated. But inner input mem may not updated due to reuse memory. For this case, inner input memory layout needs to be updated by changed outer input layout.

I'll add comment on code to explain this.

[yeonbok]

Umm Now I have a new curiousity regarding the behavior for multiple iteration of body network.
(1) Outer network's input layout node (lets say A) is {1, 10} and allocated memory is {1, 20} due to the previous allocation. And the body network's "backedge.to" node's layout is "{1,10}"

(2) At first iteration of body network, current will set input A's memory to backedge.to node.
This is okay because it is actually initial memory.
(3) If the body network runs for 2nd iteration, expected behavior is to assign backedge.from to backedge.to. Because it is "backedge". However current change, it still assigns outer input node A's memory.

[yeonbok]

In my thought initial impl of checking "from memory layout and outer input memory layout" to check whether the iteration is first or not, is wrong. We'll need another handle to chech whether to set outer mem or backedge.from mem to the backedge.to mem. @ahnyoung-paul Could you respond to this matter?

[ahnyoung-paul]

@yeonbok, The issued function (preprocess_backedge_memory) only works on preprocess memory functions that are called before running the inner network when the input layout is updated. So changing backedge memory is not related to this issue.

I think the main matter is that the network input layout is determined by the input memory layout, which might differ from the node's layout by memory reuse. If we have another way to set a layout of the input node in the network, I will suggest it. however, I can't find a way to set the layout without modifying the current network method. So I suggest interpreting the memory layout before setting the input layout of the inner network.
How about discussing it offline?

[kelvinchoi-intel]

The reshape node as a outer input node of issued loop has updated impl layout but not-updated output mem buffer layout. On previous commit, this unmatched mem buffer layout was updated in loop ops.
But now current commit updates reshape's output mem buffer layout by reinterpret_buffer in update_output_memory() of reshape.

need more invesigation