Only the first vector out of the nn.Embedding is ever used

[vnmusat]

Hi,

Is there a specific reason why only the first vector out of the nn.Embedding is ever used? tgt8 ... tgt64 are always zeros at this stage so you end up picking the 0-th vector for each spatial position, or in other words qe8 .. qe64 will always be filled with identical repeating values.... the values in the Embedding will obviously change over time with training but they will always be repeated throughout the qe-s

translating-images-into-maps/src/model/network.py

Line 1150 in 92b9627

tgt8 = torch.zeros_like(feat8[:, 0, :1]).expand(

translating-images-into-maps/src/model/network.py

Line 1161 in 92b9627

qe8 = (self.query_embed(tgt8.long())).permute(0, 3, 1, 2)

After permuting qe8 to [batch, spatial, spatial, len_embedding]:

tensor([[[ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         ...,
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921]],

        [[ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         ...,
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921]],

        [[ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         ...,
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921]]
....

[Chensiyu00]

Can you get the results shown on paper based on the given code?

[zjr-bit]

Hi，have you get the answer?