Is the implementation of SinusoidalPositionEmbeddings in diffusion tutorial correct ?

[ketvector]

class SinusoidalPositionEmbeddings(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.dim = dim

    def forward(self, time):
        device = time.device
        half_dim = self.dim // 2
        embeddings = math.log(10000) / (half_dim - 1)
        embeddings = torch.exp(torch.arange(half_dim, device=device) * -embeddings)
        embeddings = time[:, None] * embeddings[None, :]
        embeddings = torch.cat((embeddings.sin(), embeddings.cos()), dim=-1)
        return embeddings

In the tutorial : https://github.com/huggingface/blog/blob/main/annotated-diffusion.md

for a given time-step, will this implementation not place all the sines first, and all the cosines later. (due to the line embeddings = torch.cat((embeddings.sin(), embeddings.cos()), dim=-1))

should it not be alternating sines and cosines like sin , cos, sin, cos ... and so on ?

[osanseviero]

cc @NielsRogge @kashif