What is the purpose of the timestamp field of the SceneEntity class?

[reubenjohn]

Background

I have a /scenario topic where each message in the topic represents the current and next 10 future states of my robot (like a prediction). Thus, I would like to be able to animate these future states when I playback in Foxglove studio. To do this, I created a message converter that converts each scenarioMsg message into a SceneUpdate message that contains 10 SceneEntity objects each having increasing timestamps representing the evolving future states.

My Dummy Implementation

I've registered a simple message converted that creates SceneUpdates. Each time an incoming message comes in, the message converter creates 5 SceneEntity objects. Each object is just a 3D text.
The first has a timestamp of 1s, the second has a timestamp of 2s, and so on. But they all have the same lifetime of 1s.
My expectation is that the first text should appear at t=1 and disappear at t=2, the second at t=2 and so on. The first text should appear at t=1.

export function activate(extensionContext: ExtensionContext): void {
  extensionContext.registerMessageConverter({
      fromSchemaName: 'myproto.Scenario',
      toSchemaName: 'foxglove.SceneUpdate',
      converter: (scenarioMsg: any) => {
        return {
          deletions: buildSceneEntityDeletions(),
          entities: [1, 2, 3, 4, 5].map(i => ({
            timestamp: {sec: i, nsec: 0},
            frame_id: "ego_vehicle_rear_axis",
            id: `id${i}`,
            lifetime: {sec: 1, nsec: 0},
            frame_locked: true,
            cubes: [],
            lines: [],
            texts: [{
              pose: {
                position: { x: Number(i), y: 0, z: 3 },
                orientation: eulerToQuaternion(0, 0, 0),
              },
              billboard: true,
              font_size: 1.0,
              scale_invariant: false,
              color: { r: 1.0, g: 1.0, b: 1.0, a: 1.0 },
              text: `id${i}`
            }]
          })),
        };
      },
    });
}

The Problem

EDIT 1st May, 2024: However, what happens is that all texts appear at the same time at t=3 and dissapear at t=4. t=3 is the timestamp of the first scenarioMsg, after that, they all appear again when the next myMessage comes in (eg. t=8) and so on.
However, what happens is that all texts appear at the same time not at t=1, but at t=100 and dissapear at t=101. t=100 is the timestamp of the first scenarioMsg, after that, they all appear again when the next myMessage comes in and so on.
I'm confused by this behavior. Why doesn't Foxglove studio respect the timestamp field when displaying entities in the scene?

[reubenjohn]

Happy to clarify anything that might be unclear about the question.

[defunctzombie]

The entities are all rendered when they are received. In your example that would be the time of the SceneUpdate message. The 3D panel receives the update message and sees all of the entities in that message. It uses the timestamp of each individual entity to figure out where in the scene to place the message by consulting the frame transforms of the entity frame to their parent frames and eventually the fixed frame.

The typical workflow goes something like this: You have some system that detects an object and produces a message for this detection. This detection lets say happens at t=99. Either that system/node/whatever produces a SceneUpdate or something downstream produces a SceneUpdate and adds the entity information. The SceneUpdate message is dispatched/logged/whatever at t=100. But to properly place the entity in the scene when we render it, we look at its timestamp (t=99) and find the corresponding frame transforms to place it at the right location in space as if we had received information about it at t=99.

Coming back to your example. When the 3D panel receives your entities, it sees timestamps of 1, 2, ... which are older than the current timestamp (per your example) and the panel will try to lookup the appropriate coordinate frame position for the entities at that time. These entities are not being projected into the future but instead appear from when the SceneUpdate message was received for their lifetime. The assumption is that you will produce the entities when you want them to appear - the 3d panel does not "schedule" them to appear later.

[reubenjohn]

Appreciate your answer! Could you confirm my understanding?:
The entities are always rendered at the timestamp of the SceneUpdate message (in my example t=3), but the transform used to position each entity in the scene update is the transform chain that is valid at the timestamp of that individual entity (t=1, 2, 3, 4, 5 in my example).
Also, I messed up part of my example, I've edited my original question (marked as EDIT 1st May, 2024).