Possible D435i exposure time bug

[kshitijminhas]

---

Required Info
Camera Model	D435i
Firmware Version	05.12.11.00
Operating System & Version	Ubuntu 18.04 LTS
Kernel Version (Linux Only)	4.15.0-130-generic
Platform	Intel i7 laptop
SDK Version	v2.42.0
Language	C++
Segment	General

Issue Description

Hi,
We found some anomalies with Intel RealSense D435i exposure time.

• RealSense camera shutter is open for twice the exposure value specified by the user, and
• the trigger is being sent in the middle of this 'extended' frame.

In other words, if I specify a shutter time of 7 ms I see the actual frame is ~14 ms. And the trigger is in the middle of this frame.

Following are some proofs:

Experiment Setup:

Intel D435i and a FLIR camera are setup for fixed 7ms exposure time.
Intel is running at 15 Hz, and providing trigger out to a Teensy 3.2.
The teensy passes through the Intel trigger to FLIR camera. The teensy also drives a LED.

Test1:

LED is ON by default, but is turned OFF when a trigger is received by the Teensy, and turned back ON after 7ms.
You can see test1.png where the pink graph is the LED state, and green graph is the trigger out from realsense.
Expected result: LED should be off in all camera frames (both intel and PGR).
Actual result: LED stays on in ALL intel frames (bad). LED stays off in ALL PGR frames (good).

Fig 'test1.png': Intel image is bottom right.

Test2:

LED is ON by default, but is turned OFF  ~7.5 ms before the arrival of trigger. This LED is turned ON after 7ms of the arrival of trigger. Hence, the LED is off for ~14.5 ms.
Again, you can see this is test2.png. Notice the LED graph goes LOW before the trigger. Notice the negative width value of ~14.5ms.
Expected result: LED should be off in all camera frames (both intel and PGR).
Actual result: Same as expected result.

Fig 'test2.png': Intel image is top right.

Test3:

LED is ON by default, but is turned OFF  ~7 ms before the arrival of trigger. This LED is turned ON after 7ms of the arrival of trigger. Hence, the LED is off for ~14 ms.
Again, you can see this is test3.png. Notice the negative width value of ~14ms.
Expected result: LED should be off in all camera frames (both intel and PGR).
Actual result: LED stays on in ALL intel frames (bad). LED stays off in ALL PGR frames (good).

Fig 'test3.png': Intel image is bottom right.

Test4:

I also did a test where the LED is turned ON 5 ms after the arrival of trigger, and turns off in 2ms. The LED was shown ON is all frames. Which proves that Intel actually stays open for ~7ms after the trigger.

I think these tests prove my claims above. Hoping to see if Intel can confirm the bug, and make necessary firmware fix.

Thanks,
Kshitij

[RealSenseCustomerSupport]

---

Thank you very much for the details about your experiment. I think all the test results are making sense to me. I guess you are confused on the timing of exposure from the depth camera,
Anyway, here is my suggestion. Get one more probe and hook it on the Pin#6 (or Pin#7) of the 9-pin AUX connector from D435 camera. I believe currently you're using the Pin#5 for the Trigger source, and pin#9 for the GND. The Pin#6 (or Pin#7) is laser projector PWM control signal. The burst pulsing is sync'd to exposure. So you can just take it as exposure signal. (NOTE: DO NOT forget to turn On the laser projector of D435). If you're using Auto Exposure mode, the PWM burst timing varies upon your room lighting condition. Darker room, longer burst. Brighter room, shorter burst.

Thanks

[kshitijminhas]

Hi @RealSenseCustomerSupport,
Thank you for the response.
Following your instructions, I attached a probe to pin 7 and another probe to pin 5. I set the RealSense device to 7 ms fixed shutter time. And enabled the IR emitter. Looking at the following graph, pink signal represents IR emitter (or pin 7) and green represents the trigger from RealSense camera (or pin 5).

You can see, the trigger on pin 5 comes after the IR emitter signal goes low. Is this expected behavior?

And following is a zoomed in version:

[RealSenseCustomerSupport]

---

Good job. Now you can see the exposure timing vs the trigger. The trigger always outputs roughly ~150us after the exposure End time. The Start time varies depending on exposure time. You can use your hand to cover the camera, and see how the maximum exposure time looks like. So if your LED's On time doesn't overlap the Pink area, you won't see LED turn On in D435i. Hope it's clear now.

Thanks

[kshitijminhas]

Well, I did the experiment you describe above. In the following image, pink signal is the LED (high means LED is ON), yellow signal is output from pin 7, and green signal is the output from pin 5.

The LED is programmed to turn ON when the trigger is received, and turn OFF in 7 ms. Intel camera is set for 7 ms fixed exposure.

But I still see the LED on. See below (Below image has left and right D435i IR camera images - depth cameras). Thoughts?

[RealSenseCustomerSupport]

---

Good, thanks.

1. Can you zoom in the center part of the waveform? I'd like to see how long the gap is between Orange and Pink. (btw, what are the two B/W images? Is the Left one from D435i? The Right one is from PGR? )
2. Can you also try delaying the Pink for some time, i.e 100us~1 ms. see if the LED is gone from D435i?

Thanks

[kshitijminhas]

So if I understand this correctly, the trigger somewhat aligns with 'data readout and transmit commence'. And the 'RS2_FRAME_METADATA_FRAME_TIMESTAMP' will give me the timestamp that aligns with this trigger.

Which means, my camera started capturing image 7ms (assuming 7ms fixed exposure time) before this timestamp AND the trigger. So indeed there is no way to trigger an external camera at the same time as the realsense camera (in better words, we cannot have an external camera have the shutter open at the same time as the realsense based on this trigger).

Did I get this right?

To answer 1b above- Both the images are from RealSense D435i. These are the left and right IR cameras (depth cameras) on RealSense. Also edited the comment above.

To answer your Q2, please see Test 4 of my original post above.

[RealSenseCustomerSupport]

---

Thanks for the answers. Regarding the "Test 4" result, delaying 5ms sounds too much and the next frame exposure might be already started, so you still saw the LED On. What was the exposure time you set for "Test 4"? better to check it again with the PWM pulsing.

By default, the trigger is output 100~150us after shutter close, and the readout is started right after the rising edge of the Trigger.
Are you looking for a trigger like Strobe, to control your FLIR camera? Unfortunately, that pin is not pulled out to the AUX connector. You might need to think about using the PWM burst to create a strobe pulse. i.e. one shot timer.

Thanks

[kshitijminhas]

So, can you confirm the 'RS2_FRAME_METADATA_FRAME_TIMESTAMP' aligns with this pin 5 trigger?
Here is the definition of RS2_FRAME_METADATA_FRAME_TIMESTAMP from SDK docs: "Timestamp set by device clock when data readout and transmit commence. usec "

For test 4, as the post describes, RealSense is fixed at 7ms exposure time and 15 Hz. The frames are 66 ms apart, so there shouldn't be any overlap if I turn the LED on after 5 ms of the trigger.

Regardless, following is a test where I turn the LED on 2 ms after the trigger is received, and turn off 7 ms after the trigger is received. Again, RealSense is at 7ms fixed exposure and 15 Hz.

In the figure below, Pink is LED state, and green is pin 5 (trigger) from RealSense D453i.

Following are the left and right images from realsense IR cameras. You can see LED turned on, hard to see, but 2 white dots is the LED in left and right camera.

[RealSenseCustomerSupport]

---

Thanks for waveform. Did you forget to turn on the laser projector? I don't see the PWM burst on the Orange probe?
btw, can you get a mirror and place it next to the LED? so let D435i see the laser dots from the mirror. I don't see any laser dots in the pictures. 7ms exposure is not short, it's supposed to see the dots clearly.

[kshitijminhas]

The dot projector is turned off in this test. It should not affect the results.

Also, can you please answer my timestamp question above? Or, tag someone who can answer it? Thanks

[Domos]

I'm not an Intel developer, correct me if I'm wrong.

Looks like setting a shutter time of 7 ms, it's actually exposure for 14ms. First 7ms with emitter on and second 7ms with emitter off.
I guess RealSense can filter out backgrounds by subtracting these two images.

Pin 5 gets triggered at the middle of the whole 14ms exposure. I think you can consider set RS exposure time to 28ms and let PGR camera exposure for 14ms after the trigger.

Or as they suggested, trigger PGR camera by the emitter signal.

[kshitijminhas]

@Domos that's what I understand. The shutter seems to be open for 14 ms, when 7 ms is requested.
Our application does not require IR projector, so we turn it off. Hence, cannot use that signal.

We are okay to trigger the PGR camera using this trigger, as long as we have an understanding of the relation of RS2_FRAME_METADATA_FRAME_TIMESTAMP wrt pin 5 trigger.

[RealSenseCustomerSupport]

---

Hi @kshitijminhas
sorry for missing that question. The "RS2_FRAME_METADATA_FRAME_TIMESTAMP" is clocked at Start of Frame from sensor readout, but the Trigger you get from the pin#5 of AUX connector has some delay (roughly 0.1~0.2us).

Which mode did you select on the "Inter Cam Sync Mode"? 0 or 1? if it was 0, please try Mode=1.
and also under Mode=1, if the problem still exists, can you please try 30FPS instead of 15FPS?

btw, if you don't want to see the laser dots, you can cover the projector by using black tape.

[kshitijminhas]

Thanks for the response. I think we can live with 0.2 microseconds delay between the timestamp and this trigger.

Also, I have tested with Inter Cam Sync Mode = 1 before, similar results as described above.

[RealSenseCustomerSupport]

---

Hi @kshitijminhas​

Thanks a lot for the cooperation. I am very interested in this case. Just wondering if this is an isolated case or a typical one? Do you have another D435 or D435i and see same issue?

btw, can you confirm the following two pictures, were they all taken under 7ms exposure? why the 2nd one is so dark? did you turn off your room light?

"Again, RealSense is at 7ms fixed auto-exposure and 15 Hz." - what does it mean of "fixed auto-exposure"? i guess you meant manual exposure?

thanks

Attachment(s):
D435_7ms_exps.PNG - https://realsensesupport.intel.com/attachments/token/E75JYKs5vk88539aIZEdaOUDt/?name=D435_7ms_exps.PNG

[kshitijminhas]

Glad to hear that!

1. Tested with 2 different D435i, same issue.
2. I cannot see your attachment/ picture, but I can confirm all pictures were taken under the same config setting. Light was turned off in the room, and it was in the evening, so the picture was very dark.
3. Oh, I meant fixed 7 ms exposure. Edited the comment above.

[RealSenseCustomerSupport]

---

Hi @kshitijminhas, the sensor company suggests NOT to point strong light to sensor during readout time which takes about 8ms after the trigger while under 15FPS.