External RGB flash

[ArjanVroegop]

Required Info
Camera Model	{435
Firmware Version	5.11.6.250
Operating System & Version	Win10
Kernel Version (Linux Only)
Platform	PC
SDK Version	{2.17 }
Language	{N/A }
Segment	Robot

Issue Description

Hi everyone,

I'm looking into the possibilities for using the Realsense D435 in combination with an external flash for the RGB images. I use the hardware synchronization pins and use a voltage shifter to increase the voltage from 1.8V to 5V so the LED controller (Effilux strobe controller) can handle the signal.

I use these settings in the Realsense viewer:
• RGB Camera --> Controls --> Auto Exposure Priority: OFF
• Stereo Module --> Controls --> Inter Cam Sync Mode: 1
• Stereo Module --> Controls --> Output Trigger Enabled: ON

This results in a RGB image that is illuminated by the strobe. The infrared (depth) images are not affected by the strobe and show nice images with the emitter pattern.

Now, when I set the RGB camera to manual exposure, I get something I cannot explain. When running 1280x720@30FPS, I get a good image when the exposure is above 221. When I lower the exposure, the image is partially illuminated and partially dark. See this image for an example:

The lower part does not consist of black pixels (0-values), but is the same as an image without the strobe. It seems that about 50% of the image is illuminated. Now when lowering the exposure further only 25% of the image is illuminated when the exposure goes below 111. This happens again at 56 (12,5% illuminated) and so on.

It also seems that this is related to framerate, see this table: halving the framerate doubles the values, resolution does not change things.

Does anyone know what's happening here? The result I have so far is already quite good, but I would like the exposure time as much as possible for our application.

Also, does anyone know what the units for the exposure setting are? I've read something about microseconds, but I'm not sure about that.

[agrunnet]

This is happening because the sensor is a rolling shutter. This means that as you increase exposure time you increase the number lines that are on at a single time. But these lines scroll down and repeat from top to bottom. So when you flash for a short time you only expose the lines that happen to be “on”.
Btw The frame rate is the rate at which these lines scroll.
So take the example of setting the smallest exposure. Then you only have one line that is photosensitive at a time. If you have an equally short temporal flash you will only see a single horizontal line illuminated.

By contrast the D435 stereo monochrome imagers are “global shutter” and would behave like you would expect.

[ArjanVroegop]

Hi agrunnet,

Thanks for you answer. I know the sensor is a rolling shutter. I think I understand how a rolling shutter works, but I don't understand what's happening over here.

So what you are saying is that the "exposure" is not really an exposure time (for a single line or for the total image), but a parameter that affects how many lines are 'on' at a single time. So in the picture above, the top half of the image is grabbed at once, then there was some processing time and then the second half of the image was taken? This would explain why the image looks like it is now.
But if this is the case, does that mean that if the exposure setting is high enough the sensor basically act like a global shutter?

Also, in the table above it shows that the exposure needs to be higher than 221 in order to get a fully illuminated image. So it is technically not possible to have an exposure lower than this value when I use an external strobe?

Last question: the color image sensor is an OVT OV2740. Is there any technical information about this sensor available which explain how the sensor works?

[agrunnet]

As you say it is probably best to go directly to the source of this which is understanding the exact workings of the omnivision rolling shutter mechanism. All our documents are NDA from them but you should be able to get some online or from omnivision.
Btw what is the duration of the strobe?
It is surprising that you see the half and then quarter etc. I would have expected simple to see gradual shrinking of the exposed region as you reduce the exposure time. The total number of lines is convolution of exposure time and strobe-on time.

[ArjanVroegop]

There is not much information on the Omnivision website - that probably is also the reason you have an NDA, Omnivision does not want everybody to see the details. :)

I'm using a 100 microsecond strobe. When I change this value, the size of the illuminated area stays the same, only the light intensity in the illuminated area changes.

I agree with you that it is strange that the illuminated area's double in size instead of a gradual change. Also note that there is not a soft transition between the light/dark area's, but a sudden change in light level.

[RealSenseCustomerSupport]

---

Hi ArjanVroegop,
Please use the Strobe pin of OV2740 to control the external RGB flash, not the sync pin.

Thanks,

[RealSenseCustomerSupport]

---

Hi ArjanVroegop,
Do you still have question on this?

[ArjanVroegop]

I'm sorry for the late reply, I did not have time to work on this issue lately (holiday season and other projects). The problem is still relevant, it is a side-project for me and I work on it every now and then.

So far I can use the strobe synced with the camera if I have the right settings but i don't understand what the settings really do.

About the remark on using the strobe pin of the OV2740: This is not something that can be accessed easily, or am I wrong?

[ArniBarni]

Hi @ArjanVroegop,

i facing the problem, that i have to sync the d435 camera with an external device and therefor i need to shift the voltage. Can you explain how you did this?

[ArjanVroegop]

Hi @ArniBarni,

I've used a voltage level shifter to do this. It is one similar to this one: https://www.sparkfun.com/products/12009

What kind of external device do you want to connect?

[ArniBarni]

Okay thanks.
I want to connect the RealSense with the Qualisys motion capture system. To be precise, with the miqus synchronization unit. This sync unit require a positive 5V TTL Pulse.
Thats is the only explenation that i have from this System. I asume that could be somekind of a standart in the professional broadcast enviroment, because there use for that a bnc connector and no further explenation exists.