Add support for OV5640 camera

[jasaw]

Add support for OV5640 5MP camera. The camera has a resolution of 2560 x 1920 but the output will be scaled down to the configured resolution e.g. 640 x 480. When zoom is enabled, a 640 x 480 window is cropped from the full 2560 x 1920 frame, therefore giving double the magnification compared to OV2640.

This is a zoom comparison between OV5640 and OV2640.

OV5640:

OV2640:

[caco3]

what is the impact on the RAM usage?
Does it need to store a larger framebuffer for the higher resolution?

[jasaw]

@caco3 There's no impact on RAM usage at all. It works the same way as ov2640 where cropping is done on the camera module itself.

[SybexX]

Process:

1. The camera takes an image with a resolution of 2560x1920
2. The camera cuts out the desired area from the image
3. The camera scales the cropped area to 640x480, if needed (desired area is larger 640x480)
   This process applies both when using the zoom function and when zoom is deactivated.

However, further changes to the implementation are necessary, e.g. at the sharpness....

Ablauf:

1. Die Kamera nimmt ein Bild mit einer Auflösung von 2560x1920 auf
2. Die Kamera schneidet den gewünschten Bereich aus dem Bild aus
3. Die Kamera skaliert den ausgeschnittenen Bereich bei Bedarf auf 640x480 (wenn der ausgeschnittene Bereich größer 640x480 ist).
   Dieser Ablauf wird sowohl bei der Nutzung der Zoom-Funktion als auch bei deaktivierten Zoom benutzt/verwendet.

Allerdings sind noch weitere Änderungen an der Implementierung notwendig, z.B. bei sharpness.....

[jasaw]

However, further changes to the implementation are necessary

@SybexX May I know what you would like to change?

[SybexX]

e.g. so:

I made a few quick adjustments, but I don't know if it works because I don't have an OV5640.
ClassControllCamera.zip

[jasaw]

@SybexX I've adjusted the sharpness handling to your version and it's working fine on my OV5640 system. I changed the sharpness range because the officially supported sharpness range is from -3 to +3.
_sharpnessLevel = min(2, max(-2, _sharpnessLevel));

Your ClassControllCamera code enforces 4:3 ratio on the zoom parameters, but OV5640 is a lot more flexible. It doesn't have the 4:3 restriction. Currently this PR only supports zoom mode 0, but it might be possible to achieve other zoom modes by passing different parameters to set_res_raw with scaling and binning set to true. That needs more experimentation, which I think could be left for future enhancement.

[SybexX]

I specifically set the sharpnessLevel to 2.
After longer tests, I found that a value of 3 sometimes causes problems.

I know that the OV5640 can do more, but we should adapt everything to the OV2640,
otherwise we would need at least two pages, i.e. one for each camera model and it simply becomes too confusing for the user.

Sometimes less is more^^

[SybexX]

[jasaw]

@SybexX Thank you for reviewing. I've refactored my code based on your sample code but had to make a few changes to make it work with the OV5640.

only else, otherwise the other supported camera models will be ignored

My apologies, it is not clear to me that OV3660 (which is the other supported camera) behaves the same way as OV2640 and parts of the code are hitting the OV2640 registers directly (presumably the OV3660 has the same registers as OV2640?). Personally, I prefer to be more explicit in the code when it comes to handling hardware specific behaviour, like

if (sensor_info->model == CAMERA_OV5640)
{
}
else if ((sensor_info->model == CAMERA_OV2640) || (sensor_info->model == CAMERA_OV3660))
{
}

Either way, I am not particular with how the code is written as long as it is well documented via comments or code itself so other developers understand the intention of the author.

[SybexX]

The registers don't matter, the properties of the camera are more important.
The https://github.com/espressif/esp32-camera relieves us of having to pay attention to the registers.
Since no distinction was made between the camera models before I made my changes, I assumed that the OV2640 and OV3660 were very similar. After I quick look, the OV3660 is very close to the OV5640 in terms of functions.

[jasaw]

The registers don't matter

ov2640_set_sharpness and ov2640_enable_auto_sharpness functions are called for OV3660 camera as well and eventually those functions call sensor->set_reg which writes directly to the OV3660's registers. Keep in mind that OV2640 sharpness is not officially supported by Espressif SDK. I haven't looked into the OV3660 datasheet, so I don't know what the camera will do if those registers don't exist or worse, have a different function. I don't have an OV3660 to test, so can't validate.

Since no distinction was made between the camera models before I made my changes, I assumed that the OV2640 and OV3660 were very similar

That was an oversight on my part as well. We should probably test sharpness against OV3660 and update the code accordingly?

[SybexX]

If something is not available on a camera model, the esp32-camera library actually returns a -1

https://github.com/espressif/esp32-camera/blob/master/sensors/ov2640.c
https://github.com/espressif/esp32-camera/blob/master/sensors/ov3660.c
https://github.com/espressif/esp32-camera/blob/master/sensors/ov5640.c

[jasaw]

I propose this sharpness handling code:

void CCamera::SetCamSharpness(bool _autoSharpnessEnabled, int _sharpnessLevel)
{
    sensor_t *s = esp_camera_sensor_get();

    if (s != NULL)
    {
        _sharpnessLevel = min(2, max(-2, _sharpnessLevel));

        camera_sensor_info_t *sensor_info = esp_camera_sensor_get_info(&(s->id));

        if (sensor_info != NULL)
        {
            if (sensor_info->model == CAMERA_OV5640 || sensor_info->model == CAMERA_OV3660)
            {
                if (_autoSharpnessEnabled)
                {
                    // autoSharpness is not supported, default to zero
                    s->set_sharpness(s, 0);
                }
                else
                {
                    s->set_sharpness(s, _sharpnessLevel);
                }
            }
            else if (sensor_info->model == CAMERA_OV2640)
            {
                // The OV2640 does not officially support sharpness, so the detour is made with the ov2640_sharpness.cpp.
                if (_autoSharpnessEnabled)
                {
                    s->set_sharpness(s, 0);
                    ov2640_enable_auto_sharpness(s);
                }
                else
                {
                    ov2640_set_sharpness(s, _sharpnessLevel);
                }
            }
        }
    }
    else
    {
        LogFile.WriteToFile(ESP_LOG_ERROR, TAG, "SetCamSharpness, Failed to get Cam control structure");
    }
}

[jasaw]

@SybexX Looks like OV3660 set_res_raw function is the same as OV5640 as well. I should probably handle OV3660 explicitly.

[SybexX]

or so, then the ESP is spared the second comparison ^^

void CCamera::SetCamSharpness(bool _autoSharpnessEnabled, int _sharpnessLevel)
{
sensor_t *s = esp_camera_sensor_get();

if (s != NULL)
{
    _sharpnessLevel = min(2, max(-2, _sharpnessLevel));

    camera_sensor_info_t *sensor_info = esp_camera_sensor_get_info(&(s->id));

    if (sensor_info != NULL)
    {
        if (sensor_info->model == CAMERA_OV2640)
        {
            // The OV2640 does not officially support sharpness, so the detour is made with the ov2640_sharpness.cpp.
            if (_autoSharpnessEnabled)
            {
                s->set_sharpness(s, 0);
                ov2640_enable_auto_sharpness(s);
            }
            else
            {
                ov2640_set_sharpness(s, _sharpnessLevel);
            }
        }
        else
        {
			// for CAMERA_OV5640 and CAMERA_OV3660
            if (_autoSharpnessEnabled)
            {
                // autoSharpness is not supported, default to zero
                s->set_sharpness(s, 0);
            }
            else
            {
                s->set_sharpness(s, _sharpnessLevel);
            }
        }			
    }
}
else
{
    LogFile.WriteToFile(ESP_LOG_ERROR, TAG, "SetCamSharpness, Failed to get Cam control structure");
}

}

[SybexX]

        if (sensor_info->model == CAMERA_OV2640)
        {
            // The OV2640 does not officially support sharpness, so the detour is made with the ov2640_sharpness.cpp.
            if (_autoSharpnessEnabled)
            {
                s->set_sharpness(s, 0);	<<<<<<<<<< oh yes, this is actually not necessary, as it only returns a -1(not supported)
                ov2640_enable_auto_sharpness(s);
            }
            else
            {
                ov2640_set_sharpness(s, _sharpnessLevel);
            }
        }

[jasaw]

@SybexX I have updated the code to your suggested sharpness handling and also updated zoom handling code to better deal with OV3660 but I don't have an OV3660 so can't test. I have only tested with OV5640 so far.

[SybexX]

            case CAMERA_OV5640:
                frameSizeX = 2560;
                frameSizeY = 1920;
                // max imageSize = ((frameSizeX - CCstatus.ImageWidth) / 8 / 4) - 1
                // 59 = ((2560 - 640) / 8 / 4) - 1
                if (imageSize < 59)
                {
                    _imageSize_temp = (59 - imageSize);
                }
                SanitizeZoomParams(_imageSize_temp, frameSizeX, frameSizeY, _imageWidth, _imageHeight, _offsetx, _offsety);
                SetCamWindow(s, unused, frameSizeX, frameSizeY, _offsetx, _offsety, _imageWidth, _imageHeight, CCstatus.ImageWidth, CCstatus.ImageHeight);
                break;

            case CAMERA_OV3660:
                frameSizeX = 2048;
                frameSizeY = 1536;
                // max imageSize = ((frameSizeX - CCstatus.ImageWidth) / 8 / 4) -1
                // 43 = ((2048 - 640) / 8 / 4) - 1
                if (imageSize < 43)
                {
                    _imageSize_temp = (43 - imageSize);
                }
                SanitizeZoomParams(_imageSize_temp, frameSizeX, frameSizeY, _imageWidth, _imageHeight, _offsetx, _offsety);
                SetCamWindow(s, unused, frameSizeX, frameSizeY, _offsetx, _offsety, _imageWidth, _imageHeight, CCstatus.ImageWidth, CCstatus.ImageHeight);
                break;

            case CAMERA_OV2640:
                // ov2640_sensor_mode_t _mode = OV2640_MODE_UXGA; // 1600x1200
                // ov2640_sensor_mode_t _mode = OV2640_MODE_SVGA; // 800x600
                // ov2640_sensor_mode_t _mode = OV2640_MODE_CIF;  // 400x296
                _mode = 0;
                frameSizeX = 1600;
                frameSizeY = 1200;
                // max imageSize = ((frameSizeX - CCstatus.ImageWidth) / 8 / 4) -1
                // 29 = ((1600 - 640) / 8 / 4) - 1
                if (imageSize < 29)
                {
                    _imageSize_temp = (29 - imageSize);
                }
                SanitizeZoomParams(_imageSize_temp, frameSizeX, frameSizeY, _imageWidth, _imageHeight, _offsetx, _offsety);
                // _mode sets the sensor resolution (3 options available),
                // _offsetx and _offsety set the start of the ROI,
                // _imageWidth and _imageHeight set the size of the ROI,
                // CCstatus.ImageWidth and CCstatus.ImageHeight set the output window size.
                SetCamWindow(s, _mode, frameSizeX, frameSizeY, _offsetx, _offsety, _imageWidth, _imageHeight, CCstatus.ImageWidth, CCstatus.ImageHeight);
                break;

[jasaw]

@SybexX The new imageSize code that you shared above appears to be working well on my OV5640 setup and thanks for adding the comments to explain how the imageSize numbers came from because I had no idea previously.

I've also increased the OV5640 full frame size to match the datasheet.

[SybexX]

still necessary adjustment in the edit_reference.html:

        <td>
                <input required type="number" id="TakeImage_CamZoomSize_value1" value="0" min="0" max="59" step="1" onchange="cameraParameterChanged()"
                oninput="(!validity.rangeOverflow||(value=59)) && (!validity.rangeUnderflow||(value=0)) && (!validity.stepMismatch||(value=parseInt(this.value)));">
        </td>

still necessary adjustment in the edit_config_template.html:

		<td>
			<input required type="number" id="TakeImage_CamZoomSize_value1" value="0" min="0" max="59" step="1" onchange="cameraParameterChanged()"
			oninput="(!validity.rangeOverflow||(value=59)) && (!validity.rangeUnderflow||(value=0)) && (!validity.stepMismatch||(value=parseInt(this.value)));">
		</td>
		<td>$TOOLTIP_TakeImage_CamZoomSize</td>			
	</tr>

	<tr class="expert" unused_id="TakeImage_CamZoomOffsetX_ex3">
		<td class="indent1">
			<class id="TakeImage_CamZoomOffsetX_text" style="color:black;">Zoom Offset X</class>
		</td>
		<td>
			<input required type="number" id="TakeImage_CamZoomOffsetX_value1" value="0" min="-960" max="960" step="8" onchange="cameraParameterChanged()"
			oninput="(!validity.rangeOverflow||(value=960)) && (!validity.rangeUnderflow||(value=-960)) && (!validity.stepMismatch||(value=parseInt(this.value)));">Pixel
		</td>
		<td>$TOOLTIP_TakeImage_CamZoomOffsetX</td>
	</tr>

	<tr class="expert" unused_id="TakeImage_CamZoomOffsetY_ex3">
		<td class="indent1">
			<class id="TakeImage_CamZoomOffsetY_text" style="color:black;">Zoom Offset Y</class>
		</td>
		<td>
			<input required type="number" id="TakeImage_CamZoomOffsetY_value1" value="0" min="-720" max="720" step="8" onchange="cameraParameterChanged()"
			oninput="(!validity.rangeOverflow||(value=720)) && (!validity.rangeUnderflow||(value=-720)) && (!validity.stepMismatch||(value=parseInt(this.value)));">Pixel
		</td>
		<td>$TOOLTIP_TakeImage_CamZoomOffsetY</td>
	</tr>

[jasaw]

@SybexX

Can you try out whether the following changes eliminate the red coloring when you have vertical flip activated?
Does your OV5640 with auto focus also have the same problems with vertical flip?

The config order changes does not fix the vertical flip red colour problem for me. Both my normal OV5640 and OV5640 autofocus behave the same way. From I can tell, the autofocus version behaves exactly the same way as the normal OV5640 in every way, except it has the autofocus VCM circuit attached at the side.

I changed something regarding gainceiling because it didn't work properly with the OV5640

Good catch. Gain ceiling is a little hard to test because it only kicks in after it has reached maximum exposure level and the datasheet is not exactly clear that the gain ceiling registers take real value rather than enum. The espressif library gets it wrong, so would be nice to upstream your fix.

[SybexX]

@jasaw It might be an advantage if you put something like this in the AI-on-the-edge-device/param-docs/parameter-pages/TakeImage/CamVflip.md:

!!! On some OV5640 Cameras, the image becomes reddish when Vflip is used in conjunction with the zoom function!

To avoid countless inquiries/reports.

[jasaw]

@SybexX I've updated the vflip param page and resolved merge conflict.

[jasaw]

I've been running my OV5640 for some time now. It is working well for me. Can we merge this in?

[caco3]

@SybexX Is this something we should add to the official firmware?
@jasaw can you please fix the conflicts?

[SybexX]

@caco3 yes, I would like to have this in there too.
It works as it should, does not consume any additional resources compared to the ov2640 and expands the supported hardware.

[jasaw]

I've resolved the merge conflicts, but haven't eyeballed the merged code line by line to make sure everything merged correctly. I haven't tested the firmware. I'll OTA upgrade to my OV5640 system tomorrow.

[caco3]

OK, that sounds great.
We should add this to the new release and mention it clearly as this a large enhancement -> readme, documentation, release notes, ...
@jasaw since you are a core user of it, please test it it works as needed.

[SybexX]

I tested it with an ov2640 and ov5640 and I didn't notice any errors/problems.
Let's see what @jasaw says when he tests it.

[jasaw]

I've been running this firmware for more than a day now and it's working as expected.

[Pferdebockwurst]

Very nice to see some progress regarding the cameras. Can you recommend one of them? Is one of the new supported models better than the other one? Is autofocus important? Thanks a lot guys!

[jasaw]

@Pferdebockwurst Auto focus is NOT supported yet and auto focus also requires some soldering on most ESP32Cam boards. If you don't solder wires, it's still safe to use OV5640 modules with auto focus. The auto focus circuitry is completely separate from the OV5640.
I bought my OV5640 with and without auto focus from 3 different sellers on AliExpress and they're all working fine for me, so maybe any one will work as long as it's advertised to work with ESP32Cam?

[Slider0007]

@jasaw, @SybexX
I'm not sure if the issue with tinted image is solved already, but during my last activities I also faced the tinted image issue from time to time. Ensure that only an odd offset is provided, solved it for me - at least with my camera. I have only one OV5640 camera therefore I cannot say if this is transferable or the issue here is something different. Just let you know...

[SybexX]

@Slider0007 Unfortunately the problem could not be solved.
We also noticed the issue with the offset, but it didn't help solve this problem.
What worked for me caused a problem for jasaw and vice versa.
Therefore I suspect that it has something to do with the camera's firmware.
Therefore, a note about this was added to the Tooltip.