Help Needed: Camera Calibration for Tello Drone Adaptation

[d4saher]

Hi jyjblrd,

First of all, thank you so much for sharing your code and making it available to the community. I really appreciate it as I will be using it in my final degree project as a ground truth.

I wanted to let you know that I have successfully got the system up and running. I am currently working on adapting the project for the Tello drone to simplify the drone part. Additionally, I would like to contribute to the project by sharing the code I write and some 3D pieces that I have designed. The pieces are basically a wall mount for the cameras and a support for mounting the IR LEDs on the Tello Drone EDU. You can find them on my fork repository of your project, on the tello branch.

I have also written a step-by-step guide to get the project running on a Rock (Pi) 4 c+. I wrote this guide for myself, but maybe someone else will find it useful. It can be found here.

However, at this point, I am encountering an issue with camera pose calibration. I am able to get points (20 to 30 as you mentioned here), but when I run the camera pose calibration, it doesn't reach a solution. Using only two cameras seems to work better. Do I need to do this in pairs as you say in your video, or does the backend handle it? I followed the same method you used with the PS3 Eye cameras, removing the IR filter. Do I need to calculate the intrinsic matrix anyway, or is it enough to use the ones provided in your code?

Thank you once again for your amazing work!

Best regards,
D4saher

[jyjblrd]

Hi,

Thank you, I appreciate the contributions and guide.

The code automatically does the camera pose calibration in pairs. I would recommend generating new camera parameters for your cameras, as it can make a big difference. Instructions to do so are here: #11 (comment)

The cameras should also be ordered in a clockwise/anti-clockwise pattern, so that adjacent cameras are also adjacent on the dashboard.

[d4saher]

Hi,

This is what I got so far:
In the acquire_floor function, I am not sure, but I think there is a mistake with the up_normal. Originally, it is up_normal = np.array([[0],[0],[1]], dtype=np.float32). But since three.js uses a right-handed coordinate system, I think it should be up_normal = np.array([[0],[1],[0]], dtype=np.float32).

With that change, I got the cameras almost perpendicular to the floor, so I also had to apply a 90-degree rotation to the x-axis:

rotationMatrixX = np.array([
    [1, 0, 0],
    [0, 0, -1],
    [0, 1, 0]
])

Now it seems almost correct to me:

Need to check with cameras on Monday as I don't have them with me this weekend.

What do you think? Does it make sense?

[jyjblrd]

If I'm honest, the calibration stuff is not very well written and is quite hacky, especially since threejs uses a different coordinate system to the backend. I would like to clean it up one day, or perhaps someone else can.

Have you had any success?

[nbpilot007]

Hi,

This is what I got so far: In the acquire_floor function, I am not sure, but I think there is a mistake with the up_normal. Originally, it is up_normal = np.array([[0],[0],[1]], dtype=np.float32). But since three.js uses a right-handed coordinate system, I think it should be up_normal = np.array([[0],[1],[0]], dtype=np.float32).

With that change, I got the cameras almost perpendicular to the floor, so I also had to apply a 90-degree rotation to the x-axis:

rotationMatrixX = np.array([
    [1, 0, 0],
    [0, 0, -1],
    [0, 1, 0]
])

Now it seems almost correct to me:

Need to check with cameras on Monday as I don't have them with me this weekend.

What do you think? Does it make sense?

I'm facing the same issue. Did it solve your problem after changing the matrix? If yes, can you please share the entire modified function where you made the changes? Thanks

[d4saher]

If I'm honest, the calibration stuff is not very well written and is quite hacky, especially since threejs uses a different coordinate system to the backend. I would like to clean it up one day, or perhaps someone else can.

Have you had any success?

Hi jyjblrd,

I apologize for the delay in responding; I've been away for a couple of weeks. Unfortunately, the last solution I proposed did not work as expected. I'm still investigating the cause of the issue. Additionally, I'm working on cleaning up the code and adapting it to better fit programming design patterns.

I have a question: when plotting the captured points on the graph, should they be drawn in the same direction as in reality? Right now, if we look at the graph as if it were the room from above and I draw a stroke with the marker from left to right, it gets plotted from right to left on the graph. I’m not sure if this might be affecting something.

I'll keep you updated on my progress.

Thanks,

[d4saher]

Hi,
This is what I got so far: In the acquire_floor function, I am not sure, but I think there is a mistake with the up_normal. Originally, it is up_normal = np.array([[0],[0],[1]], dtype=np.float32). But since three.js uses a right-handed coordinate system, I think it should be up_normal = np.array([[0],[1],[0]], dtype=np.float32).
With that change, I got the cameras almost perpendicular to the floor, so I also had to apply a 90-degree rotation to the x-axis:

rotationMatrixX = np.array([
    [1, 0, 0],
    [0, 0, -1],
    [0, 1, 0]
])

Now it seems almost correct to me:
Need to check with cameras on Monday as I don't have them with me this weekend.
What do you think? Does it make sense?

I'm facing the same issue. Did it solve your problem after changing the matrix? If yes, can you please share the entire modified function where you made the changes? Thanks

Hi, I am still having the same problem, trying to figure out what is the problem...

[d4saher]

Hi,

I am still trying to figure out where the problem is, but meanwhile, I have modified the program to allow manual adjustment of the scene with some buttons. I will be unavailable this weekend, so I won't be able to test if it works yet. If someone wants to try, the changes are on my fork repository on the dev branch: https://github.com/d4saher/Low-Cost-Mocap.git

Note that I am working on splitting the frontend into views to make the code more readable and modular, but I haven't added the drone control parts yet.

[jyjblrd]

Sorry, I have also been on holiday the last few weeks. I am hoping to refine the code for this project too at some point, as the calibration step is quite confusing currently (both in the code and in practice...)

I'm not sure what your specific problem is, it has worked for me and others so perhaps it is something to do with your setup? Perhaps you can send some pictures?

[d4saher]

Don't worry, I greatly appreciate your effort and dedication.

I'll try to send some photos this afternoon. Meanwhile, I had a thought: maybe in the backend function that calculates the floor, the calculated rotation isn't being applied to the old toWorldCoordsMatrix. Could this be the issue?

@socketio.on("acquire-floor")
def acquire_floor(data):
    cameras = Cameras.instance()
    object_points = data["objectPoints"]
    object_points = np.array([item for sublist in object_points for item in sublist])

    tmp_A = []
    tmp_b = []
    for i in range(len(object_points)):
        tmp_A.append([object_points[i,0], object_points[i,1], 1])
        tmp_b.append(object_points[i,2])
    b = np.matrix(tmp_b).T
    A = np.matrix(tmp_A)

    fit, residual, rnk, s = linalg.lstsq(A, b)
    fit = fit.T[0]

    plane_normal = np.array([[fit[0]], [fit[1]], [-1]])
    plane_normal = plane_normal / linalg.norm(plane_normal)
    up_normal = np.array([[0],[0],[1]], dtype=np.float32)

    plane = np.array([fit[0], fit[1], -1, fit[2]])

    # https://math.stackexchange.com/a/897677/1012327
    G = np.array([
        [np.dot(plane_normal.T,up_normal)[0][0], -linalg.norm(np.cross(plane_normal.T[0],up_normal.T[0])), 0],
        [linalg.norm(np.cross(plane_normal.T[0],up_normal.T[0])), np.dot(plane_normal.T,up_normal)[0][0], 0],
        [0, 0, 1]
    ])
    F = np.array([plane_normal.T[0], ((up_normal-np.dot(plane_normal.T,up_normal)[0][0]*plane_normal)/linalg.norm((up_normal-np.dot(plane_normal.T,up_normal)[0][0]*plane_normal))).T[0], np.cross(up_normal.T[0],plane_normal.T[0])]).T
    R = F @ G @ linalg.inv(F)

    R = R @ [[1,0,0],[0,-1,0],[0,0,1]] # i dont fucking know why

    cameras.to_world_coords_matrix = np.array(np.vstack((np.c_[R, [0,0,0]], [[0,0,0,1]])))

    socketio.emit("to-world-coords-matrix", {"to_world_coords_matrix": cameras.to_world_coords_matrix.tolist()})

cameras.to_world_coords_matrix = np.array(np.vstack((np.c_[R, [0,0,0]], [[0,0,0,1]]))) Here, the matrix is replaced, but the rotation is not applied to the old one. Shouldn't it be applied?

[d4saher]

Here are some photos of my setup. As you can see I have 4 cams each one on a corner of the room: