I have added some well-known hand-eye calibration methods:

• solve rotation then translation:
  • R. Tsai, R. Lenz A New Technique for Fully Autonomous and Efficient 3D Robotics Hand/EyeCalibration
  • F. Park, B. Martin Robot Sensor Calibration: Solving AX = XB on the Euclidean Group
  • R. Horaud, F. Dornaika Hand-Eye Calibration
• solve simultaneously rotation and translation:
  • N. Andreff, R. Horaud, B. Espiau On-line Hand-Eye Calibration
  • K. Daniilidis Hand-Eye Calibration Using Dual Quaternions

I have also moved the code to a new file calibration_handeye.cpp.

Many thanks to Mili Shah for providing a clean Matlab implementation for these algorithms.

M. Shah, R. D. Eastman, T. Hong, An Overview of Robot-Sensor Calibration Methods for Evaluation of Perception Systems, Performance Metrics for Intelligent Systems, (2012).

For now, I did not used Matx44d and explicit matrix shape since the conversion is not direct and requires more time and extra work.
Missing features or I did not found a way to do it:

• matrix slicing
• Matx::copyTo() to copy for example a Matx44d to a bigger matrix, something like:

Matx31d diff = Pcij - Pgij;
diff.copyTo(B(Rect(0, idx*3, 1, 3)));

• Matx and Mat cannot be mixed. For instance with matrix multiplication:

    Mat M1 = (Mat_<double>(4,4,CV_64FC1) << 1, 2, 3, 4,
             5, 6, 7, 8,
             9, 10, 11, 12,
             13, 14, 15, 16);
    Matx44d M2(1, 2, 3, 4,
               5, 6, 7, 8,
               9, 10, 11, 12,
               13, 14, 15, 16);
    std::cout << "M1*M1:\n" << M1*M1 << std::endl; //OK
    std::cout << "M2*M2:\n" << M2*M2 << std::endl; //OK
    std::cout << "M1*M2:\n" << M1*M2 << std::endl; //NOK
    std::cout << "M1*Mat(M2):\n" << M1*Mat(M2) << std::endl; //OK

Also, I have found a weird thing, probably a missing operator:

cv::Matx33d skew2(const cv::Matx31d& v)
{
  double vx = v(0,0);
  double vy = v(1,0);
  double vz = v(2,0);
  return Matx33d(0, -vz, vy,
                 vz, 0, -vx,
                 -vy, vx, 0);
}
int main()
{
    Matx31d P1(1, 2, 3);
    std::cout << "skew(P1/2):\n" << skew2(P1/2) << std::endl; //NOK
    std::cout << "skew(P1*2):\n" << skew2(P1*2) << std::endl; //OK
}

Some functions that I had to defined but are worth or could be available in OpenCV:

• skew matrix from a vector
• conversion functions between rotation matrix and quaternion, quaternion type like in Eigen, quaternion multiplication, ...?
• homogeneous matrix inverse?
• kronecker product, I had to take the implementation from here. Since it should be also the same thing than tensor product, maybe there is already this operation in the DNN module?

@catree Could you please fill an issue about Matx multiplication with your code? I will try to look on it later.

Thanks for the review.

@catree Can I put the params which from "solvePnp->Rodrigues" into "calibrateHandEye" directly? And I get the "rx ry rz" euler angles from my robot, which kinds of method can help me to get the rotation matrix which exactly match "calibrateHandEye"?

@msnh2012

For R_target2cam, you can use findChessboardCorners() to get the chessboard corners and solvePnP() to estimate the chessboard pose with respect to the camera frame. solvePnP() returns a Rodrigues rotation vector so you can use Rodrigues() to get the corresponding rotation matrix.

For R_gripper2base, if your robot framework returns "rx, ry, rz" Euler angles, you have to check the manufacturer documentation to properly convert the Euler angles to the rotation matrix according to the chosen convention.

The best would be to check if you can get directly either the rotation matrix or quaternion. With quaternion, the only pitfall is to be sure to get the correct "qx, qy,qz, qw" values.

@catree 感谢如此快的回复，从solvePnp到Rodrigues输出的旋转矩阵转成欧拉角，采用哪种旋向？Z-Y-X还是X-Y-Z？然后calibrateHandEye采用的是统一的Z-Y-X还是X-Y-Z旋向？还是只考虑旋转矩阵，没考虑由欧拉角转换得到的旋转矩阵这种情况？然后就是对于标定的结果，有没有一种误差的评定方法？谢谢:D

@msnh2012 Please write in English.

calibrateHandEye needs 3x3 rotation matrix. Depending on what you have, the possible conversions are:

• Axis-angle to rotation matrix, done in cv::Rodrigues()
• Quaternion to rotation matrix
• Euler angles to rotation matrix

This is the representation of the Axis-Angle rotation formalism, also called rotation vector and also called Rodrigues vector:

You should read a texbook on this subject, for instance. There are plenty of useful ressources on the net if you want more information.

@catree Thanks! Can u give an example of hand-eye calibration for us? That will be very nice .

@msnh2012

You can find here some example code.

@catree Thanks! 👍

This pullrequest changes

Add Hand-Eye calibration from Tsai method. This allows estimating the rigid transformation between a camera mounted on a robot gripper:

See also:

• Hand-Eye Calibration
• A new technique for fully autonomous and efficient 3D robotics hand/eye calibration, R.Y. Tsai ; R.K. Lenz

The current interface is:

void calibrateHandEyeTsai( InputArrayOfArrays R_gripper2base, InputArrayOfArrays t_gripper2base,
                                            InputArrayOfArrays R_target2cam, InputArrayOfArrays t_target2cam,
                                            OutputArray R_cam2gripper, OutputArray t_cam2gripper );

• currently there is only one method (Tsai). Should we keep this or use a more generic one calibrateHandEye with a method parameter to choose different calibration methods? This can be a more general question also (e.g. calibrateCamera, calibrateCameraRO, fitEllipse, fitEllipseAMS, fitEllipseDirect, ...).
• inputs are rotation matrices and translation vectors. It could have been rotation and translation vectors
• naming is consistent with the other functions, alternative is HandEyeCalibration()
• code follows the paper notations
• C++ code from this Matlab code

Is it necessary to include more hand-eye calibration methods? It seems that most of these analytical methods do not work well for real-world tasks.

@zarathustr
I think that the best would be to have a non-linear optimization method. The few methods I had seen were using the Ceres solver. But I have no experience with it, and it can be tricky to integrate in OpenCV.

Feel free to contribute in OpenCV (BSD-3 license) if you have better methods.