Point mapping

Point mapping

Direct Linear Transformation

Compute the 11 intrinsic and extrinsic parameters of a camera.

Zhang’s Method

Compute the 5 intrinsic parameters in \(K\).

Zhang’s Method

Camera calibration using images of a checkerboard.

Checkerboard

• Board is of known size and structure.
• The board must be flat.

Checkerboard Method

Set the world coordinate system to the corner of the checkerboard.

Simplification

The \(Z\) coordinate of each point is zero.

\[ \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = \begin{bmatrix} c & s & x_H \\ 0 & c(1 + m) & y_H \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} r_{11} & r_{12} & r_{13} & t_1 \\ r_{21} & r_{22} & r_{23} & t_2 \\ r_{31} & r_{32} & r_{33} & t_3 \end{bmatrix} \begin{bmatrix} X \\ Y \\ \color{red}{Z} \\ 1 \end{bmatrix} \]

Simplification

The last column of the rotation matrix has no effect on the system.

• we can delete these components from the system

\[ \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = \begin{bmatrix} c & s & x_H \\ 0 & c(1 + m) & y_H \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} r_{11} & r_{12} & \color{red}{r_{13}} & t_1 \\ r_{21} & r_{22} & \color{red}{r_{23}} & t_2 \\ r_{31} & r_{32} & \color{red}{r_{33}} & t_3 \end{bmatrix} \begin{bmatrix} X \\ Y \\ \color{red}{Z} \\ 1 \end{bmatrix} \]

Simplification

• The \(Z\) coordinate of each point is zero.
• Deleting the third column of \(R\) gives us:

\[ \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = \begin{bmatrix} c & s & x_H \\ 0 & c(1 + m) & y_H \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} r_{11} & r_{12} & t_1 \\ r_{21} & r_{22} & t_2 \\ r_{31} & r_{32} & t_3 \end{bmatrix} \begin{bmatrix} X \\ Y \\ 1 \end{bmatrix} \]

Simplification

• Each observed point gives this equation.
• The intrinsics persist for all images.
• The extrinsics persist for each image.

\[ \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = \begin{bmatrix} c & s & x_H \\ 0 & c(1 + m) & y_H \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} r_{11} & r_{12} & t_1 \\ r_{21} & r_{22} & t_2 \\ r_{31} & r_{32} & t_3 \end{bmatrix} \begin{bmatrix} X \\ Y \\ 1 \end{bmatrix} \]

Setting up the equations

Define a matrix \(H\):

\[ H = \begin{bmatrix} \textbf{h}_1 , \textbf{h}_2 , \textbf{h}_3 \end{bmatrix} = \begin{bmatrix} c & s & x_H \\ 0 & c(1 + m) & y_H \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} r_{11} & r_{12} & t_1 \\ r_{21} & r_{22} & t_2 \\ r_{31} & r_{32} & t_3 \end{bmatrix} \]

One point generates this equation:

\[ \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = H \begin{bmatrix} X \\ Y \\ 1 \end{bmatrix} \]

Setting up the equations

For multiple point observations:

\[ \begin{bmatrix} x_i \\ y_i \\ 1 \end{bmatrix} = \underset{3 \times 3}{H} \begin{bmatrix} X_i \\ Y_i \\ 1 \end{bmatrix}, \quad i = 1 ..., n \]

Analogous to the DLT.

Parameter Estimation

We estimate a \(3 \times 3\) homography instead of \(3 \times 4\) projection.

\[ a_{x_i}^T \textbf{h} = 0, \quad a_{y_i}^T \textbf{h} = 0 \]

with:

\[ \begin{aligned} \textbf{h} &= vec(H^T) \\ a_{x_i}^T &= (-X_i, -Y_i, -1, 0, 0, 0, x_i X_i, x_i Y_i, x_i) \\ a_{y_i}^T &= (0, 0, 0, -X_i, -Y_i, -1, y_i X_i, y_i Y_i, y_i) \end{aligned} \]

Parameter Estimation

Solving the system of linear equations leads to an estimate of the parameters of \(H\).

• We need to identify at least 4 points.
• \(H\) has 8 Dof (degrees of freedom)
• each point provides 2 observations

We now have the parameters of \(H\), how do we find \(K\)?

Decompose Intrinsic Parameters

For the DLT, we could use QR decomposition to find the rotation matrix of the extrinsic parameters.

• We can not do this for Zhang’s method.
• We eliminated part of \(R\) earlier.

Decompose Intrinsic Parameters

\[ H = \begin{bmatrix} \textbf{h}_1 , \textbf{h}_2 , \textbf{h}_3 \end{bmatrix} = \underbrace {\begin{bmatrix} c & s & x_H \\ 0 & c(1 + m) & y_H \\ 0 & 0 & 1 \end{bmatrix}}_{K} \underbrace {\begin{bmatrix} r_{11} & r_{12} & t_1 \\ r_{21} & r_{22} & t_2 \\ r_{31} & r_{32} & t_3 \end{bmatrix}}_{[\textbf{r}_1, \textbf{r}_2, \textbf{t}]} \]

Decompose Intrinsic Parameters

We need to extract \(K\) from the matrix \(H = K[\textbf{r}_1, \textbf{r}_2, \textbf{t}]\) we computed using SVD.

Decompose Intrinsic Parameters

We need to extract \(K\) from the matrix \(H = K[\textbf{r}_1, \textbf{r}_2, \textbf{t}]\) we computed using SVD.

Four step process:

Exploiting Constraints

\[ K = \begin{bmatrix} c & s & x_H \\ 0 & c(1 + m) & y_H \\ 0 & 0 & 1 \end{bmatrix} \]

\(K\) is invertible.

Exploiting Constraints

\[ H = \begin{bmatrix} \textbf{h}_1 , \textbf{h}_2 , \textbf{h}_3 \end{bmatrix} = \underbrace {\begin{bmatrix} c & s & x_H \\ 0 & c(1 + m) & y_H \\ 0 & 0 & 1 \end{bmatrix}}_{K} \underbrace {\begin{bmatrix} r_{11} & r_{12} & t_1 \\ r_{21} & r_{22} & t_2 \\ r_{31} & r_{32} & t_3 \end{bmatrix}}_{[\textbf{r}_1, \textbf{r}_2, \textbf{t}]} \]

\[ [\textbf{r}_1, \textbf{r}_2, \textbf{t}] = K^{-1} [\textbf{h}_1 , \textbf{h}_2 , \textbf{h}_3] \]

\[ \Rightarrow \textbf{r}_1 = K^{-1} \textbf{h}_1, \quad \textbf{r}_2 = K^{-1} \textbf{h}_2 \]

Exploiting Constraints

As \([\textbf{r}_1 , \textbf{r}_2 , \textbf{r}_3]\) are the columns of a rotation matrix, they form an orthonormal basis.

\[ \textbf{r}_1^T \textbf{r}_2 = 0, \quad ||\textbf{r}_1|| = ||\textbf{r}_2|| = 1 \]

Exploiting Constraints

\[ \textbf{r}_1 = K^{-1} \textbf{h}_1, \quad \textbf{r}_2 = K^{-1} \textbf{h}_2, \quad \textbf{r}_1^T \textbf{r}_2 = 0, \quad ||\textbf{r}_1|| = ||\textbf{r}_2|| = 1 \]

\[ \textbf{h}_1^T K^{-T} K^{-1} \textbf{h}_2 = 0 \]

\[ \begin{aligned} \textbf{h}_1^T K^{-T} K^{-1} \textbf{h}_1 = \textbf{h}_2^T K^{-T} K^{-1} \textbf{h}_2 \\[10pt] \textbf{h}_1^T K^{-T} K^{-1} \textbf{h}_1 - \textbf{h}_2^T K^{-T} K^{-1} \textbf{h}_2 = 0 \end{aligned} \]

Exploiting Constraints

\[ \textbf{h}_1^T K^{-T} K^{-1} \textbf{h}_2 = 0 \]

\[ \textbf{h}_1^T K^{-T} K^{-1} \textbf{h}_1 - \textbf{h}_2^T K^{-T} K^{-1} \textbf{h}_2 = 0 \]

Exploiting Constraints

Define a matrix \(B := K^{-T}K^{-1}\)

\[ \textbf{h}_1^T B \textbf{h}_2 = 0 \]

\[ \textbf{h}_1^T B \textbf{h}_1 - \textbf{h}_2^T B \textbf{h}_2 = 0 \]

Exploiting Constraints

From \(B\) the calibration matrix can be recovered using Cholesky decomposition.

\[ B = \begin{bmatrix} b_{11} & b_{12} & b_{13} \\ b_{21} & b_{22} & b_{23} \\ b_{31} & b_{32} & b_{33} \end{bmatrix} \]

\[ chol(B) = AA^T \Rightarrow A = K^{-T} \]

If we know \(B\), we can recover the calibration matrix \(K\).

Exploiting Constraints

What do we have so far?

\[ \textbf{h}_1^T B \textbf{h}_2 = 0 \]

\[ \textbf{h}_1^T B \textbf{h}_1 - \textbf{h}_2^T B \textbf{h}_2 = 0 \]

• Matrix \(B\), which is symmetric positive, so 6 unknowns.
• \(\textbf{h}\) are known.
• Two equations that relate \(B\) and \(\textbf{h}\).

Exploiting Constraints

Define a vector \(\textbf{b} = (b_{11}, b_{12}, b_{13}, b_{22}, b_{23}, b_{33})\)

\[ B = \begin{bmatrix} \color{red}{b_{11}} & \color{red}{b_{12}} & \color{red}{b_{13}} \\ b_{12} & \color{red}{b_{22}} & \color{red}{b_{23}} \\ b_{13} & b_{23} & \color{red}{b_{33}} \end{bmatrix} \]

There are 6 unknowns in \(B\), because it is symmetric.

Exploiting Constraints

Construct a system of equations \(V\textbf{b}=0\) exploiting our constraints.

\[ v^{T}_{12}\textbf{b} = 0, \quad v^{T}_{11}\textbf{b} - v^{T}_{22}\textbf{b} = 0 \]

Matrix \(V\)

The matrix \(V\) is given by:

\[ V = \begin{bmatrix} v^{T}_{12} \\ v^{T}_{11} - v^{T}_{22} \end{bmatrix}, \quad with \quad v_{ij} = \begin{bmatrix} h_{1i}h_{1j} \\ h_{1i}h_{2j}+h_{2i}h_{1j} \\ h_{3i}h_{1j}+h_{1i}h_{3j} \\ h_{2i}h_{2j} \\ h_{3i}h_{2j}+h_{2i}h_{3j} \\ h_{3i}h_{3j} \end{bmatrix} \]

Matrix \(V\)

For each image we get:

\[ \begin{bmatrix} v^{T}_{12} \\ v^{T}_{11} - v^{T}_{22} \end{bmatrix} \textbf{b} = 0 \]

Matrix \(V\)

For multiple images we stack the matrices to a \(2n \times 6\) matrix:

\[ \begin{bmatrix} v^{T}_{12} \\ v^{T}_{11} - v^{T}_{22} \\ \dots \\ v^{T}_{12} \\ v^{T}_{11} - v^{T}_{22} \end{bmatrix} \textbf{b} = 0 \]

We need to solve the linear system of \(V\textbf{b}=0\) to find \(b\) and hence \(K\).

Solving the Linear System

The system \(V\textbf{b}=0\) has a trivial solution when \(\textbf{b}=0\) which will not provide a valid matrix \(B\).

• Apply additional constraint \(||\textbf{b}|| = 1\) .

Solving the Linear System

Real world measurements are noisy.

• Find the solution that minimises the least squares error:

\[ b^* = arg\underset{b}{min}||V \textbf{b}|| \quad \text{with} \quad ||\textbf{b}|| = 1 \]

Use SVD and choose the singular vector corresponding to the smallest singular value.

Minimum Requirements

• At least 4 points in each target image.
• Each target image gives two equations.
• \(B\) has 6 DoF so we need 3 different views of the target.
• Solve \(V\textbf{b}=0\) using SVD to compute \(K\).

Non-Linear Distortion

A general calibration matrix is obtained by multiplying the affine camera with a general mapping.

\[ {}^{a}H(\textbf{x}, q)K = \begin{bmatrix} 1 & 0 & x \Delta(\textbf{x}, q) \\ 0 & 1 & y \Delta(\textbf{x}, q) \\ 0 & 0 & 1 \end{bmatrix} \]

Lens Distortion

Radial Distortion

A standard approach for radial distortion:

\[ \begin{aligned} {}^{a}x &= x(1 + q_1r^2 + q_2r^4) \\ {}^{a}y &= y(1 + q_1r^2 + q_2r^4) \end{aligned} \]

• with \([x, y]^{T}\) a point projected by the ideal camera.
• with \(r\) the distance from the camera principal point to the pixel.
• \(q_1\) and \(q_2\) are the radial distortion parameters.

Lens Distortion

Lens distortion can be calculated by minimising a non-linear function.

• Make an initial guess for the distortion parameters.
• Calculate \(K\) using Zhang’s method.
• Measure the reprojection error.
• Refine the distortion parameters.