Camera Calibration: Direct Linear Transform

Credits go to the following video. The rest is my note.

Direct Linear transform (DLT) maps world coords to image coords

$$\begin{gathered} \begin{array}{}\text{(1)}& \begin{array}{c}\mathit{x}\\ 3\times 1\end{array}=\begin{array}{c}\mathit{K}\\ 3\times 3\end{array}\begin{array}{c}\mathit{R}\\ 3\times 3\end{array}[\begin{array}{c}\mathit{I}\\ 3\times 3\end{array}|\begin{array}{c}-{\mathit{X}}_{\mathbf{0}}\\ 3\times 1\end{array}]\begin{array}{c}\mathit{X}\\ 4\times 1\end{array}=\begin{array}{c}\mathit{P}\\ 3\times 4\end{array}\begin{array}{c}\mathit{X}\\ 4\times 1\end{array} \\ \end{array} \end{gathered}$$

Given $M=6$ or more pairs of world coord ${\mathit{X}}_{\mathit{i}}$ and image coord ${\mathit{x}}_{\mathit{i}}$, figure out intrinsic parameter $\mathit{K}$, Rotation matrix $\mathit{R}$, and projection center ${\mathit{X}}_{\mathbf{0}}$. For each pair, ${\mathit{x}}_{\mathit{i}}=\mathit{P}{\mathit{X}}_{\mathit{i}}=\left[\begin{array}{c}{\mathit{A}}^{\mathit{T}}\\ {\mathit{B}}^{\mathit{T}}\\ {\mathit{C}}^{\mathit{T}}\end{array}\right]{\mathit{X}}_{\mathit{i}}$ can be rewritten as:

$$\begin{array}{}\text{(2)}& \left[\begin{array}{c}-{\mathit{X}}_i^T,{\mathbf{0}}^T,x_i{\mathit{X}}_i^T\\ {\mathbf{0}}^T,-{\mathit{X}}_i^T,y_i{\mathit{X}}_i^T\end{array}\right]\left[\begin{array}{c}\mathit{A}\\ \mathit{B}\\ \mathit{C}\end{array}\right]=\mathbf{0}\end{array}$$

We can stack $M$ of such equation together and get

$$\begin{array}{}\text{(3)}& \begin{array}{c}\mathit{M}\\ 2M\times 12\end{array}\begin{array}{c}\mathit{p}\\ 12\times 1\end{array}=\begin{array}{c}\mathbf{0}\\ 2M\times 1\end{array}\end{array}$$

Now the problem becomes:

Given $\mathit{M}$, find $\hat{\mathit{p}}=argma{x}_{\mathit{p}}{\mathit{p}}^{\mathit{T}}{\mathit{M}}^{\mathit{T}}\mathit{M}\mathit{p}$, such that $|\mathit{p}|=1$.

The solution to this is do a SVD of $\mathit{M}$, and set $\mathit{p}={\mathit{v}}_{\mathbf{12}}$. ${\mathit{v}}_{\mathbf{12}}$ is the eigenvector corresponds to the smallest singular value $s_{12}$ (all singular values are non-negative). Try proof this yourself. Hint: decompose $\mathit{p}$ into the space defined by ${\mathit{v}}_{\mathit{i}}$.

There are two corner cases that there is no solution: 1. All world coords are on the same plane. 2. all world coords and the projection center are locate on a twisted cubic curve.

After this, we get matrix $\hat{\mathit{P}}$.

$$\begin{array}{}\text{(4)}& \hat{\mathit{P}}=[\hat{\mathit{H}}|\hat{\mathit{h}}]=[\hat{\mathit{K}}\hat{\mathit{R}}|-\hat{\mathit{K}}\hat{\mathit{R}}\widehat{{\mathit{X}}_{\mathbf{0}}}]\end{array}$$

Therefore, $\widehat{{\mathit{X}}_{\mathbf{0}}}=-{\hat{\mathit{H}}}^{-1}\hat{\mathit{h}}$. QR decomposition gives a orthogonal matrix times a upper diagonal matrix, while what we need is a upper diagonal matrix times a orthogonal matrix. In order to achieve that, we QT decompose ${\hat{\mathit{H}}}^{-1}$ to get ${\hat{\mathit{R}}}^{-1}={\hat{\mathit{R}}}^T$ and ${\hat{\mathit{K}}}^{-1}$

Some final touchups: times $\hat{\mathit{K}}$ with $1/\widehat{K_{33}}$ to make $\widehat{K_{33}}=1$. $\widehat{K_{11}}$ and $\widehat{K_{22}}$ need to be positive, so throw the sign to rotation matrix $\hat{\mathit{R}}$.