# Stereo calibration using C++ and OpenCV

## September 9, 2016

# Introduction

Before viewing this, it is recommended that you know how to calibrate a single camera and what is meant by calibrating a camera. You can find a tutorial for the same here:

If you’re just looking for the code, the full implementation can be found here:

# Stereo Setup

The following two images describe a stereo camera setup.

We will use the checkerboard method for calibration. It is required that the intrinsics of each camera be known beforehand. Recall the camera matrix $\mathbf{K}$ as

where $f_x$ and $f_y$ are the focal length of the camera in the x-axis and the y-axis respectively. $(c_x, c_y)$ is coordinate of the principal point. Therefore $\mathbf{K_1}$ and $\mathbf{K_2}$ should be known for both cameras before moving on with stereo calibration.

Stereo calibration will essentially find out the rotation $\mathbf{R}$ and translation $\mathbf{t}$ between both the cameras and this will help us find point correspondences in the left and right image planes.

Let $\mathbf{x}$ and $\mathbf{x’}$ be a point in the left image and right image respectively, then the correspondence relation is defined by the fundamental matrix $\mathbf{F}$ as

You can find a tutorial to calculate $\mathbf{F}$ given $\mathbf{K_1}$, $\mathbf{K_2}$, $\mathbf{R}$, and $\mathbf{t}$ here:

http://sourishghosh.com/2016/fundamental-matrix-from-camera-matrices/

It is advised that in case you do not understand the theory behind stereo vision, then a little bit of reading is to be done before you proceed with this just to get a feel of what is happening and what all these symbols/variables mean. The math behind stereo vision is mostly projective geometry and matrix algebra.

# Dependencies and Datasets

You must have `OpenCV 2.4.8+`

and `libpopt`

(command line args) to run the code. Also, you should have a dataset of calibration left/right image pairs beforehand of a fixed image resolution. Here are is a sample left/right image pair.

Left Image | Right Image |
---|---|

I have made two of my own image sets available here:

https://github.com/sourishg/stereo-calibration/tree/master/calib_imgs

It is recommended to get at least 30 image pairs of the checkerboard in all possible orientations of the checkerboard to get good calibration results.

**Note**: In this example, a standard 9x6 calibration board is used. The size of the square is 24.23 mm.

# Stereo Calibration

The code is almost similar to the one explained here. I will only explain the important parts of the code, and you can find the full source here:

https://github.com/sourishg/stereo-calibration/blob/master/calib_stereo.cpp

Declare all the necessary vectors to store the image points and the object points. Image points are the checkerboard corner coordinates in the image whereas object points are the actual 3D coordinate of those checkerboard points.

We create a function called `load_image_points`

to find all the corner points of each image and their corresponding 3D world points and prepare the `object_points`

, `left_img_points`

, and `right_img_points`

vectors. `board_n`

is the total number of corner points in the checkerboard. In our example it is $9\times 6=54$. Note that we also take a bunch of args, but I hope the variable names are self explanatory. Also note that the object points would be same for both the left and right images.

We loop through all the images in our directory and convert them to grayscale images using the function `cv::cvtColor`

. `leftimg_dir`

is the directory containing the left images and `leftimg_filename`

is the prefix for each image file name. Similar concept for the right images.

Next we use the `findChessboardCorners`

function to find all the checkerboard corners. I would suggest you go through the OpenCV documentation for more details about the arguments of this function. If the corners are found then `found`

is set to `true`

and the corners are further refined by the `cornerSubPix`

function. The `drawChessboardCorners`

function is optional, it only helps you visualize the checkerboard corners found.

Next we store the object points. Ideally we should keep the origin at the camera centre and measure the 3D points of the checkerboard corners manually but you can image how difficult it would be. So we introduce a small but beautiful hack - we keep the world origin as the top left corner point. Mathematically this doesn’t change anything (think how). Now the geometry of the checkerboard helps us find the other 3D coordinates of the corners very easily. The $Z$ coordinate is always $0$ since all the points lie on a plane. Since the square size for this example is 24.23 mm (units are important!) then the other points become $(24.23, 0, 0)$, $(48.46, 0, 0)$ and so on.

Note that if corners are found for **both** the left and right images then only the points are stored, otherwise that image pair is ignored.

We get all the necessary user input using `libpopt`

and call the `load_image_points`

function. Also we input the individual camera matrices using the objects `fsl`

and `fsr`

.

Next we read the intrinsics and distortion coefficients for each camera and store them in `Mat`

format. We set `flag`

to `CV_CALIB_FIX_INTRINSIC`

which tells the `stereoCalibrate`

function to not guess the individual intrinsics for each camera. `F`

stores the fundamental matrix, `E`

stores the essential matrix, `R`

stores the rotation from the left to the right camera, `T`

stores the translation from the left to the right camera.

Next we store all the calibration data in a `YAML`

file so that we don’t have to recalibrate again if we are using the same setup.

Note:If you disturb the stereo setup anyhow, by either rotating or moving one camera slightly, then you would have to recalibrate again.

# Stereo Rectification

Stereo rectification is the task of applying a projective transformation to both image planes such that the resulting epipolar lines become horizontal scan lines. You can read up on the theory here:

http://www.sci.utah.edu/~gerig/CS6320-S2013/Materials/CS6320-CV-F2012-Rectification.pdf

`R1`

is the rectification transform for the left camera, `R2`

for the right camera. `P1`

is projection matrix in the new rectified coordinate system for the left camera, `P2`

for the right camera. `Q`

is known as the disparity-to-depth mapping matrix. `Q`

is a very important matrix and it is of immense use during 3D reconstruction.

# Building the Code

The following repository contains the full source. The file you are looking for is `calib_stereo.cpp`

I have used `cmake`

to build the source and the README should help you build and run the program on your machine.