Adventures in Aerial Image Stitching Episode #7

Lens Distortion Issues (or Fun with Optimizers)

For this episode I present a plot of optimized camera locations. I have a set of 840 images. Each image is taken from a specific location and orientation in space. Let us call that a camera pose. I can also find features in the images and match them up between pairs of images. The presumption is that all the features and all the poses together create a giant puzzle with only one correct solution. When all the features are moved to their correct 3d location, when all the cameras are in their correct poses, the errors in the system go to zero and we have a correct stitch.

However, look at the plot of camera pose locations. You can see a distinct radial pattern emerges. The poses at the fringes of the set have much different elevation (color) compared to the poses in the middle.

I have set up my optimizer to find the best fit for 3d feature location, camera pose (location and orientation) as well as solving for the focal length of the lens and the distortion parameters. (Sometimes this is referred to as ‘sparse bundle adjustment’ in the literature.)

Optimizers are great at minimizing errors, but they often do some strange things. In this case the optimizer apparently came up with wrong lens distortion parameters but then moved all the poses around the fringe of the set to compensate and keep the error metric minimized.

How can I fix this? My first try will return to a set of precomputed camera calibration and lens distortion parameters (based on taking lots of pictures of a checkerboard pattern.) I will rerun the optimization on just the 3d features and camera poses and see how that affects the final optimized camera locations. I can also set bounds on any of the parameters depending on how certain I am of the correct locations (which I’m not.)

Fun side note: the code that estimates camera calibration and lens distortion parameters is itself an optimizer. As such, it can distribute the error into different buckets and come up with a range of solutions depending on the input image set.

Jan 14, 2018 update: I have taken several steps forward understanding the issues here.

In the original scatter plot, I was plotting something called tvec. Anyone who has done significant camera pose work with opencv will recognize rvec and tvec. “tvec” gets rotated through the rvec transform to produce the 3d location of the camera pose, so plotting tvec itself was not useful. I have done the extra work to derive 3d point location and that makes a huge difference.

After plotting actual 3d camera pose location it became clear that globally optimizing the camera focal length and distortion parameters does seem to be far more productive and correct than attempting to optimize these parameters for each individual camera pose separately.

Giving the optimizer too much leeway in picking camera calibration and distortion paramters seems to lead to bad results. The closer I can set these at the start, and the tighter I can constrain them during the optimization, the better the final results.

A new issue is emerging. During the optimization, the entire system seems to be able to warp or tip. One corner of the plot seems lower and more compressed. The other side seems higher and more spread out. Here are some ideas for dealing with this:

I could apply a global affine transform to refit the cameras as best as possible to their original locations, however I would need to reproject and triangulate all the feature points to come up with their new 3d locations.

I could apply some sort of constraint to the camera locations. For example I could pretend I know location to +/- 10 meters and add that as a constraint to the camera pose location during the optimization. But do I know the relative positions this accurately?

Here is my latest plot of camera locations:

Jan 15, 2018 update: Here is a quick youtube video showing the optimizer in action. Each frame shows the result of an optimizer step. Altitude is encoded as color. The result isn’t perfect as you can tell from the various artifacts, but this is all a work in progress (and all open-source, built on top of python, numpy, scipy, and opencv.)