14: Using multiple images
The previous chapter discussed corner detectors which find particularly distinctive points in a scene that can be reliably detected in different views of the same scene irrespective of viewpoint or lighting conditions. However the 3-dimensional coordinate of the corresponding world point was lost in the perspective projection process which we discussed in Chapter 11 — we mapped a 3-dimensional world point to a 2-dimensional image coordinate. To recover the missing third dimension we need additional information which from multiple views of the same scene. This allows us to determine the 3D location of the point relative to the camera, and even more powerfully we can estimate the 3D motion of the camera between the views as well as the 3D structure of the world. This chapter covers:
- Finding corresponding features between two images, feature matching using Harris and SURF features
- Geometry of two views, epipolar plane, fundamental matrix, epipolar lines
- The essential matrix, its relationship to the fundamental matrix, and its relationship to camera translation and rotation
- Estimating the fundamental matrix from data, the effect of bad correspondence, the need for robust matching and RANSAC
- Planar homographies, estimation from data, relationship to camera translation and rotation, determining points within a plane
- Sparse stereo
- Dense stereo, disparity, matching measures and matching similarity, disparity space image, stereo failure modes, sub pixel disparity estimation
- 3D reconstruction, texture mapping, anaglyph
- Stereo image rectification
- Dealing with 3D points: plane fitting, matching sets of points (ICP)
- Structure and motion, the scale estimation problem
- Applications
- Perspective correction, remove keystone distortion from an image
- Mosaicing, stitch images into a mosaic
- Image matching and retrieval using the “bag of words” technique
- Image sequence processing, track features over time
Links
General
- The fundamental matrix song (audio)
- Anaglyph images of MARS (NASA)
- Epipolar geometry applets (Sylvain Bougnoux)
- Homography estimation from points and lines, masters essay (Elan Dubrofsky)
Stereo datasets
- Tsukuba stereo dataset 1800 stereo pairs with ground truth disparity maps, occlusion maps and discontinuity maps. Available as a 5GB download only.
- Middlebury stereo dataset has quite high resolution color images, some with ground truth disparity
- Mars stereo images in anaglyph format
- JISCT stereo dataset, perhaps the first ever stereo dataset, somewhat low resolution images by todays standards
Image sequence datasets
Software
- Peter Kovesi’s software for Computer Vision and Image processing.
- Point feature detectors from Chapter 13
- Epipolar geometry toolkit by Gian Luca Mariottini and Domenico Prattichizzo
- Structure from Motion toolkit for Matlab by Phil Torr
- The Structure from Motion Toolbox by Vincent Rigaud
- Simple Sparse Bundle Adjustment (SSBA) by Christoper Zach
- Sparse Bundle Adjustment (SBA) by Lourakis (in C)
- Image stitching Hugin autostitch
- Range Image Registration Toolbox, Carles Matabosch Geronès
- Multiple View geometry MATLAB functions, Hartley and Zisserman book