Department of Computer Engineering
University of California at Santa Cruz
Epipolar Geometry and the Eight-Point Algorithm
CMPE 264: Image Analysis and Computer Vision
Hai Tao
Department of Computer Engineering
University of California at Santa Cruz
Camera motion estimation and 3D reconstruction
Q
Q
Q
Problem statement: Given multiple images of a static scene
and the camera parameters, recover the camera motion and
the 3D structure of the scene
The image feature points corresponding to a set of 3D
points are first detected in these images. Then the camera
motion and the 3D coordinates of these points are
estimated based on multiple view geometry.
We will focus on multiple view geometry first and then
discuss how to find feature correspondences across images
Department of Computer Engineering
University of California at Santa Cruz
Two view geometry
Q
Notations
Pl = [ X l , Yl , Z l ]T - the 3D coordinates of a point P in the left camera coordinate
system
Pr = [ X r , Yr , Z r ]T - the 3D coordinates of a point P in the right camera coordinate
system
T
pl = [ xl , yl ,1]
- the coordinates of p in the left image
pr = [ xr , yr ,1]T - the coordinates of p in the right image
Pr = R( Pl − T ) - the transformation between the two camera coordinate
systems
Department of Computer Engineering
University of California at Santa Cruz
Two-view geometry
Q
Fundamental matrix
Since Ol P , Ol Or , and Or P are coplanar, therefore
(Or P)Tl (Ol Or × Ol P) = (Or P)Tl (T × Pl ) = 0
According to the transformation between the two camera frames
therefore
(Or P )l = ( Pl − T ) = ( R −1Pr + T − T ) = RT Pr
PrT R(T × Pl ) = PrT ( RT× ) Pl = 0
 0

T× =  Tz
 − Ty

− Tz
0
Tx
Ty 

− Tx 
0 
so that T × P = T× P
−1
Since K −1 pl ≅ Pl and K pr ≅ Pr
PrT ( RT× ) Pl ≅ prT ( K −T RT× K −1 ) pl = prT Fpl = 0
Department of Computer Engineering
University of California at Santa Cruz
Fundamental matrix
Q
Q
F = K −T RT× K −1 is called the fundamental matrix. It links correspondences
in two images through a linear relation prT Fpl = 0 . E = RT× is called the
essential matrix
Properties of the fundamental matrix
• F is defined up to a scale factor and is rank 2 – because T× is rank 2
• If el is the image of Or in the left image, then el is called the left epipole and
vice versa
• The plane formed by Ol , Or , and P is called epipolar plane
• The intersection of the epipolar plane and an images plane is called epipolar
line.
• All epipolar lines passing through the epipoles.
• Except epipoles, only one epipolar line passing through each image point
• Corresponding points must lie on conjugated epopolar lines
• Fel = 0 and F T er = 0
(Homework)
• el is the null space of F
• Fpl is the epipolar line in the right image; F T pr is the epipolar line in the
left image (Homework)
Department of Computer Engineering
University of California at Santa Cruz
Computing F and E – the eight-point algorithm
Q
Q
Each pair of correspondences gives one constraint. To recover the
fundamental matrix up to a scale factor, we need eight constraints, i.e.
eight pairs of correspondences.
For the ith correspondence
priT Fpli = 0 ⇔
[ xri xli , xri yli , xri , yri xli , yri yli , yri , xli , yli ,1][ f11, f12 , f13 , f 21 , f 22 , f 23 , f31, f 32 , f33 ]T = 0
Q
With N ≥ 8 correspondences, we have a linear system
Af = 0
Q
Q
where A is formed by stacking the data vectors as indicated above
The solution is the eigenvector corresponding to the smallest
eigenvalue of AT A
To compute the essential matrix, use the relation E = K T FK
To make the algorithm numerical stable, normalize image coordinates
so that RMS is 2
Department of Computer Engineering
University of California at Santa Cruz
From essential matrix to camera motion R and T
Q
Q
Theorem 1. A 3 by 3 real matrix Q can be factored as the product of a
rotation matrix and a non-zero skew symmetric matrix iff Q has two
equal non-zero singular values and one singular value equal to 0
Theorem 2. Suppose Q can be factored into a product RS where R is
orthonormal and S is skew symmetric. Let the SVD of Q be Q = UDV T.
Then up to a scale factor the factorization is one of the following
S = VZV T
R = UGV T or R = UGTV T
Q = RS
where
0 − 1 0 
Z = 1 0 0


0 0 0
 0 1 0
G =  − 1 0 0


 0 0 1
since Sv 3 = VZV T v 3 = [ v 2 ,− v1 ,0][ v1, v 2 , v 3 ]T v 3 = v 2 v1T v 3 − v1vT2 v 3 = 0
therefore T = λv 3 = λ[0,0,1]V , where λ is an arbitrary scale factor
Department of Computer Engineering
University of California at Santa Cruz
From essential matrix to camera motion R and T
Q
Algorithm From_E_to_RT
• Find the singular value decomposition E = UDV T
• The (R,T) pair is one of the following four solutions
(UGV T , λV [0,0,1]T )
(UGV T ,−λV [0,0,1]T )
(UGTV T , λV [0,0,1]T )
(UGTV T ,−λV [0,0,1]T )
• Based on the constraint that scene points are in front of both
cameras, a single pair of correspondences can be used to find the
correct solution
Department of Computer Engineering
University of California at Santa Cruz
Linear triangulation
Q
Q
After estimating the projection matrix M l = K [I ,0] and M r = K [ R,− RT ] ,
3D coordinates of feature points can be computed from their 2D
projections
pl ≅ M l Pl
Since
pr ≅ M r Pl
iT
if we denote ml and mriT as the rows of M l and M r , then
 xl ml3T − ml1T 

3T
2T 
y
m
−
m
l 
 l l
P = AP = 0
 xr mr3T − m1rT 

3T
2T 
y
m
−
m

 r r
r 
The solution is the eigenvector corresponding to the smallest eigenvalue
of of AT A
Department of Computer Engineering
University of California at Santa Cruz
Bundle Adjustment
Q
Gradient methods such as the Levenberg-Marquardt method is then
employed to refined the result by minimizing the “reprojection error”
2 M
min ∑ ∑ pij − pˆ ij
ˆ
2
R ,T, Pj =1,... M i =1 j =1
pˆ 2 j
pˆ1 j
p1 j
p2 j