What Is Multiview Geometry? A Beginner’s Guide

Posted on

Web Design

Posted at

Beginners and technical readers (developers, ML engineers, robotics students) searching to understand what multiview geometry is, how it works, and how its core building blocks — epipolar geometry, the fundamental/essential matrices, homography, triangulation, and structure from motion — fit together.

A single photograph tells you a lot about what's in front of a camera, but it hides one crucial thing: depth. Look at a photo of a coffee cup on a table, and you can't tell exactly how far the cup is from the camera, or how big it really is, just from that one image. The 2D picture is a flattened shadow of a 3D world.

Now take a second photograph of the same cup from a slightly different angle. Suddenly something interesting happens. The cup shifts position in the frame relative to the background. That shift — small as it looks — encodes information about depth, about how far the camera moved, and about the true 3D shape of the scene.

This relationship between multiple images of the same scene, and what it reveals about 3D structure and camera motion, is the subject of multiview geometry. It's one of the mathematical foundations underneath modern computer vision, robotics, augmented reality, and 3D reconstruction.

Multiview geometry is the branch of computer vision that studies the mathematical relationships between two or more images of the same scene taken from different camera viewpoints. By analyzing how the same physical points appear in different images, multiview geometry lets a computer recover information that no single image can provide on its own: the 3D positions of points in the scene, the relative position and orientation of the cameras, and the overall structure of the environment. It underlies technologies such as stereo vision, 3D reconstruction, visual SLAM, and photogrammetry.

What Is Multiview Geometry?

At its core, multiview geometry is about exploiting correspondence. If you can identify the same physical point — say, the rim of the coffee cup — in two or more images, and you know something about how those images were captured, you can work backward to figure out where that point actually sits in 3D space.

A single camera performs a projection: it takes a 3D world and compresses it onto a flat 2D sensor. In doing so, it throws away depth information — many different 3D points, at different distances along the same line of sight, would produce the exact same pixel in the image. This is the fundamental ambiguity of monocular (single-image) vision.

Multiview geometry resolves that ambiguity by adding a second (or third, or hundredth) viewpoint. If a point is visible in two images taken from different positions, its 3D location is constrained by both viewing rays simultaneously — and it must lie where those rays intersect. This idea, extended and formalized with rigorous mathematics, is what gives multiview geometry its power.

Why Is Multiview Geometry Important?

Multiview geometry isn't just an academic exercise — it's the mathematical backbone of many technologies people use every day, even if they never see the underlying equations.

  • 3D reconstruction: Turning a set of photographs into a 3D model, used in mapping, heritage preservation, and game/film asset creation.

  • Stereo vision: Two cameras working together to estimate depth in real time, used in robotics and some autonomous vehicle sensor suites.

  • Robotics: Robots use multiview geometry to localize themselves and understand the 3D layout of a workspace.

  • Autonomous vehicles: Multiple cameras (and sometimes other sensors) combine to build a 3D understanding of the road environment.

  • Augmented and virtual reality: Head-mounted and phone cameras track their own motion through space using techniques rooted in multiview geometry.

  • Photogrammetry and drone mapping: Aerial or handheld photo sets are turned into accurate terrain and building models.

  • Visual SLAM (Simultaneous Localization and Mapping): Robots and AR devices build a map of an unknown environment while tracking their own position within it.

  • Camera calibration and industrial inspection: Manufacturing lines use multi-camera setups for precise 3D measurement of parts.

  • Digital twins: Physical spaces are captured and represented digitally using image-based 3D reconstruction.

In each case, the same underlying question is being asked: given several 2D views of a scene, what can we infer about the 3D world that produced them?

Single View vs. Multiple Views

Feature

Single Image

Multiple Images

2D appearance

Yes

Yes

Depth estimation

Limited (requires learned priors or cues)

Stronger (geometric constraints)

Camera motion recovery

Difficult

Possible

3D reconstruction

Limited

Possible

Stereo information

No

Yes

Scene geometry understanding

Limited

Richer

A single image can still hint at depth through cues like shading, occlusion, or relative object size, and modern learning-based methods can estimate rough depth from one photo. But multiview geometry provides geometric constraints — grounded in the physics of how cameras project light — rather than statistical guesses.

The Basic Idea Behind Multiview Geometry

The core pipeline looks like this conceptually:

3D Point → Camera Projection → 2D Image

A point in the real world gets projected through a camera's lens onto its sensor, producing a pixel. Do this with multiple cameras (or one moving camera), and you get:

Multiple 2D Observations → Geometric Constraints → 3D Information

Each image alone is ambiguous about depth. But when you combine observations from different viewpoints and apply the geometric rules that describe how cameras project the world, the ambiguity collapses, and 3D structure becomes recoverable.

The Pinhole Camera Model

Before multiview geometry makes sense, it helps to understand how a single camera works. The simplest and most widely used model is the pinhole camera model.

Imagine a box with a tiny hole on one side. Light from the scene passes through that hole and lands on the opposite wall, forming an (upside-down) image. Real lenses are more complex, but the pinhole model captures the essential geometry: straight lines from 3D points, through a single center of projection, onto an image plane.

The pinhole model is usually described with a projection equation:

x ≈ K [R | t] X

Here's what each symbol means:

  • X — a 3D point in the world, expressed in homogeneous coordinates.

  • R — the camera's rotation (its orientation in space).

  • t — the camera's translation (its position in space).

  • [R | t] — together, these describe the camera's extrinsic parameters, i.e., where the camera is and which way it's facing. This maps a world point into the camera's own coordinate frame.

  • K — the intrinsic matrix, describing the camera's internal properties: focal length, principal point, and pixel scaling. This maps a point in the camera's coordinate frame onto the 2D image plane.

  • x — the resulting 2D image point (in homogeneous pixel coordinates).

Note that different textbooks use slightly different notations and conventions (row vs. column vectors, different sign conventions for translation), so it's worth checking which convention a given source uses before mixing formulas from different places.

Camera Intrinsics vs. Extrinsics

It helps to clearly separate two categories of camera parameters.

Intrinsic parameters describe properties of the camera itself, independent of where it is in the world:

  • Focal length (how strongly the lens bends light, effectively controlling field of view)

  • Principal point (where the optical axis meets the image sensor)

  • Pixel scaling and skew (how physical units map to pixel units)

Extrinsic parameters describe the camera's position and orientation in the world:

  • Rotation (which direction the camera is facing)

  • Translation (where the camera's center is located)

Parameter Type

Describes

Examples

Intrinsic

The camera itself

Focal length, principal point, pixel scale

Extrinsic

The camera's placement in the world

Rotation, translation, pose

Multiview geometry problems typically involve recovering one or both of these — sometimes the intrinsics are known in advance from calibration, and the goal is to recover extrinsics (pose) and 3D structure.

Coordinate Systems in Multiview Geometry

A 3D point passes through several coordinate systems on its way to becoming a pixel:

  1. World coordinates — a fixed reference frame for the whole scene.

  2. Camera coordinates — the scene as seen from a specific camera's position and orientation (obtained by applying extrinsics).

  3. Image coordinates — the 2D projection onto the image plane, often in normalized units.

  4. Pixel coordinates — the final discrete grid of pixels, after applying the intrinsic matrix.

Understanding this chain matters because different multiview geometry tools operate at different stages: the essential matrix works with normalized camera coordinates, while the fundamental matrix works directly with pixel coordinates.

Correspondence: Finding the Same Point in Multiple Images

Before any geometric reasoning can happen, a computer needs to answer a deceptively hard question: which point in image A corresponds to which point in image B?

This is solved through feature matching:

  • Keypoints — distinctive, easily re-identifiable locations in an image (corners, blobs, textured regions).

  • Descriptors — numerical fingerprints that summarize the local appearance around a keypoint, designed to be robust to changes in viewpoint, lighting, or scale.

  • Feature correspondences — pairs of keypoints (one from each image) that are judged to represent the same physical point, based on how similar their descriptors are.

Classic algorithms like SIFT (Scale-Invariant Feature Transform) and ORB (Oriented FAST and Rotated BRIEF) are widely used for this step, and OpenCV implements several of them.

Correspondence is a critical and fragile step. If the matching algorithm links the wrong points together — a false correspondence — every downstream geometric calculation built on that match will be corrupted. This is one reason robust estimation techniques like RANSAC (discussed later) are essential in practice.

Epipolar Geometry

Epipolar geometry describes the geometric relationship between two camera views of the same scene, and it's one of the most important ideas in multiview geometry.

Picture two cameras looking at the same 3D point. Each camera has a center of projection, and each observes the point as a 2D image location. Connect the two camera centers and the 3D point, and you get a triangle in space — this is called the epipolar plane.

The key insight: once you know a point's location in one image, you don't need to search the entire second image to find its match. Its corresponding point must lie somewhere along a specific line in the second image — the epipolar line — which is where the epipolar plane intersects the second image plane.

This constraint dramatically simplifies the correspondence problem: instead of a 2D search, it becomes a 1D search along a line.

What Is an Epipolar Line?

The epipolar line in one image is the projection of the other camera's viewing ray into that image. Because a 3D point could lie anywhere along the first camera's viewing ray, all of those possible positions project onto a single straight line in the second image. Any valid match for the point must fall on that line.

What Is an Epipole?

The epipole is the point in one image where the other camera's center projects to. Practically, it's the vanishing point of the baseline (the line connecting the two camera centers) as seen in each image. Every epipolar line in an image passes through that image's epipole, which is why epipolar lines form a converging "fan" pattern.

Fundamental Matrix

The fundamental matrix (F) captures the epipolar geometry between two uncalibrated cameras — meaning it works directly with raw pixel coordinates, without requiring knowledge of the cameras' internal parameters.

Its defining relationship is:

x′ᵀ F x = 0

Here, x and x′ are corresponding points (in homogeneous pixel coordinates) in the first and second images, and F is a 3×3 matrix. This equation is the algebraic version of the epipolar constraint: for any true correspondence, the equation must hold exactly.

Key properties of F:

  • It's a 3×3 matrix, but it has only 7 degrees of freedom due to a rank-2 constraint (its determinant is zero).

  • Given F and a point in one image, multiplying gives the equation of its epipolar line in the other image.

  • F can be estimated directly from a set of point correspondences between two images (commonly using at least 7–8 matched points, combined with a robust estimator to reject outliers).

Essential Matrix

The essential matrix (E) is closely related to F, but it applies to calibrated cameras — that is, cameras whose intrinsic parameters (like focal length) are already known. It works with normalized image coordinates rather than raw pixels.

It's defined as:

E = [t]ₓ R

  • R is the rotation between the two camera views.

  • t is the translation between the two camera centers.

  • [t]ₓ denotes the skew-symmetric matrix form of t, which turns the translation vector into a matrix so it can represent a cross product.

Because E directly encodes rotation and translation, decomposing it (once estimated) recovers the relative pose between the two cameras — this is one of the most useful outputs in multiview geometry, since it tells you how the camera moved between shots.

Concept

Fundamental Matrix

Essential Matrix

Requires calibration

No (works with raw pixels)

Yes (works with normalized coordinates)

Encodes

General epipolar geometry

Relative rotation and translation

Degrees of freedom

7

5

Typical use

Uncalibrated image pairs

Calibrated cameras, pose recovery




Fundamental Matrix vs. Essential Matrix — When to Use Which

A simple way to think about it: if you don't know your camera's intrinsic parameters, or you're working with images from an unknown source, you're in fundamental-matrix territory. If your camera has been calibrated and you want to recover actual camera motion (rotation and translation) rather than just epipolar lines, you want the essential matrix. In practice, many pipelines estimate F first, then convert to E once intrinsics are available.

Homography

A homography is a mapping that relates corresponding points between two views of the same planar surface, or between two views taken from the same camera center with only rotation (no translation).

x′ ≈ H x

Here, H is a 3×3 matrix that directly maps a point in one image to its corresponding point in the other — no depth information required, because the underlying assumption (a flat surface, or pure rotation) removes the depth ambiguity.

Homographies are used in:

  • Document scanning and perspective correction (mapping a skewed photo of a page to a flat rectangle)

  • Panorama stitching (aligning images taken by rotating a camera in place)

  • Augmented reality (overlaying content on a tracked planar surface)

  • General image alignment tasks

Homography vs. Fundamental Matrix

Feature

Homography

Fundamental Matrix

Main purpose

Map points directly between views

Describe epipolar geometry between views

Typical scenario

Planar scene, or pure camera rotation

General 3D scene with camera translation

Matrix size

3×3

3×3

Requires planar assumption?

Usually

No

Main use

Image warping and alignment

Two-view geometric analysis

Applying the wrong model matters: if you try to fit a homography to a scene with significant 3D depth variation and camera translation, the mapping will be inaccurate for points that don't lie near the assumed plane.

Triangulation

Triangulation is the process of estimating a 3D point's location from its 2D projections in two or more images.

Think of it this way: camera A sees a point somewhere along a particular ray extending out from its center. Camera B sees the same point along a different ray from its own center. In an ideal world with no noise, these two rays intersect at exactly one point — the 3D location being sought.

In practice, due to measurement noise, feature-matching imprecision, and calibration error, the two rays usually don't intersect perfectly. So triangulation methods typically find the 3D point that minimizes the combined distance (or reprojection error) across both rays, rather than assuming a perfect intersection.

Relevant factors:

  • Baseline — the distance between the two camera centers.

  • Triangulation accuracy — generally improves with a larger baseline (up to a point) and more precise correspondences.

  • Numerical issues — triangulation becomes poorly conditioned when the two viewing rays are nearly parallel, which happens with a very small baseline or points very far from the cameras.

Why Camera Baseline Matters

A wider baseline between two cameras generally increases the "triangulation angle" for a given 3D point, which tends to make depth estimates more sensitive and accurate. But a wider baseline also has downsides: the two views look more different from each other, making feature matching harder, and it can introduce more occlusion (surfaces visible to one camera but hidden from the other). Choosing an appropriate baseline is a practical trade-off, not a "bigger is always better" decision.

Stereo Vision

Stereo vision is a specific, widely used application of multiview geometry involving two cameras (or two viewpoints) with a fixed, known relationship — mimicking how human binocular vision estimates depth.

The typical stereo pipeline:

  1. Capture a left image and a right image simultaneously (or from a calibrated stereo rig).

  2. Find corresponding points between the two images.

  3. Measure how far each point has shifted between the images — this shift is called disparity.

  4. Convert disparity into depth using the camera geometry.

Disparity and Depth

Objects closer to the camera shift more between the left and right images (larger disparity); distant objects shift less (smaller disparity). This relationship can be expressed with a simplified formula for a rectified stereo pair:

Z = fB / d

  • Z — depth (distance from the camera to the point)

  • f — focal length

  • B — baseline (distance between the two camera centers)

  • d — disparity (how much the point shifts between images, in the same units as focal length)

This formula assumes the two cameras are calibrated, aligned so their image planes are parallel (a process called rectification), and that correspondences are accurate — it's a useful approximation, not a universal law for arbitrary camera setups.

Structure from Motion

Structure from Motion (SfM) generalizes the two-view stereo idea to many images, often captured by a single moving camera (rather than a fixed stereo rig) with unknown motion between shots.

A typical SfM pipeline:

  1. Detect and match features across many images.

  2. Estimate relative camera poses between overlapping image pairs.

  3. Triangulate 3D points from these poses and correspondences.

  4. Incrementally (or globally) combine information across all images to build a consistent set of camera poses and 3D points.

  5. Refine everything jointly using bundle adjustment.

Aspect

Structure from Motion

Stereo Vision

Camera setup

Often a single moving camera, unknown poses

Typically fixed, calibrated stereo rig

Number of images

Many (dozens to thousands)

Usually two (a stereo pair)

Poses

Estimated as part of the process

Usually known in advance

Typical output

Sparse or dense 3D reconstruction + camera trajectory

Depth map

Bundle Adjustment

Bundle adjustment is a joint optimization step that refines both camera poses and 3D point positions simultaneously, in order to minimize reprojection error across the whole reconstruction.

Reprojection Error

Once you have an estimated 3D point and a set of camera poses, you can project that 3D point back into each image using the camera model. The difference between where the point actually was observed and where the model predicts it should appear is the reprojection error. Bundle adjustment adjusts camera poses and 3D points together to minimize this error across all points and all images, producing a globally consistent and more accurate reconstruction.

Camera Calibration and Lens Distortion

Multiview geometry assumes the pinhole model is a reasonably accurate description of how the camera behaves. Real lenses depart from this ideal, so camera calibration is often a necessary preprocessing step.

Calibration typically involves:

  • Estimating intrinsic parameters (focal length, principal point)

  • Estimating lens distortion coefficients

  • Sometimes estimating extrinsics relative to a known calibration target (like a checkerboard pattern)

Lens distortion commonly appears as:

  • Radial distortion — causes straight lines near the edges of the image to bow outward (barrel distortion) or inward (pincushion distortion).

  • Tangential distortion — arises from slight misalignment between the lens and the image sensor.

Uncorrected distortion introduces systematic errors into correspondence and triangulation, so it's typically corrected before precise geometric computation.

Pose Estimation and PnP

Perspective-n-Point (PnP) solves a different but related problem: given a set of known 3D points and their corresponding 2D image locations (plus known camera intrinsics), estimate the camera's pose (rotation and translation) that produced those observations.

This differs from two-view triangulation, which recovers unknown 3D points from known camera poses. PnP instead recovers an unknown camera pose from known 3D points.

PnP is widely used in:

  • Robotics (localizing a robot relative to known landmarks)

  • Augmented reality (tracking a device's pose relative to a known marker or map)

  • Visual navigation and camera tracking in film/VFX

RANSAC in Multiview Geometry

Real-world feature matching is never perfect — some correspondences will be wrong. RANSAC (Random Sample Consensus) is a robust estimation technique commonly used to handle this.

The basic idea: repeatedly select a small random subset of correspondences, fit a geometric model (like a fundamental matrix or homography) to that subset, and check how many of the remaining correspondences agree with the resulting model within some tolerance. The model supported by the largest consistent set of matches ("inliers") is kept, and outlier matches are discarded.

For example, if 100 feature matches exist between two images but 15 of them are mismatches, RANSAC can help identify a geometric model supported by the 85 consistent matches, effectively filtering out the bad ones.

Common Algorithms and Tools

Several tools implement multiview geometry algorithms in practice:

  • OpenCV — a widely used open-source computer vision library with functions for feature detection, matching, fundamental/essential matrix estimation, homography estimation, pose recovery, triangulation, and camera calibration.

  • COLMAP — a general-purpose Structure-from-Motion and Multi-View Stereo pipeline, popular in research and 3D reconstruction workflows.

  • OpenMVG ("Open Multiple View Geometry") — an open-source library focused specifically on multiview geometry and SfM.

  • OpenMVS — a companion library to OpenMVG focused on dense multi-view stereo reconstruction from sparse SfM output.

  • MATLAB Computer Vision Toolbox — offers built-in functions for stereo calibration, disparity computation, and 3D reconstruction.

Real-World Example: Reconstructing a 3D Object

Here's how the concepts above combine in a practical pipeline — reconstructing a 3D model of a statue from photographs:

  1. Capture images from multiple viewpoints around the object, with good overlap between neighboring shots.

  2. Detect features in each image using a method like SIFT or ORB.

  3. Match features across image pairs to establish correspondences.

  4. Estimate camera relationships — computing fundamental/essential matrices between image pairs, then recovering relative camera poses.

  5. Triangulate 3D points from matched features and estimated poses, producing a sparse point cloud.

  6. Refine the reconstruction with bundle adjustment, jointly optimizing camera poses and 3D points to minimize reprojection error.

  7. Generate a denser model using multi-view stereo techniques, filling in detail beyond the sparse point cloud.

At every step, multiview geometry supplies the mathematical rules that connect 2D observations back to a coherent 3D structure.

Real-World Applications

  • Robotics: Camera-based localization and 3D perception for manipulation and navigation.

  • Autonomous vehicles: Depth estimation and scene understanding from onboard cameras.

  • AR/VR: Real-time camera tracking and spatial mapping for overlaying digital content.

  • Photogrammetry: Deriving accurate measurements and 3D models from photographs, used in surveying and archaeology.

  • Drone mapping: Reconstructing terrain and structures from aerial imagery.

  • Manufacturing: 3D inspection and dimensional measurement of parts on production lines.

  • Film and VFX: Camera tracking to insert digital elements convincingly into live-action footage.

  • Digital twins: Building accurate 3D digital representations of real-world spaces.

  • Navigation: Visual localization and mapping for indoor and outdoor positioning.

Multiview Geometry vs. Computer Vision

Computer vision is the broad discipline concerned with extracting information from images and video. Multiview geometry is one of the mathematical toolkits within that field — specifically the one concerned with relationships between multiple viewpoints. Other areas of computer vision (like object recognition or image segmentation) may not rely on multiview geometry at all, while 3D-focused applications lean on it heavily.

Multiview Geometry vs. Photogrammetry

Multiview Geometry

Photogrammetry

A mathematical framework describing camera and scene relationships

A broader practice of using images to derive measurements and models

Focuses on geometric relationships between views

Applies those (and other) principles to produce real-world measurements

Realized through algorithms (F/E matrices, triangulation, SfM, etc.)

Used in surveying, mapping, cultural heritage, construction, and more

The two overlap significantly — photogrammetry as a practice depends heavily on multiview geometry as its mathematical foundation, but photogrammetry also includes practical concerns like accuracy standards, ground control points, and domain-specific workflows.

Multiview Geometry vs. Stereo Vision

Stereo vision is best understood as a specific, well-constrained application of multiview geometry — typically two cameras with a fixed, known relationship, optimized for fast depth estimation. Multiview geometry is the more general theory, covering any number of views with known or unknown relationships, including single moving cameras used across many frames (as in Structure from Motion).

Challenges in Multiview Geometry

Real-world conditions introduce difficulties that the clean mathematical models don't capture on their own:

  • Featureless surfaces — blank walls or uniform textures give feature matchers little to work with.

  • Repetitive textures — patterns like tiled floors can cause incorrect correspondences.

  • Occlusion — parts of the scene visible in one image may be hidden in another.

  • Motion — moving objects violate the static-scene assumption behind most multiview methods.

  • Lighting changes — different exposure or illumination between shots can hurt feature matching.

  • Reflections and transparent objects — these don't behave like solid, opaque surfaces and can produce misleading correspondences.

  • Poor calibration or synchronization — inaccurate intrinsics or timing mismatches between cameras introduce systematic error.

  • Rolling shutter — some cameras capture rows of pixels at slightly different times, distorting geometry for moving cameras or fast motion.

  • Degenerate configurations — certain camera/scene arrangements make some quantities impossible or very hard to recover reliably.

Degenerate Cases

Certain configurations make multiview geometry problems poorly conditioned or ambiguous:

  • Pure camera rotation (no translation) — no baseline means no triangulation is possible; only homography-based relationships apply.

  • Planar scenes — when all visible points lie on (or very near) a single plane, the fundamental matrix estimation becomes unstable, and homography is often the more appropriate model.

  • Very small baseline — nearly parallel viewing rays make triangulated depth highly sensitive to noise.

Accuracy Factors

Reconstruction quality depends on many interacting factors:

  • Camera calibration accuracy

  • Image resolution

  • Feature detection and matching quality

  • Baseline and camera overlap

  • Number of contributing views

  • Scene texture and lighting conditions

  • Scene or camera motion during capture

  • Choice of algorithms and optimization settings

Beginner Roadmap to Learn Multiview Geometry

Stage 1 — Basic Geometry: points, lines, triangles, and coordinate systems.

Stage 2 — Linear Algebra: vectors, matrices, matrix multiplication, and eigenvalues/eigenvectors — the mathematical language multiview geometry is written in.

Stage 3 — Camera Models: the pinhole camera, projection, intrinsics, and extrinsics.

Stage 4 — Feature Matching: keypoints, descriptors, and establishing correspondences.

Stage 5 — Two-View Geometry: epipolar geometry, the fundamental matrix, the essential matrix, and homography.

Stage 6 — 3D Reconstruction: triangulation, pose estimation, and bundle adjustment.

Stage 7 — Advanced Topics: Structure from Motion, visual SLAM, dense reconstruction, and newer neural-rendering-based 3D vision techniques.

Each stage builds on the last — trying to jump straight to Structure from Motion without understanding the pinhole camera model and correspondences tends to make the mathematics feel arbitrary rather than intuitive.

Common Beginner Mistakes

  • Jumping into formulas before understanding camera geometry — memorizing the epipolar constraint equation means little without a mental picture of camera centers and viewing rays.

  • Confusing intrinsic and extrinsic parameters — remember: intrinsics describe the camera itself, extrinsics describe where it is.

  • Confusing homography with the fundamental matrix — homography assumes a plane or pure rotation; the fundamental matrix handles general 3D scenes.

  • Treating feature matches as automatically correct — always assume some fraction of matches are wrong, and use robust estimation (RANSAC) accordingly.

  • Ignoring camera calibration and lens distortion — skipping this step silently degrades accuracy in ways that are hard to diagnose later.

  • Assuming more images always mean a better reconstruction — poor overlap, blur, or inconsistent lighting can hurt more than help.

  • Misunderstanding disparity — disparity is inversely related to depth, not directly proportional to it.

  • Confusing camera pose with object pose — PnP estimates camera pose relative to known 3D points, not the pose of a moving object.

Important Formulas to Know

Formula

Meaning

x ≈ K[R | t]X

Camera projection: maps a 3D world point to a 2D image point

x′ᵀFx = 0

Epipolar constraint using the fundamental matrix

E = [t]ₓR

Essential matrix in terms of relative rotation and translation

x′ ≈ Hx

Homography mapping between two views of a plane

Z = fB/d

Simplified stereo depth from disparity (rectified, calibrated cameras)

Quick Reference Table

Concept

What It Does

Why It Matters

Camera Matrix

Projects 3D points into 2D

The basic model underlying all camera geometry

Homography

Maps points directly between two views of a plane

Enables alignment, stitching, and AR overlays

Fundamental Matrix

Relates corresponding points across uncalibrated views

Encodes epipolar geometry

Essential Matrix

Encodes relative camera rotation and translation

Recovers calibrated relative pose

Triangulation

Recovers 3D points from 2D observations

Core of depth and 3D reconstruction

PnP

Estimates camera pose from known 3D-2D correspondences

Localization and tracking

RANSAC

Filters out incorrect correspondences

Makes estimation robust to outliers

Bundle Adjustment

Jointly refines poses and 3D points

Improves overall reconstruction accuracy

FAQ

1. What is multiview geometry? It's the study of the mathematical relationships between multiple images of the same scene, used to recover 3D structure and camera motion.

2. Why is multiview geometry important in computer vision? It underlies 3D reconstruction, stereo depth estimation, robotic localization, AR/VR tracking, and many other applications that need to understand 3D structure from 2D images.

3. What is epipolar geometry? The geometric relationship between two camera views of the same scene, which constrains where a matching point in one image can appear in the other — along a line, not anywhere in the image.

4. What is an epipolar line? The line in one image along which a corresponding point for a given point in the other image must lie.

5. What is an epipole? The point in one image where the other camera's center projects to; all epipolar lines in that image pass through it.

6. What is the fundamental matrix? A 3×3 matrix that encodes the epipolar geometry between two uncalibrated camera views, working directly with pixel coordinates.

7. What is the essential matrix? A 3×3 matrix, related to the fundamental matrix, that encodes the relative rotation and translation between two calibrated camera views.

8. What's the difference between the fundamental and essential matrices? The fundamental matrix works with uncalibrated, raw pixel coordinates; the essential matrix requires known camera intrinsics and works with normalized coordinates, directly encoding relative pose.

9. What is triangulation in computer vision? Estimating a 3D point's location from its 2D projections in two or more images, based on where the corresponding viewing rays intersect (or nearly intersect).

10. How does stereo vision estimate depth? By measuring how much a point shifts between a left and right image (disparity) and converting that shift into depth using the camera's baseline and focal length.

11. What is disparity? The horizontal shift in a point's position between two stereo images; it decreases as depth increases.

12. What is homography? A mapping that relates corresponding points between two views of a planar surface, or between two views from a camera that only rotated.

13. What is Structure from Motion? A technique for reconstructing 3D structure and camera poses from many images, often captured by a single moving camera with unknown motion.

14. What is bundle adjustment? A joint optimization step that refines camera poses and 3D points together to minimize reprojection error across a reconstruction.

15. What is PnP? Perspective-n-Point: a method for estimating a camera's pose given known 3D points and their corresponding 2D image locations.

16. Why is camera calibration important? Because real lenses depart from the ideal pinhole model; calibration corrects for intrinsic parameters and lens distortion, which improves the accuracy of all downstream geometric calculations.

17. What's the difference between multiview geometry and photogrammetry? Multiview geometry is the underlying mathematical framework; photogrammetry is the broader applied practice of deriving real-world measurements and models from images, built largely on multiview geometry.

18. Is multiview geometry difficult to learn? It requires comfort with linear algebra and some geometric intuition, but the core ideas — projection, correspondence, and triangulation — are conceptually approachable once built up step by step.

19. What mathematics is required to learn multiview geometry? Primarily linear algebra (vectors, matrices, matrix operations) and basic geometry; some areas also use elements of projective geometry and numerical optimization.

20. Which tools can I use to practice multiview geometry? OpenCV is the most accessible starting point, with COLMAP, OpenMVG, and OpenMVS useful for more complete Structure-from-Motion and multi-view stereo pipelines.


Create a free website with Framer, the website builder loved by startups, designers and agencies.