Visual SLAM: The State of the Art

4.1Why Cameras Are the Hard Mode

A lidar hands you 3D geometry directly. A camera hands you a grid of brightness values, and geometry must be inferred by watching how pixels move between frames. Worse, a single camera fundamentally cannot tell a small room nearby from a big room far away. Everything is known only up to scale, the famous monocular scale ambiguity.

In exchange, cameras are ten times cheaper than lidar, tiny, power-sipping, and they see color and meaning. A camera can read an exit sign; a lidar sees a flat rectangle. The history of visual SLAM is the history of squeezing geometry out of pixels with progressively more machine learning.

Two more hardships that lidar people never think about, both covered in depth in Chapter 2: cameras care about light (auto-exposure swings break brightness assumptions, darkness ends the mission), and most cheap cameras have rolling shutter, so every row of the image was taken at a slightly different time. At walking speed this is a nuisance. On a fast-yawing drone it bends straight lines into curves and quietly poisons the geometry.

4.2The Six Eras, Your Master Taxonomy

4.3Eras 1 and 2, the Baselines Production Robots Still Run

Where It Started

MonoSLAM (Davison, 2003) put a single camera inside an EKF: a dozen landmarks, 30 Hz, on a laptop of that era. It proved real-time visual SLAM possible and immediately hit the filter's ceiling: cost grows with the square of landmark count, so the map stays tiny (Chapter 1 explains why).

PTAM (Klein & Murray, ISMAR 2007) contributed the architectural idea everything since has copied: split tracking (every frame, real-time, against the current map) from mapping (bundle adjustment over keyframes, in a background thread, as slow as it needs to be). Once mapping no longer had to finish per frame, maps could grow to thousands of points. Every modern system, learned ones included, still has this two-thread heartbeat.

The ORB-SLAM Lineage, the Interview Baseline

ORB-SLAM3 (Campos et al., 2021) is the reference feature-based system, and its anatomy is worth knowing part by part:

• ORB features: FAST corners with a binary descriptor. Not the best features ever designed, but computable at frame rate on a CPU, and matchable with hamming distance.
• Covisibility graph: keyframes are nodes, edges connect keyframes that see the same points. Tracking and local bundle adjustment only touch the neighborhood of the current keyframe, which is what keeps a city-block map real-time.
• DBoW2 place recognition: a bag-of-visual-words vocabulary turns each image into a sparse histogram; loop candidates come from histogram lookups, then survive geometric verification. Fast, memory-light, and blind to viewpoint change beyond about 30 degrees.
• Atlas multi-map: when tracking is lost, instead of dying the system spawns a fresh map and merges it with the old one when a place is re-recognized. Kidnap-resistant by construction.
• IMU fusion: tightly coupled visual-inertial mode using preintegration (Chapter 1), with a careful initialization dance: the IMU needs seconds of excited motion before scale and gravity are observable.

Weaknesses, stated plainly: hand-crafted ORB features die in motion blur and low texture, and the output map is a sparse point cloud, great for localizing and useless for navigating directly (pair it with Chapter 3's occupancy mapping or Chapter 5's RTAB-Map for that).

Direct Methods, the Contrarian School

LSD-SLAM (Engel et al., 2014) and DSO (Engel et al., 2016) ask: why extract features at all? Align images by minimizing the brightness difference of pixels directly. More of the image gets used, semi-dense maps fall out for free, and low-texture regions with gradients (a shaded wall) still contribute.

The price is a hard assumption: that a surface keeps its brightness between frames. Auto-exposure changes, vignetting, rolling shutter, and gain swings all violate it, which is why DSO demands photometric calibration (exposure times, response curve, vignette map) to perform at its best. Feature-based systems shrug at all of that. This trade is the classic "feature vs direct" interview question, and the honest answer is that era 3 made it mostly moot: learned features got robust enough that the direct school's advantages narrowed.

Reprojection error

"I triangulated this 3D point; when I project it back into the image, how far is it from where I actually detected it?" Era 1 minimizes this.

Photometric error

"If I warp frame A onto frame B using my pose and depth estimate, how different are the pixel brightnesses?" Era 2 minimizes this.

Keyframe

A frame promoted to long-term citizenship: it anchors map points and joins bundle adjustment. Everything between keyframes is tracked and forgotten.

Bundle adjustment (BA)

Joint least-squares refinement of camera poses and 3D points so points re-project where they were seen. The accuracy engine of eras 1 and 3, the embedded layer of era 4. Chapter 1 covers the solver.

4.4Visual-Inertial Odometry Deserves Its Own Section

If the question is "drone state estimation" or "AR headset tracking," the answer is not full SLAM. It is VIO: camera plus IMU, tightly fused, producing smooth 6-DOF odometry with no loop closure and little or no persistent map. Nearly every phone, headset, and drone autopilot runs VIO, which makes it the most deployed state estimator on Earth.

Why the IMU changes everything: the accelerometer measures metric acceleration, so scale becomes observable for a single camera. The gyro carries tracking through the 100 ms the camera is blurred. And the IMU pins roll and pitch to gravity, so only yaw and position drift. The price is the initialization dance (scale and biases need excited motion to become observable) and the calibration and sync burden of Chapter 2.

• MSCKF (Mourikis & Roumeliotis, 2007): the filter lineage. A sliding window of camera poses inside an EKF, features marginalized without ever entering the state. Cheap and elegant; the ancestor of most phone VIO.
• VINS-Mono / VINS-Fusion (Qin et al., 2018): the optimization lineage. Sliding-window bundle adjustment with IMU preintegration factors, loop closure optional. The default academic baseline.
• OpenVINS (Geneva et al., 2020): a clean, well-documented MSCKF that doubles as a teaching codebase. Its config file is where the Chapter 2 Allan-variance numbers go.
• OKVIS2 (Leutenegger, 2022): keyframe-based tightly coupled VIO with loop closure, the modern version of the 2015 original.

4.5Era 3, Learned Front-Ends: The Pragmatic Production Answer

The observation behind era 3: classical geometric back-ends are near-optimal, and what actually breaks in the field is appearance. Detecting and matching features under blur, weak texture, glare, and day/night change is where missions die. So replace exactly those parts with networks and keep the geometric guarantees:

• SuperPoint (DeTone et al., 2018): learned keypoint detector and descriptor. The training story is worth retelling: it teaches itself on synthetic corners, then bootstraps to real images by "homographic adaptation," no human labels anywhere. Alternatives with different trade-offs: DISK (differentiable, denser) and ALIKED (lighter, deformable).
• SuperGlue (Sarlin et al., CVPR 2020): matching as a learned assignment problem. A graph neural network with attention looks at both images' keypoints together before deciding matches, which buys huge robustness at real compute cost.
• LightGlue (Lindenberger et al., 2023): the deployable successor. It adapts compute to difficulty, exiting early on easy pairs, and runs several times faster than SuperGlue at equal accuracy. LoFTR and RoMa go further with dense, detector-free matching for the hardest pairs.
• Place recognition upgraded from bag-of-words to learned global descriptors: NetVLAD (2016), then MixVPR, SALAD, and MegaLoc (2025). These survive viewpoint and season changes that break DBoW2 (Chapter 7 goes deeper).
• hloc: the standard toolbox gluing all of the above into visual relocalization pipelines; the Chapter 7 worked setup builds one.

When does swapping learned parts into a classical stack actually pay? Three honest triggers: your robot works across lighting changes (warehouse day/night shifts), your environment is texture-poor (corridors, white walls), or relocalization against an old map keeps failing at more than about 30 degrees of viewpoint change. If ORB features track fine in your environment, era 3 buys you latency for nothing.

4.6Era 4, the DROID Lineage

DROID-SLAM (Teed & Deng, NeurIPS 2021) is one idea executed with unusual thoroughness. A recurrent network (a conv-GRU, borrowed from the RAFT optical-flow architecture) looks at pairs of frames and repeatedly revises a dense correspondence field, along with a confidence weight per pixel. After each revision, a differentiable bundle adjustment (DBA) layer updates all camera poses and per-pixel depths to be consistent with those correspondences. The DBA layer is an actual Gauss-Newton solver embedded inside the network, with gradients flowing through the solve during training. Predict, solve, repeat.

The division of labor is the insight: the network handles what networks are good at (recognizing which pixels correspond, judging which correspondences to trust), and the solver handles what solvers are good at (exact geometry). Trained almost entirely on the synthetic TartanAir dataset, it transfers to real footage embarrassingly well.

The result is eerily robust. It essentially never loses tracking where ORB-SLAM3 dies. The costs: it wants a serious GPU (11 GB class for full SLAM with loop closure), and run monocular it still drifts in scale like any camera-only system.

DPVO (Teed, Lipson & Deng, NeurIPS 2023) made the idea affordable: track sparse random patches instead of dense flow, and real-time visual odometry fits on one modest GPU. DPV-SLAM (2024) added loop closure and global optimization back on top.

The positioning sentence to remember: "DROID showed that end-to-end learning plus a differentiable geometric solver beats classical robustness; DPVO made it affordable."

git clone https://github.com/princeton-vl/DPVO.git && cd DPVO
conda env create -f environment.yml
conda activate dpvo
pip install .

# model weights + a EuRoC sequence
./download_models_and_data.sh          # fetches dpvo.pth and sample data
wget http://robotics.ethz.ch/~asl-datasets/ijrr_euroc_mav_dataset/machine_hall/MH_01_easy/MH_01_easy.zip
unzip MH_01_easy.zip -d euroc/MH_01

# run visual odometry on the left camera stream
python demo.py \
  --imagedir euroc/MH_01/mav0/cam0/data \
  --calib calib/euroc.txt \
  --stride 2 --viz

4.7Era 5, Neural-Rendering SLAM: The Map Is a Renderer

Radical inversion: make the map a differentiable rendering function. To track the camera, render what the map predicts you would see from a guessed pose, and optimize the pose (and the map) until the render matches the real image. Tracking and mapping become the same operation, gradient descent on image difference.

• The NeRF wave: iMAP (Sucar et al., 2021) put a single MLP map inside a live SLAM loop; NICE-SLAM (2022) swapped in hierarchical feature grids so bigger rooms fit; Co-SLAM, ESLAM, and Point-SLAM (2023) refined the representation. The shared bottleneck: NeRF rendering is slow, and SLAM needs to render constantly.
• The Gaussian splatting wave (2024): represent the scene as millions of soft 3D Gaussian blobs that rasterize fast, and suddenly real-time photorealistic RGB-D SLAM exists: SplaTAM (2024), MonoGS, "Gaussian Splatting SLAM" (CVPR 2024), Photo-SLAM, RTG-SLAM, and GS-ICP-SLAM (which couples ICP with the splats).

Why care: the output is a dense, photorealistic map you can re-render from any viewpoint. For inspection, telepresence, digital twins, and simulation assets, that is the product.

The honest critique to voice out loud (interviewers respect it): benchmarks are mostly room-scale RGB-D; drift handling, loop closure, and metric scale outdoors still lag geometric systems; and compute is heavy. As of early 2026 this is the reconstruction answer, not the navigation-stack answer. A robot that needs a costmap tonight should read Chapter 3.

4.8Era 6, Feed-Forward Geometry: The Frontier

Every previous era solves for geometry per scene. Era 6 asks: what if a transformer, trained on millions of scenes, just predicts the geometry in one forward pass?

• DUSt3R (Wang et al., CVPR 2024): feed two images into a transformer, uncalibrated, it does not even need the camera intrinsics. Out come dense pointmaps: a 3D point for every pixel, already expressed in a shared coordinate frame. No feature matching, no triangulation; the network has internalized geometry as a prior. MASt3R (ECCV 2024) adds metric scale and a matching head.
• MASt3R-SLAM (Murai et al., 2025): real-time SLAM built on those priors. Uncalibrated monocular video, in the wild, with a global point-cloud map.
• VGGT (Wang et al., CVPR 2025): one feed-forward transformer emits cameras, depths, and pointmaps for many views in seconds; VGGT-SLAM scales it with submap alignment. Streaming and causal variants: Spann3R, CUT3R, StreamVGGT. MegaSaM (2024-25) gets robust structure and motion from casual phone video.
• Supporting cast: monocular metric depth foundation models (Depth Anything v2 (2024), Metric3D v2, UniDepth) dropped into classical pipelines as priors that dissolve the scale ambiguity.

Where the claims need scrutiny: "no bundle adjustment" usually means per-pair geometry is feed-forward, while multi-view consistency still comes from an alignment optimization over pointmaps. And metric scale, when advertised, is learned from training data, so it is as good as the training distribution's resemblance to your scene. Chapter 7 returns to both caveats.

4.9Choosing a Visual System in 2026

4.10Benchmarks, in One Paragraph

The vocabulary you need here is two acronyms deep: ATE for global consistency, RPE for drift rate, both measured after trajectory alignment, on datasets like EuRoC, TUM RGB-D, KITTI, and TartanAir. That is deliberately shallow because Chapter 8 now owns evaluation end to end: the metrics' math, the evo toolchain, every dataset worth knowing, and how to benchmark your own robot without lying to yourself.

# dependencies
sudo apt install build-essential cmake libeigen3-dev libopencv-dev \
                 libglew-dev libpython3-dev python3-numpy

git clone --branch v0.6 https://github.com/stevenlovegrove/Pangolin.git
cd Pangolin && cmake -B build -DCMAKE_BUILD_TYPE=Release && sudo cmake --build build --target install

git clone https://github.com/UZ-SLAMLab/ORB_SLAM3.git && cd ORB_SLAM3
./build.sh        # if compile errors mention C++ standard, set C++14 in CMakeLists.txt

# dataset
wget http://robotics.ethz.ch/~asl-datasets/ijrr_euroc_mav_dataset/machine_hall/MH_01_easy/MH_01_easy.zip
mkdir -p ~/euroc/MH01 && unzip MH_01_easy.zip -d ~/euroc/MH01

./Examples/Stereo-Inertial/stereo_inertial_euroc \
  Vocabulary/ORBvoc.txt \
  Examples/Stereo-Inertial/EuRoC.yaml \
  ~/euroc/MH01 \
  Examples/Stereo-Inertial/EuRoC_TimeStamps/MH01.txt

4.11Reading List, in Priority Order

4.12Interview Questions, Answer Out Loud First