ORB-SLAM: Every Version, Explained and Run

6.1Why ORB-SLAM Earns Its Own Chapter

A fair question, since Chapter 4 already spent a full section on ORB-SLAM3. The answer is the same reason RTAB-Map got Chapter 5 instead of a paragraph in Chapter 3: a taxonomy chapter tells you where a system sits among its peers, and a deep-dive chapter tells you how to actually run the thing, argue about it in an interview down to the mechanism, and pick the right release for the job in front of you. ORB-SLAM has earned that treatment more than almost any system in the field. Three papers, one continuously maintained lineage, 2015 to 2021, and a decade later it is still the answer to "what would you actually ship on a CPU."

The lineage also rewards close reading because each release solved one specific, nameable problem with the previous one: ORB-SLAM (2015) proved a single feature-based pipeline could do tracking, mapping, relocalization, and loop closing in real time on a laptop CPU, monocular only. ORB-SLAM2 (2017) generalized the same core to stereo and RGB-D without touching the ideas that worked. ORB-SLAM3 (2021) fixed the two things that were left: it fused an IMU properly, and it stopped dying forever when tracking was lost. Read version-by-version, the story is a masterclass in disciplined engineering rather than a pile of unrelated tricks.

6.2Anatomy: The Threads Every Version Shares

Every ORB-SLAM release is organized as a small number of threads running at different rates, an architecture PTAM introduced in Chapter 4 and ORB-SLAM inherited wholesale, then extended.

Tracking, Every Frame

Runs at camera rate and must never miss a deadline. Extracts ORB features from the incoming frame, gets an initial pose guess (a constant-velocity motion model, or full relocalization if the previous frame was lost), then projects the local map into the image and refines the pose by minimizing reprojection error over just this one frame's pose, the map held fixed. This is motion-only bundle adjustment, the cheapest of the three BA flavors this chapter covers. Tracking's last job each frame is deciding whether the frame deserves to become a keyframe: enough time or motion has passed, or the map points visible have dropped enough that this view of the world is not yet on record.

Local Mapping, Per Keyframe

Runs whenever a new keyframe arrives, at a fraction of camera rate, and does the work tracking cannot afford: insert the keyframe into the covisibility graph, triangulate new map points against covisible neighbors, cull the map points and keyframes that stopped earning their keep (below), and run local bundle adjustment: joint optimization of a window of nearby keyframes and every point they see, with the boundary keyframes held fixed as an anchor. This is where map quality is actually made.

Loop Closing and Map Merging, Occasional

Runs whenever a keyframe's bag-of-words signature matches a place the system has seen before, rarely, and expensively. Verifies the match geometrically, computes a relative transform between now and then, instantly propagates a correction through a sparse pose graph (the essential graph, section 6.4), then launches a full bundle adjustment in the background to polish the whole map without blocking tracking or mapping. In ORB-SLAM3 this thread does one more job with no equivalent in v1 or v2: merging two previously separate maps when the robot recognizes a place across them, the mechanism that makes the Atlas (section 6.5) worth having.

6.3The Two Load-Bearing Ideas

ORB Features, Chosen for Speed and Rotation Invariance

ORB (Rublee et al., ICCV 2011), Oriented FAST and Rotated BRIEF, is the reason the whole lineage runs on a CPU at all. FAST corners are detected across an 8-level image pyramid so scale changes don't lose the feature, and the octree-based distribution ORB-SLAM adds on top forces detections to spread across the image instead of clumping wherever texture happens to be densest, which matters directly for triangulation quality: bunched-up features triangulate poorly. Each corner gets an orientation from its intensity centroid, and BRIEF, a binary descriptor built from a fixed set of pairwise pixel-intensity comparisons, is computed after rotating that comparison pattern to match, which is what makes matching survive in-plane rotation that plain BRIEF cannot. Matching two ORB descriptors is a Hamming distance, a hardware popcount instruction, which is why extraction and matching together cost single-digit milliseconds per frame on a laptop core.

DBoW2, Recognizing a Place You've Seen Before

DBoW2 (Gálvez-López & Tardós, T-RO 2012) quantizes each keyframe's ORB descriptors against a vocabulary tree pre-trained offline on a large generic image corpus, turning an image into a sparse bag-of-words vector, then scores candidates with a tf-idf-style similarity. Chapter 5 walks through the same idea in RTAB-Map, so if you read that chapter first, the mechanism is already familiar; the difference worth naming is that DBoW2's vocabulary is fixed at build time, which loads in a couple of seconds and never adapts to a new environment, versus RTAB-Map's vocabulary, which grows online as the robot explores. Fixed and offline buys DBoW2 speed and predictable memory; online and incremental buys RTAB-Map robustness to environments the vocabulary never saw. Neither choice is wrong, they are answers to different design priorities.

A single high bag-of-words score is never enough to accept a loop closure on its own. Every version geometrically verifies the candidate first: match individual ORB descriptors between the two keyframes, solve for a relative pose with RANSAC, and only accept the loop if enough inliers agree. Chapter 5's "one lucky match against a lookalike corridor gets diluted before it can do damage" applies here in spirit, though the exact mechanism differs: RTAB-Map dilutes bad matches with a Bayesian filter over consecutive frames, ORB-SLAM dilutes them with geometric verification plus, in v3, a requirement that consistency hold across several consecutive keyframes before a candidate is trusted.

ORB

Oriented FAST and Rotated BRIEF. A corner detector plus a binary descriptor, both fast enough to run on every frame with a CPU budget to spare.

Bag of words (DBoW2)

Quantize an image's features against a fixed vocabulary; the resulting histogram is compared against stored places to shortlist loop-closure candidates.

Motion-only BA

Optimize a single frame's camera pose against the existing map, points held fixed. Runs every frame, in Tracking.

Local BA

Optimize a window of covisible keyframes and every point they see, with the window's boundary keyframes fixed as an anchor. Runs per keyframe, in Local Mapping.

Full / Global BA

Optimize every keyframe and every point in the map at once. Expensive, triggered by an accepted loop closure or map merge, runs in its own background thread so tracking never stalls waiting for it.

6.4Covisibility Graph, Essential Graph, and Why Both Exist

The covisibility graph is a weighted undirected graph over keyframes: an edge connects two keyframes if they share at least 15 map points, weighted by how many they share. It answers "which keyframes are relevant right now" in constant time, which is exactly what Tracking needs to decide which map points to project, and what Local Mapping needs to decide the window for local BA. This is architecturally the same idea as Chapter 5's memory management, bounding cost by searching only a relevant neighborhood instead of the whole map, achieved through a graph rather than a memory hierarchy.

The essential graph is a much sparser subgraph built for one purpose: propagating a loop-closure correction through the whole trajectory in milliseconds. It keeps a spanning tree (the minimum connectivity needed, essentially keyframe creation order), every accepted loop-closure edge, and only the strongest covisibility edges, sharing 100 or more points rather than 15. Optimizing over the essential graph is a pose-graph optimization, poses only, no map points, which is orders of magnitude cheaper than a full bundle adjustment and is what makes the map visibly snap straight the instant a loop closes, while the expensive full BA that actually polishes point positions runs afterward, in the background, without anyone waiting on it.

For monocular sequences, the essential-graph and full-BA optimizations run in Sim(3), similarity transforms with 7 degrees of freedom (rotation, translation, and scale), because a monocular map's scale itself drifts over a long loop, not just its pose. Stereo, RGB-D, and any visual-inertial mode fix scale from the sensor, so their optimizations run in plain SE(3), 6 degrees of freedom, with no scale term to correct. This is a genuinely common interview trap: say "Sim3" for a stereo system and you have just implied its scale drifts, which it does not.

6.5Version by Version

ORB-SLAM (2015): The Original

ORB-SLAM (Mur-Artal, Montiel & Tardós, T-RO 2015) is monocular only, and its real contribution was proving that a single system could do tracking, local mapping, loop closing, and relocalization, all in real time on a CPU, at a time when the field was fragmented into specialists that did one of those well. Two mechanisms worth knowing by name: automatic map initialization computes both a homography and a fundamental matrix in parallel from the first two frames and picks whichever model better explains the observed motion (homography for near-planar scenes or pure rotation, fundamental matrix otherwise), which is how the system decides on its own when there's enough parallax to safely triangulate a starting map. And "survival of the fittest" culling discards map points that fail to be tracked in a minimum fraction of the frames that should have seen them, and discards keyframes whose points are almost entirely redundant with three or more neighbors, keeping the map lean without a human tuning a decay schedule. The repository (raulmur/ORB_SLAM) is explicitly archived by its own author in favor of ORB-SLAM2; read it for the ideas, build ORB-SLAM2 or 3 for anything you actually run.

ORB-SLAM2 (2017): One Codebase, Three Sensors

ORB-SLAM2 (Mur-Artal & Tardós, T-RO 2017) generalized the exact same tracking/mapping/loop-closing skeleton to stereo and RGB-D without touching the ideas that made v1 work; the difference is entirely in how map points get their initial depth. Monocular still triangulates from motion and needs the parallax check from v1. Stereo triangulates instantly from every frame's own left-right disparity, no motion required, which is why stereo ORB-SLAM2 has no monocular-style initialization dance and can produce a metric map from frame one. RGB-D takes depth straight from the sensor, skipping triangulation entirely for the points a depth pixel already covers. The other headline addition is a localization mode: load a previously saved map, disable local mapping and loop closing, and the system does nothing but track and relocalize against what is already there, the same map-once-reuse-daily idea Chapter 5 walks through for RTAB-Map, arriving at ORB-SLAM2 two years earlier.

ORB-SLAM3 (2021): The IMU and the Atlas

ORB-SLAM3 (Campos, Elvira, Rodríguez, Montiel & Tardós, T-RO 2021) closed the two gaps that remained after v2, and both are worth understanding at the mechanism level, not just the feature-list level.

Tightly coupled visual-inertial fusion. The IMU is fused as factors in the same bundle-adjustment graph as the visual reprojection errors, using IMU preintegration (Chapter 1), rather than bolted on as a separate filter. What actually earned the paper attention is the initialization: a four-stage procedure that gets an initial estimate of gyroscope bias, scale, gravity direction, accelerometer bias, and velocities in roughly two seconds of motion, refined immediately with a proper inertial-only bundle adjustment, versus the 10 to 30 seconds most contemporary visual-inertial systems needed. Chapter 4's VIO section covers why the observability problem exists in the first place; ORB-SLAM3's contribution was making the estimate usable almost immediately instead of waiting for it to slowly converge.

The Atlas. Every earlier version treated tracking loss as an emergency: freeze, enter relocalization, and hope. ORB-SLAM3 instead keeps a collection of maps, the Atlas, one active map that tracking currently writes into and any number of non-active stored maps. Lose tracking, and instead of stalling, the system simply starts a brand-new active map immediately and keeps going. Place recognition keeps running against every map in the Atlas, active or not, and the moment a keyframe in the new map matches a place from an old one, the two maps merge: a local welding bundle adjustment aligns them into one coordinate frame on the spot, and a full BA in the background cleans up the seam afterward. The practical result is a system that degrades to "start a new map" instead of "go blind," and heals the moment it recognizes familiar ground.

6.6The Wider Family: Forks, Clean-Room Rewrites, Dynamic-Scene Variants

ORB-SLAM's influence extends well past the three official releases, and knowing the family tree signals you've looked past the headline repo.

• stella_vslam (formerly OpenVSLAM, Sumikura, Shibuya & Sakurada, ACM Multimedia 2019): a clean-room reimplementation of the same feature-based, bag-of-words, covisibility-graph recipe, adding first-class support for fisheye and equirectangular (360°) cameras. OpenVSLAM was pulled from GitHub in 2020 after its maintainers cited patent concerns tied to the original ORB-SLAM patent held by the Universidad de Zaragoza; the community picked it back up under a new name and a clarified license, and stella_vslam is the one to clone today.
• DynaSLAM (Bescos, Fácil, Civera & Neira, RA-L 2018): built directly on ORB-SLAM2, adds a Mask R-CNN segmentation pass that detects and masks out dynamic objects (people, vehicles) before they ever reach feature matching, and can inpaint the resulting holes in the background using earlier frames. The system this chapter and Chapter 5 both build assumes a mostly static world; DynaSLAM is the concrete answer for when that assumption breaks.
• UcoSLAM (Muñoz-Salinas & Medina-Carnicer, Pattern Recognition 2020): fuses ORB-style natural features with square fiducial markers (ArUco) in the same map, so a handful of printed markers in a warehouse or lab dramatically speed up initialization and relocalization, at the cost of needing to place markers at all.
• Community forks of ORB-SLAM3 add pieces the official repo doesn't ship: Python bindings for scripting and dataset experiments, and integrations with newer feature front-ends like SuperPoint in place of ORB. None are official, so pin a commit and read the fork's own test results before trusting it in anything that matters.

6.7Choosing a Version in 2026, and the License You're Actually Agreeing To

sudo apt install build-essential cmake libeigen3-dev libopencv-dev libglew-dev
git clone https://github.com/raulmur/ORB_SLAM2.git && cd ORB_SLAM2
chmod +x build.sh && ./build.sh   # builds Thirdparty/DBoW2, Thirdparty/g2o, then the main library

# dataset: TUM RGB-D fr1/desk, a short handheld desk-scanning sequence
wget https://cvg.cit.tum.de/rgbd/dataset/freiburg1/rgbd_dataset_freiburg1_desk.tgz
mkdir -p ~/tum/fr1_desk && tar -xzf rgbd_dataset_freiburg1_desk.tgz -C ~/tum/fr1_desk --strip-components=1

# associate color and depth frames by timestamp (TUM's own tool)
wget https://cvg.cit.tum.de/_media/data/datasets/rgbd-dataset/tools/associate.py
python3 associate.py ~/tum/fr1_desk/rgb.txt ~/tum/fr1_desk/depth.txt > ~/tum/fr1_desk/associations.txt

./Examples/RGB-D/rgbd_tum \
  Vocabulary/ORBvoc.txt \
  Examples/RGB-D/TUM1.yaml \
  ~/tum/fr1_desk \
  ~/tum/fr1_desk/associations.txt

git clone https://github.com/UZ-SLAMLab/ORB_SLAM3.git && cd ORB_SLAM3
chmod +x build.sh && ./build.sh

# dataset: TUM RGB-D fr3/long_office_household, the same sequence Chapter 5 used
# for a headless RTAB-Map run, so both chapters' results are directly comparable
wget https://cvg.cit.tum.de/rgbd/dataset/freiburg3/rgbd_dataset_freiburg3_long_office_household.tgz
mkdir -p ~/tum/fr3_office && tar -xzf rgbd_dataset_freiburg3_long_office_household.tgz -C ~/tum/fr3_office --strip-components=1
python3 associate.py ~/tum/fr3_office/rgb.txt ~/tum/fr3_office/depth.txt > ~/tum/fr3_office/associations.txt

System.SaveAtlasToFile: "office_session"

# first run: builds the map and writes office_session.osa on a clean shutdown
./Examples/RGB-D/rgbd_tum \
  Vocabulary/ORBvoc.txt \
  Examples/RGB-D/TUM3.yaml \
  ~/tum/fr3_office \
  ~/tum/fr3_office/associations.txt

6.8Field-Tested Gotchas

6.9Reading List, in Priority Order

6.10Interview Questions, Answer Out Loud First