RTAB-Map + RealSense D435i

5.1Why This Exact Pairing Matters

Bolt a RealSense D435i onto anything with wheels, launch RTAB-Map, and out come a 2D occupancy grid any grid planner can consume directly, a colored 3D point cloud, and an OctoMap. The bill, at 2026 prices: the D435i lists at $334 and sells street for about $250 now that the D455 and D456 have taken over Intel's front page. Add a Raspberry Pi 5 (8 GB, about $80) or a Jetson Orin Nano Super (about $249) and the whole perception stack lands under $600. The software is free, apt-installable, and maintained by the same lab that wrote the papers.

That is why it is the standard indoor-UGV prototype rig, and why interviewers love it. Questions about this pairing test two things at once: whether you understand SLAM concepts, and whether you have actually fought real hardware. Plenty of candidates can define loop closure. Far fewer can explain why the map ghosts when the robot rotates, or why their depth image died when they swapped USB cables.

One piece of honesty up front. RTAB-Map does not win trajectory-accuracy benchmarks; ORB-SLAM3 beats it on ATE on most sequences (Chapter 8 covers how to measure that claim). RTAB-Map wins at something else: producing usable maps, staying up for hours, and surviving deployment. Those are different virtues, and for a product robot they are usually the ones that matter.

5.2RTAB-Map from Zero

RTAB-Map stands for Real-Time Appearance-Based Mapping, by Mathieu Labbé and François Michaud at the Université de Sherbrooke (ICRA 2011, IROS 2014, JFR 2019). Underneath, it is graph SLAM. Nodes hold RGB-D keyframes: a color image, a depth image, the visual words extracted from them, and the odometry pose where they were taken. Edges are constraints: odometry links between consecutive nodes, loop-closure links between recognized places, and proximity links between nodes that are near each other in space. Two ideas are its signature, and both are worth knowing cold.

Appearance-Based Loop Closure with a Bag of Words

"Appearance-based" means the robot recognizes places by how they look, not by matching geometry. The mechanism, step by step:

1. Extract local visual features from the current keyframe. The default detector is GFTT corners with BRIEF descriptors; ORB is a common alternative, and SIFT/SURF are available if you built with the nonfree modules.
2. Quantize each descriptor against a bag-of-visual-words vocabulary: a dictionary of typical feature appearances. The image becomes a histogram of which words it contains, the way you might describe a book by its word counts. Unlike ORB-SLAM's DBoW2, which ships a vocabulary pre-trained offline, RTAB-Map builds its vocabulary incrementally from the images it actually sees, adding new words when a descriptor matches nothing (nearest-neighbor search over a FLANN kd-tree).
3. Score the current histogram against stored locations with a tf-idf-style likelihood: words that appear everywhere carry little weight, rare words carry a lot.

Historical context makes the name make sense. Around 2008-2012, appearance-based place recognition was the hot answer to loop closure at scale; FAB-MAP had shown you could recognize places over a 1000 km trajectory from images alone. RTAB-Map's contribution was making that style of recognition run online, forever, on a robot's own CPU, and wrapping a complete mapping system around it rather than leaving place recognition as a party trick.

The Bayesian Filter over Loop Hypotheses

A single good histogram match is not a loop closure. RTAB-Map runs a discrete Bayesian filter whose state is "which past location, if any, am I looking at right now?" It maintains a probability for every location in Working Memory plus one extra hypothesis: this is a new place. Each image updates the posterior in two steps you will recognize from Chapter 1:

predict:  belief spreads to each hypothesis' neighbors in the graph
          (if I was probably at node 42, I am now probably near node 42)
update:   multiply by the BoW likelihood of the current image
          against each location, then normalize

Evidence therefore has to accumulate over consecutive frames; one lucky match against a lookalike corridor gets diluted before it can do damage. When some location's posterior clears Rtabmap/LoopThr (default 0.11), the candidate goes to geometric verification: match individual features between the two keyframes, solve the relative camera pose with PnP + RANSAC, and accept only if at least Vis/MinInliers features agree (default 20). Only then is a loop-closure edge added and the graph re-optimized.

Visual word

A cluster of similar feature descriptors treated as one symbol. "Corner of a door frame, lit from the left" becomes word #4127.

tf-idf

Term frequency times inverse document frequency. A scoring trick from text retrieval: a word you see in every image tells you nothing about where you are.

PnP (Perspective-n-Point)

Given n known 3D points and their pixel locations in an image, solve for the camera pose. The workhorse of visual relocalization.

RANSAC

Fit a model to random small subsets of the data, keep the fit that the most points agree with. Standard armor against wrong feature matches.

Keyframe

A frame selected for long-term storage. Keeping all 30 fps forever is pointless; consecutive frames are nearly identical.

STM, WM, LTM: The Memory That Keeps the CPU Bounded

A naive system compares each new frame against every past location, so cost grows with map size until real time dies. RTAB-Map's fix is shaped like human memory, and it is the reason the system exists:

• Short-Term Memory (STM). New nodes enter here first (Mem/STMSize, default 10 nodes). STM nodes are exempt from loop-closure detection, because you would trivially "recognize" the place you saw one second ago. STM is also where node weights are set: if consecutive images look alike, the robot is lingering, and the location's weight goes up.
• Working Memory (WM). Nodes graduate here from STM. WM is the set actively searched by the Bayesian filter. This is the memory that costs CPU.
• Long-Term Memory (LTM). When the map update exceeds the time budget Rtabmap/TimeThr (ms), or WM exceeds Rtabmap/MemoryThr (nodes), the oldest, lowest-weight locations are transferred out of WM into the SQLite database on disk. They are not deleted; they are just not searched.
• Retrieval. When a loop closure lands next to a region that was transferred out, its neighbors are pulled back from LTM into WM. Return to an old wing of the building, recognize one doorway, and the whole wing wakes up.

new frame
   └─► STM (last ~10 nodes; no loop detection here)
          └─► WM (actively searched by the Bayesian filter)
                 ├─ update time > Rtabmap/TimeThr?  transfer oldest,
                 │  lowest-weight nodes ──► LTM (sqlite, on disk)
                 └─ loop closure near an LTM region? retrieve its
                    neighbors back ◄── LTM

The consequence: loop-closure cost is O(WM size), which is bounded by your own budget, not by the size of the environment. That is how RTAB-Map does multi-session, building-scale mapping in real time on a CPU, and it is the answer to "why does RTAB-Map scale?"

Graph SLAM Underneath

Every accepted constraint feeds a pose graph: node poses as variables, odometry and loop-closure edges as measurements, solved by sparse nonlinear least squares exactly as in Chapter 1. RTAB-Map does not ship its own optimizer; it links one of TORO, g2o, GTSAM, or Ceres, selected by Optimizer/Strategy (0 through 3). Stock binaries default to GTSAM. Optimization runs when a loop or proximity link is accepted, and the corrected poses propagate into every map product: the 2D grid, the cloud, and the OctoMap are all re-rendered from keyframes at their new poses, which is why the map visibly "snaps" straight when a big loop closes.

5.3The Full Architecture on a Robot

RTAB-Map is a C++ library (rtabmap::Rtabmap and friends) with a Qt desktop app on top, and on a robot the system is three or four components wired by data flow, where most integration pain lives. The camera driver (librealsense2) produces color, aligned depth, and intrinsics. A pairing step bundles them into a single RGB-D frame by timestamp so the SLAM core never sees a color frame matched with the wrong depth frame. The rtabmap core consumes that plus an odometry stream.

Odometry, Three Honest Options

RTAB-Map is agnostic about where odometry comes from, and choosing is the first real engineering decision on this rig.

• Wheel odometry. Smooth, cheap, always publishes, immune to lighting. Drifts with wheel slip, dies on carpet transitions and thresholds, and knows nothing about z. On a slow indoor base with decent encoders it is often the most robust choice, with the camera reserved for loop closure and mapping.
• RTAB-Map's own RGB-D odometry (built into the library). No extra hardware. The default registration is F2M, frame-to-local-map: track features against a small local map of recent keyframes rather than only the previous frame (F2F). F2M beats F2F for the same reason scan-to-map beats scan-to-scan in Chapter 3: the reference accumulates evidence instead of resetting every frame. Weaknesses: blank walls, darkness, fast rotation, and it eats a CPU core.
• External VIO. VINS-Fusion, ORB-SLAM3 in odometry mode, or a vendor VIO module. Best behavior under fast or jerky motion because the IMU carries you through visual dropouts (Chapter 4 explains why). Costs integration effort and calibration care (Chapter 2).

Production robots frequently fuse wheel and visual odometry through an EKF (Chapter 1), so either source failing degrades accuracy instead of ending the run. RTAB-Map also adds proximity detection (RGBD/ProximityBySpace): it links nodes that odometry says are close in space even when appearance matching fails, which is what saves you when you re-traverse a corridor in the opposite direction and everything looks different.

What Comes Out

• A 2D occupancy grid, built from depth by the Grid/ parameters. This is what Chapter 9's planners eat.
• The graph itself, nodes and edges, exposed through the C++ API and stored in the database for debugging.
• The assembled 3D point cloud, re-rendered from keyframes after each optimization (export with rtabmap-export).
• An OctoMap if enabled, for 3D collision checking (Chapter 3 discusses when you want it).
• The transform map → odom, which is arguably the real product.

Who Owns Which Transform

The frame contract from Chapter 0 applies verbatim:

map ──► odom ──► base_link ──► camera_link
 │        │           │
 │        │           └─ static: your measured extrinsic
 │        └─ odometry source: fast, continuous, drifts
 └─ rtabmap: slow correction, absorbs loop-closure jumps

The odometry source owns odom → base_link, fast and continuous but drifting. RTAB-Map owns map → odom: a slow correction that absorbs loop-closure jumps, so the pose your controller consumes never teleports mid-motion. base_link → camera_link is your measured extrinsic from Chapter 2, and getting it wrong produces the ghost-map symptom in section 5.7. Chapter 9 builds navigation on exactly this contract.

5.4The D435i, Hardware Facts to Get Exactly Right

Interviewers check these details because they separate people who read the box from people who shipped the camera.

• Depth principle: active IR stereo. Two infrared imagers compute depth by stereo triangulation over a 50 mm baseline, and an IR dot projector sprays pseudo-random texture onto blank surfaces so the stereo matcher has something to grip. Because the underlying method is plain stereo, it degrades gracefully to passive stereo outdoors. It is not time-of-flight and not coded structured light (that was the older SR300 family); say the wrong one and the interview changes tone.
• Shutters, the detail almost everyone gets wrong. The two depth/IR imagers are global shutter: the whole frame is captured at one instant, so fast motion does not warp geometry. The RGB sensor is rolling shutter: rows are read out sequentially, so fast yaw skews the color image. Consequence: feature tracking on RGB suffers exactly when the robot moves fast. The known trick is to run visual odometry on the left IR image instead, which is global shutter and shares the depth imager's viewpoint.
• When to disable the emitter. The projector is rigidly attached to the camera, so its dots slide across surfaces as the camera moves. They are excellent texture for stereo matching (dots at one instant are consistent between the two IR imagers) but poison for feature tracking, because a dot is not a fixed 3D world point. If you run odometry on IR, either turn the emitter off, or use the driver's emitter on/off alternation so depth frames get dots and odometry frames get clean IR.
• The "i" is a 6-DOF IMU, a Bosch BMI055. It is mediocre: fine for gravity direction and short-horizon motion prediction through a Madgwick or complementary filter, nowhere near good enough to be your only odometry. It ships uncalibrated; run rs-imu-calibration once per unit or your gravity vector is subtly wrong forever, and your camera-IMU extrinsics still need the Chapter 2 treatment if you feed it to a real VIO.
• Envelope. Depth is trustworthy from about 0.3 m to 3 m. Error grows quadratically with distance, a stereo-baseline fact: depth error scales as z² divided by focal length times baseline. In practice that means a few millimeters at 1 m, roughly 2% at 2 m, and 4 to 6% by 4 m, at which point walls turn into fog. Depth FOV is about 87°×58°.
• Knobs. Emitter power, visual presets (High Accuracy, High Density), and post-processing filters (decimation, spatial, temporal). Filters make clouds prettier and add latency, and temporal filtering smears moving objects into geometry that ICP will then trust. Nothing is free.

5.5Parameters Worth Knowing by Name

RTAB-Map exposes several hundred parameters; rtabmap --params prints them all with defaults and descriptions, and that command is the authoritative reference, not this table. But a working set of ten covers most tuning conversations. Values below are stock defaults; treat every suggested change as a starting point.

In conversation, drop two or three of these, no more, or it sounds recited. The pairs that signal experience: Rtabmap/TimeThr with Mem/STMSize (the memory budget), and Vis/MinInliers with RGBD/OptimizeMaxError (the two gates a loop closure must pass).

5.6The Database

Everything RTAB-Map knows lives in one file, by default ~/.rtabmap/rtabmap.db, and it is a plain SQLite database. Open it with the sqlite3 CLI if you are curious; the schema has tables for nodes, links, visual words, and calibration. Each node stores its pose, its BoW signature, and compressed sensor data: JPEG for color, PNG for 16-bit depth. This single-file design is a quiet superpower: copying a map to another robot is scp, and a bug report can attach the entire session.

rtabmap-databaseViewer is the post-mortem tool and you should live in it. Open yesterday's database and you can scrub through every keyframe, see each loop closure with its inlier matches drawn between the two images, step through rejected closures and read why they failed (too few inliers, optimization error too high), inspect the graph before and after optimization, and regenerate 2D grids with different Grid/ parameters without re-driving the robot.

Multi-session mapping falls out of the memory design. Start rtabmap against an existing database without deleting it and a new session begins, initially disconnected from the old graph. The moment the Bayesian filter recognizes a place from a previous session, an inter-session loop closure welds the sessions together and optimization aligns them. Set Mem/InitWMWithAllNodes=true to preload the whole old map into WM at startup so that first weld happens sooner. This is how you map a building floor by floor across a week.

Size and pruning. Budget roughly half a megabyte per node at 848×480. At the default 1 Hz detection rate while moving, that is around 25 to 30 MB per minute; a 20-minute apartment session lands near 300 to 500 MB and a multi-hour facility run reaches several gigabytes. When that hurts: lower Rtabmap/DetectionRate, reduce image size, or set Mem/BinDataKept=false to stop storing raw images entirely (the map still works, but databaseViewer post-mortems and grid regeneration are gone, so leave it on during development). rtabmap-reprocess can rebuild a database with new parameters or assemble several databases into one.

# easiest: the prebuilt binaries from the GitHub releases page
#   https://github.com/introlab/rtabmap/releases   (AppImage / deb / exe)
# or build from source with RealSense support, plain CMake:
sudo apt install libopencv-dev libpcl-dev librealsense2-dev   # + g2o/GTSAM, see wiki
git clone https://github.com/introlab/rtabmap.git
cmake -S rtabmap -B build -DCMAKE_BUILD_TYPE=Release -DWITH_REALSENSE2=ON
cmake --build build -j4 && sudo cmake --install build

Mem/STMSize              30
Rtabmap/DetectionRate    1
Vis/MinInliers           25
RGBD/OptimizeMaxError    3.0
Grid/RangeMax            3.0
Grid/MaxObstacleHeight   1.5

rtabmap-databaseViewer ~/.rtabmap/rtabmap.db

5.7Field-Tested Gotchas

5.8Localization-Only Mode, the Deployment Split

Set Mem/IncrementalMemory=false and RTAB-Map stops adding nodes. The same BoW machinery, the same Bayesian filter, the same geometric verification now serve one purpose: recognize where the robot is inside the stored graph and publish the map → odom correction. Nothing new is learned; CPU and memory flatten out; the database stops growing. This is the standard product lifecycle: map once, localize daily. Chapter 9 picks up the navigation half of that lifecycle.

Behavior on wake is worth stating precisely, because it drives system design. At startup the robot does not know where it is; until the first loop-closure-style match against the stored map, the map → odom transform is not corrected and any pose you read is a guess. Good deployments start the robot at a distinctive, well-mapped dock so relocalization fires within the first few frames. After that, localization updates arrive whenever a view matches the map, and odometry carries the robot between updates.

Then the world changes, because worlds do. A moved chair costs a few inliers and nothing else; the Bayesian filter over many words shrugs off partial change. Rearrange the furniture wholesale, change the lighting from daylight to sodium lamps, or repaint, and match scores sag until relocalization becomes sporadic exactly where the change happened. The honest options: remap on a schedule, or run occasional sessions in mapping mode (with Mem/InitWMWithAllNodes=true) so the map tracks the world, then return to localization mode. RTAB-Map's multi-session memory was built for exactly this maintenance loop.

# poses, TUM format (feed these to evo per Chapter 8)
rtabmap-export --poses ~/.rtabmap/rtabmap.db
# assembled cloud + 2D occupancy grid regenerated with your Grid/ params
rtabmap-export --cloud --2d_map ~/.rtabmap/rtabmap.db

5.9When to Outgrow This Rig

The rig has a ceiling, and recognizing it is cheaper than fighting it. Symptoms and their prescriptions:

• Long corridors, big warehouses, outdoor yards. The D435i's useful depth ends near 3 m and sunlight eats the projector. You want lidar, and probably lidar-inertial odometry with it (Chapter 3). The safety conversation points the same way: a certified stop system wants a safety-rated lidar, which no RealSense will ever be.
• Fast or aggressive motion. Drones, handheld scanning, anything that yaws faster than the tracker survives. You want tightly coupled VIO with a properly calibrated IMU, which is Chapter 4's territory, feeding RTAB-Map as the external odometry or replacing the stack outright.
• Dense reconstruction or semantics as the product. If the deliverable is a mesh or a labeled scene rather than a navigation map, the learning-based systems of Chapter 7 and real GPU compute enter the picture.

What you keep even after outgrowing the camera: almost everything around it. RTAB-Map runs happily on lidar input (Reg/Strategy=1), the map-once-localize-daily lifecycle carries over unchanged, the database workflow and databaseViewer habits carry over, and the tf contract is identical no matter what sensor feeds it. The rig is a starter home; the plumbing is permanent.

5.10Reading List

5.11Interview Questions, Answer Out Loud First