Learning-Based SLAM

7.1The Mental Model: Three Depths of Learning

Every "AI + SLAM" idea ever published fits one of three buckets. Sorting a question into its bucket out loud, before answering it, is what reads as senior. The buckets differ in how much of the classical pipeline survives:

1. Learned modules inside classical pipelines. Keep the geometric skeleton; swap individual organs for networks. Features, matching, depth priors, place recognition, dynamic masking. Production-ready today.
2. End-to-end differentiable systems. The whole pipeline trains as one network, but with a real geometric solver embedded inside it. The research-to-product edge (the DROID lineage).
3. Learned map representations. The map itself becomes a neural object: implicit fields, Gaussians, scene graphs, or the weights of a feed-forward geometry model. Research, moving fast.

Use it as a sorting machine in conversation. "Should we use learning for X" becomes three narrower, answerable questions: is there a module swap (depth 1) with a classical veto behind it, does X need gradients flowing through the solver (depth 2), or is X really a question about what the map stores (depth 3)? The depths also order the risk. Each step down replaces more auditable geometry with more learned behavior, and Sections 7.5 and 7.6 are about deciding how far down your product can afford to go.

The history explains the buckets. Around 2015 the field tried to replace everything at once (PoseNet regressed camera pose straight from pixels) and it lost badly to classical relocalization. From roughly 2018 the module era took over: SuperPoint and NetVLAD proved that networks beat hand-crafted appearance processing while geometry stayed in charge. In 2021 DROID-SLAM showed how to differentiate through the geometry rather than around it. And since 2023 the feed-forward line (DUSt3R and its children) has been asking whether a large enough model can carry the geometry prior in its weights. Each wave kept more of what the previous one learned about the division of labor: networks handle appearance, solvers handle consistency.

Global descriptor

One vector per image summarizing its appearance. Place recognition becomes nearest-neighbor search in descriptor space.

Pointmap

A per-pixel 3D coordinate image: H×W×3, each pixel mapped to a point in some camera frame. The output format of the DUSt3R family.

Affine-invariant depth

Depth predicted up to an unknown per-image scale and shift. Great for ordering, useless as a metric measurement until you solve for those two numbers.

Differentiable bundle adjustment

A least-squares solver written so gradients pass through its iterations, letting a network upstream train against the solver's output error.

Sim-to-real gap

The accuracy drop when a model trained on synthetic images meets real sensors. Small for low-level tasks like matching, large for semantic ones.

VPR

Visual place recognition: "have I seen this place before?", answered by retrieval rather than geometry.

7.2Depth 1: Learned Modules, One Good Option per Slot

Recall the pipeline from Chapters 3 through 5. Each slot has a classical incumbent and a learned drop-in, and for each slot you should be able to name the 2026 pick and defend it:

Features: SuperPoint, DISK, ALIKED

SuperPoint (2018) is still the default, less because it is the best detector than because everything downstream was trained against it: matchers, localization toolboxes, pretrained weights everywhere. Its training trick is worth knowing on its own. There are no human labels for "what is a good keypoint", so it bootstraps: train on synthetic corners, then warp real images with random homographies and require the detector to agree with itself across warps (homographic adaptation). DISK trains detection and description jointly with a reward signal on successful matches, and tends to fire more densely, which helps offline reconstruction recall. ALIKED uses deformable convolutions for a lighter, faster network with strong accuracy per watt. The 2026 rule: SuperPoint when you want the ecosystem (hloc, pretrained LightGlue), ALIKED when you are counting milliseconds on an embedded GPU, DISK for max-recall offline mapping.

Matching: SuperGlue to LightGlue, and the Dense Option

SuperGlue (2020) replaced the nearest-neighbor-plus-ratio-test with a transformer that reasons over both keypoint sets jointly, using position and context, and it roughly doubled match quality under viewpoint and lighting stress. LightGlue (2023) is the deployable version: it prunes easy points early and exits early on easy pairs, so compute adapts to difficulty. An easy pair costs a fraction of a hard one. When keypoints themselves fail (texture-poor walls, repetitive facades), detector-free dense matchers apply: LoFTR matches on a coarse grid then refines; RoMa (2024) is the current heavyweight, robust to extreme baselines but too slow for a hot loop. Pick LightGlue by default; reach for RoMa offline or when overlap is desperate.

Monocular Depth, and the Metric Trap

Depth Anything V2 made single-image depth genuinely good: crisp edges, robust across scenes, trained by distilling a huge teacher through synthetic data. But read the fine print on what "depth" means. The base models are affine-invariant: correct up to an unknown scale and shift per image. Feed that raw into a metric SLAM back-end and you have injected a measurement whose units the estimator cannot know. The metric-tuned variants (and intrinsics-aware models like Metric3D v2 and UniDepth) help, with caveats: they are biased toward their training distribution (indoor models fail outdoors and vice versa), they silently assume your camera's intrinsics resemble what they saw in training, and consecutive frames can flicker in scale because each prediction is independent. The honest uses: resolving monocular scale ambiguity as a prior with generous covariance, densifying sparse maps, and bootstrapping initialization. Not as a stereo replacement in the measurement model.

Optical Flow: RAFT and Its Descendants

RAFT (2020) matters to SLAM less as a module you deploy and more as the architecture that made depth 2 possible: an all-pairs correlation volume plus a recurrent update operator that iteratively refines a flow field. DROID-SLAM is RAFT with a geometric solver in the loop. If you need standalone flow in 2026 (dynamic object checks, video stabilization), SEA-RAFT gives near-RAFT accuracy several times faster.

Place Recognition: NetVLAD to MixVPR and SALAD

This is the slot where learning wins hardest, because bag-of-words dies precisely when you need it: revisiting a place under different lighting, weather, or season. NetVLAD (2016) made the global-descriptor recipe standard: CNN features aggregated into one vector, trained on Street View revisits so the vector is invariant to the things that change and sensitive to the things that do not. MixVPR (2023) swapped the aggregation for feature mixing and jumped the benchmarks; SALAD (2024) put a DINOv2 backbone under an optimal-transport aggregation and jumped them again. Retrieval is cheap at runtime (one descriptor per keyframe, nearest-neighbor search), so there is little excuse for BoW in a new design unless you are CPU-only. Every retrieved candidate still goes through geometric verification before it touches the graph; retrieval proposes, RANSAC disposes.

Dynamic Object Masking: The DynaSLAM Idea

Classical SLAM assumes a static world and treats moving things as outliers for the robust kernel to absorb. That works until the moving things cover half the image (a bus at a stoplight, a crowd). DynaSLAM (2018) made the fix explicit: run a segmenter (Mask R-CNN then, any fast instance segmenter now), mask out the a-priori-dynamic classes (people, vehicles) before feature extraction, and add a multi-view geometric consistency check to catch movable-but-currently-static objects, like a parked chair that later gets carried away. The recipe survives unchanged in 2026 because it respects the boundary: the network answers "what pixels are untrustworthy", the geometry does everything else. Budget one segmentation pass per keyframe, not per frame, and it fits on an embedded GPU.

Learned Uncertainty

The quietest and most underrated slot. Classical pipelines carry hand-tuned constants where covariances should be; networks can predict per-measurement confidence instead: which matches to trust, how much to weight a depth pixel, when tracking quality is degrading. DROID-SLAM's confidence weights are the proof it matters, since the system's accuracy lives in its ability to down-weight bad correspondences. The catch is calibration: a raw confidence head is a ranking signal, not a probability. Before you use one as a covariance, recalibrate it on held-out data (temperature scaling is often enough) and verify the χ² statistics downstream actually look χ². Mentioning this unprompted is an interview point few candidates score.

git clone --recursive https://github.com/cvg/Hierarchical-Localization
cd Hierarchical-Localization
python -m pip install -e .        # pulls torch, pycolmap, lightglue
mkdir -p datasets/room/mapping datasets/room/query
# copy 30 mapping photos and 3 query photos into those folders

from pathlib import Path
from hloc import extract_features, match_features, reconstruction, \
                 pairs_from_exhaustive, localize_sfm

images  = Path("datasets/room/mapping")
outputs = Path("outputs/room"); outputs.mkdir(parents=True, exist_ok=True)
sfm_pairs, feats, matches = outputs/"pairs-sfm.txt", outputs/"feats.h5", outputs/"matches.h5"

fconf = extract_features.confs["superpoint_max"]      # ~2048 kpts/image
mconf = match_features.confs["superpoint+lightglue"]

refs = [p.name for p in images.iterdir()]
extract_features.main(fconf, images, feature_path=feats)
pairs_from_exhaustive.main(sfm_pairs, image_list=refs)   # 30 imgs = 435 pairs
match_features.main(mconf, sfm_pairs, features=feats, matches=matches)
model = reconstruction.main(outputs/"sfm", images, sfm_pairs, feats, matches)

7.3Depth 2: End-to-End Done Right

Chapter 4 covered DROID-SLAM as a visual SLAM system; here it matters as a lesson in architecture. Read it from the learning side and ask: what did the network actually have to learn, and what was it spared from learning?

What the network learns: a RAFT-style GRU that, given correlation features between frame pairs, predicts revisions to the current dense correspondence field, plus a confidence weight per pixel. That is all. What stays geometric: the differentiable bundle adjustment layer, an honest damped Gauss-Newton solver over poses and per-pixel depths, Schur complement and all, that converts those weighted correspondences into globally consistent geometry. The network never learns projection, triangulation, or consistency; those come for free, exactly, from the solver. And because gradients flow through the solver's iterations during training, the loss on final pose error teaches the network to produce correspondences and confidences that the solver digests well. The network learns to feed geometry, not to imitate it. That sentence is the answer to "why does differentiable BA matter?"

DPVO (2023) is the same recipe made frugal: track sparse random patches instead of dense fields, cutting compute and memory several-fold to reach real time on a modest GPU, with DPV-SLAM (2024) adding loop closure back. The dense-to-sparse move cost surprisingly little accuracy, which tells you the value was never in density; it was in learned correspondence plus exact optimization.

The training data story is the part people skip. These systems train almost entirely on TartanAir, a large synthetic dataset rendered in Unreal Engine with perfect ground-truth poses, depth, and flow across aggressively varied environments (forests, factories, fog, snow). Sim-to-real transfer works here because correspondence is a low-level task: matching textures across two rendered views exercises the same machinery as matching real ones, and the domain gap that ruins synthetic training for semantic tasks barely bites. TartanVO made this point first: a learned odometry model trained only in simulation that generalized across real datasets. The uncomfortable corollary: if your deployment domain has structure TartanAir lacks (fisheye optics, rolling shutter artifacts, underwater scattering), the pretrained model has never seen anything like your data, and you will find out at runtime.

Know the cautionary tale at the other extreme: absolute pose regression (PoseNet, 2015, and descendants). Feed an image, regress the 6-DoF pose directly, no geometry anywhere. It consistently loses to classical relocalization because without a geometric bottleneck the network can only memorize the scene, interpolating between training viewpoints and failing on new ones. The one-line verdict: end-to-end without a geometric bottleneck degenerates into memorization.

7.4Depth 3: Learned Maps

Implicit Fields and Gaussians as Maps

A map does not have to be a point cloud or a voxel grid. It can be a function: query a 3D coordinate, get back occupancy, signed distance, color, or a feature vector. On the camera side the line runs iMAP (a single MLP as the whole map) through NICE-SLAM (hierarchical feature grids for scale) to Co-SLAM (hash grids for speed). Gaussian-splatting SLAM (SplaTAM, MonoGS) trades the implicit function for an explicit soup of Gaussians that rasterizes fast; Chapter 4 covers that era from the SLAM side. On the lidar side the standout is PIN-SLAM (2024): an elastic, point-anchored implicit neural SDF that deforms with loop closures and runs at practical speed. It is the best single citation for "something newer than OctoMap, learned edition".

Semantic Fields and Scene Graphs

Robots increasingly need "which room am I in, where is the chair", not just occupied cells. Kimera builds metric-semantic meshes in real time; Hydra (2022) builds full 3D scene graphs, a live hierarchy of places, rooms, and objects; ConceptGraphs and Clio (2024) make the graphs open-vocabulary and task-driven. A parallel thread fuses CLIP-style language features into the 3D map itself (OpenScene, LERF-style feature fields) so you can query the map with a sentence: "find the fire extinguisher". Related and rising: scene completion, where a network hallucinates the geometry your sensors have not seen yet (the far side of furniture, occluded corridors) so the planner has something better than "unknown". Useful, provided everyone downstream remembers which voxels are measured and which are guessed.

The Feed-Forward Line: The Model Is the Geometry Prior

DUSt3R (2024) took the most radical position in this chapter: give a transformer two images, no intrinsics, no poses, and it directly regresses a pointmap for each, both expressed in the first camera's frame, with per-pixel confidence. Correspondence, calibration, relative pose, and dense geometry all fall out of one forward pass, recoverable by post-processing. MASt3R grounds it further with a metric-scale head and a dense matching head, and MASt3R-SLAM (2025) wraps a real-time SLAM system around it. The philosophical shift deserves stating plainly: in the classical stack, geometric constraints are the prior and pixels are the evidence. Here, the network's weights, distilled from millions of image pairs, are the prior, and two-view geometry is something the model has internalized rather than something you enforce. That is why it works on impossible inputs (almost no overlap, unknown cameras) and also why it fails differently: it will always answer, and nothing structural prevents a confident wrong answer.

What Querying a Learned Map Costs at Robot Rates

A planner (Chapter 9) asks two queries at brutal rates: "is this cell occupied" and "how far is the nearest obstacle", thousands of times per planning cycle, with hard latency bounds. Classical grids and ESDFs answer in nanoseconds per lookup. A NeRF-style map answers with an MLP evaluation per sample, and distance queries need marching or many samples; Gaussian maps rasterize images fast but answer spatial occupancy queries awkwardly, since splats are neither surfaces nor volumes. The workable patterns in 2026: keep a classical occupancy or SDF layer for planning and treat the neural map as the reconstruction and semantics product, or use a learned SDF like PIN-SLAM's where distance queries are native. Sending a planner to query a radiance field directly is a demo, not an architecture.

git clone --recursive https://github.com/naver/mast3r
cd mast3r
conda create -n mast3r python=3.11 -y && conda activate mast3r
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
mkdir -p checkpoints
wget https://download.europe.naverlabs.com/ComputerVision/MASt3R/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth -P checkpoints/

from mast3r.model import AsymmetricMASt3R
from dust3r.inference import inference
from dust3r.utils.image import load_images

model = AsymmetricMASt3R.from_pretrained(
    "checkpoints/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth").to("cuda")
imgs = load_images(["a.jpg", "b.jpg"], size=512)
out  = inference([tuple(imgs)], model, "cuda", batch_size=1)

pts_a  = out["pred1"]["pts3d"]                 # H x W x 3, camera-A frame
pts_b  = out["pred2"]["pts3d_in_other_view"]   # B's pixels, also in A's frame
conf_a = out["pred1"]["conf"]                  # per-pixel confidence, >= 1

7.5Where ML Still Loses to Geometry

This section is the debate ammunition. Not "networks bad", but five specific places where the geometric stack retains a structural advantage in 2026.

Extrapolation off the training manifold. Epipolar geometry holds for every camera that ever will be built; a matcher trained on photographs holds for data resembling photographs. Fisheye lenses, thermal imagers, underwater scattering, upside-down mounting, heavy rolling shutter: each is a step off the manifold, and performance degrades unpredictably rather than gracefully. The classical stack needs a calibration file to handle a new lens. The learned stack may need a retraining campaign, and you will not know until you test.

Latency and VRAM on embedded compute. The robot's GPU is shared: perception, learned SLAM modules, video encode, maybe a language model, all contending on a Jetson-class device with a power budget in the tens of watts. SuperPoint plus LightGlue at 640×480 costs on the order of 15 to 30 ms per frame on an Orin with TensorRT, which is affordable; DROID-class dense systems want roughly 8 to 11 GB of VRAM and desktop-class throughput, which is not. DPVO shrinks it to a few GB and remains a meaningful line item. Meanwhile the factor-graph back-end runs on two CPU cores. Every learned module must argue for its slice of the GPU against everything else the robot does.

Calibration and metric guarantees. A geometric pipeline consumes your camera's actual intrinsics and extrinsics (Chapter 2) and produces estimates whose error statistics you can derive and audit: reprojection errors in pixels, covariances from the information matrix. A network was trained on other people's cameras and carries their assumptions invisibly. When a customer asks "what is your localization accuracy, with what confidence bound", the geometric stack has an answer that survives cross-examination.

Certification and testability. Safety cases want requirements traced to components, bounded input-output behavior, and regression tests that mean something. A χ² gate is one line of auditable math. A 300M-parameter matcher is a statistical artifact whose behavior on untested inputs is an empirical question, and "the transformer felt strongly about it" is not an incident report. This is why learned modules ship inside classically guarded loops in regulated domains, if they ship at all.

Failure style: silent versus graceful. When ICP degrades, residuals grow, inlier fractions fall, and the covariance balloons; you can watch it fail in telemetry and act. When a network fails off-distribution, it often fails at full confidence, producing outputs shaped exactly like good ones. The MASt3R no-overlap experiment in Setup B is the canonical demonstration. Geometric degradation is a slope; learned failure is a cliff with a painted-on road.

7.6The Hybrid Default

The 2026 production pattern is not in dispute: a learned front-end feeding a classical estimator, the factor-graph machinery of Chapter 1 under the sensor pipelines of Chapters 3 through 5. What separates a considered design from a buzzword is where you draw the boundary, because the boundary determines which failures the learned side absorbs and which safeguards the classical side retains. Three concrete placements, in increasing order of learned responsibility:

Architecture A: classical stack, learned retrieval. Keep ORB-SLAM3 or RTAB-Map (Chapter 5) intact and replace only the loop-candidate source with SALAD or MixVPR descriptors. The boundary sits at loop closure proposals. What it absorbs: appearance change across hours, seasons, and lighting, exactly where BoW dies. What it risks: almost nothing, because every proposal still passes geometric verification and graph-consistency gating; a bad network output is a wasted RANSAC call, not a corrupted map. This is the highest benefit-to-risk swap in SLAM, and the right first move for an existing product.

Architecture B: learned features and matching, classical estimation. SuperPoint plus LightGlue produce correspondences; a classical VIO or factor-graph back-end consumes them as it would any measurements. The boundary sits at 2D matches. What it absorbs: front-end brittleness under blur, low light, and weak texture, meaning tracking itself gets tougher, not just loop closure. What it preserves: the failure signature stays classical and observable (inlier counts, reprojection residuals, χ² tests), so your existing health monitoring still works. The cost: a GPU in the per-frame hot loop, and a latency budget you must defend. This is the default architecture for new visual systems in 2026, and it is exactly the stack Setup A runs offline.

Architecture C: learned dense geometry as a measurement. Monodepth or DUSt3R-style pointmaps enter the estimator as depth measurements with honest (inflated, correlated) covariances, or in the extreme, MASt3R-SLAM makes the feed-forward model the front-end outright. The boundary sits at 3D measurements. What it absorbs: monocular scale ambiguity, initialization in textureless scenes, dense geometry a sparse pipeline can never provide. What it demands: covariance discipline and consistency gating, because the failure modes of Section 7.5 (correlated scale error, confident hallucination) now enter through the measurement model, the one door the estimator trusts most. Run this boundary only with per-frame scale/shift alignment against sparse geometric points and residual gating on everything.

7.7The Trade-Offs, Argued Both Sides

Interviewers push here, and they usually push whichever side you did not take. Prepare both.

Robustness versus interpretability is the axis interviewers reach for first, and the honest answer is that it is not a strict trade: Architecture B keeps learned robustness while preserving classical observability, because the failure signals live downstream of the boundary. The real trade sits at depth 2 and beyond, where the observable signals thin out.

Data cost versus engineering cost is the axis managers reach for. A classical stack costs senior engineering time: months of tuning, per-site parameter files, an expert on call. A learned stack converts that into data cost: collection campaigns, labeling or self-supervision infrastructure, evaluation harnesses, retraining pipelines. Fleets amortize data cost beautifully, which is why fleet operators lean learned. A three-robot pilot project amortizes neither, which is why sensible small deployments start classical with learned retrieval bolted on.

And sometimes learned is the only option. Say this in the debate, because it is true and it shows range: no classical descriptor survives severe appearance change. Cross-season place recognition (summer foliage to bare winter branches), day-night relocalization, smoke and fog to clear air: hand-crafted features and BoW simply die, while descriptors trained on revisit pairs across those exact changes keep working. If the requirement is "recognize this place in February that you mapped in July", there is no geometric alternative to buy. The question is not whether to use learning there, but how thick a geometric verification wall to build behind it.

7.8Reading List, in Priority Order

7.9Interview Questions, Answer Out Loud First