Evaluation, Datasets & Benchmarks

8.1Why Evaluation Is Its Own Skill

Every SLAM demo looks right. The point cloud is dense, the trajectory is a smooth ribbon, the loop visibly snaps shut in the viewer. And none of that tells you whether the robot ended the run 4 cm or 4 m from where it believed it was. Human eyes are spectacularly bad at judging metric accuracy from a 3D viewer: a map can be off by 2% of scale, sheared by a degree of uncalibrated IMU mounting, or missing an entire wing that the operator never drove through, and it will still screenshot beautifully.

The deeper problem is that in the field there is no ground truth. Your warehouse has no motion capture rig. So evaluation splits into two distinct activities: benchmarking on public datasets where someone spent real money on a reference trajectory, which tells you how your algorithm compares to the literature, and field evaluation on your own robot, which requires you to manufacture partial ground truth (surveyed markers, return-to-start tricks, a slower but better reference system) and to be honest about what you can and cannot claim. Both are engineering work, not an afterthought for the last week of the project.

It is also the interviewer's favorite trap. "Your SLAM works. Prove it." Candidates who answer with viewer screenshots stop being senior candidates at that moment. The expected answer names a metric, an alignment convention, a dataset, a number of runs, and a failure-counting policy. By the end of this chapter that answer will be reflex.

8.2Trajectory Metrics That Matter

Two metrics dominate, both introduced to the community in the form we use today by Sturm et al.'s TUM RGB-D benchmark (IROS 2012): absolute trajectory error and relative pose error. They answer different questions, and each one lies in a specific way, so you report both.

Align First, Then Measure

Before any comparison, remember that your estimator starts at its own arbitrary origin. The estimate lives in one coordinate frame, the ground truth in another, and the offset between them is not an error, it is a gauge freedom (Chapter 1's unobservable directions, back again). So step zero is to find the rigid transform that best fits the estimate onto the reference, in the least-squares sense. The closed-form solution is Umeyama's method (1991): given two matched point sets it returns the optimal rotation, translation, and optionally a scale factor. For metric systems (stereo, RGB-D, lidar, VIO) you align with a scale-free SE(3) transform. For monocular vision, where scale is unobservable, you must align in Sim(3), letting scale float; anything else measures the arbitrary initial scale rather than the algorithm.

That scale switch is not a technicality. A Sim(3)-aligned monocular result silently forgives all global scale error, so a mono number and a stereo number in the same table are not comparing the same thing. Papers that hide this in a footnote are committing sin number two of Section 8.8.

ATE and RPE, Precisely

Notation: Q_1..Q_N ground truth poses, P_1..P_N estimated poses, both in SE(3),
          matched by timestamp. trans(.) is the translation part, angle(.) the
          rotation angle of a transform.

Step 0 (alignment):  S = argmin over S of  SUM || trans(S * P_i) - trans(Q_i) ||^2
                     S in SE(3) for metric systems, Sim(3) for monocular.
                     Closed form: Umeyama 1991.

ATE:   F_i = Q_i^-1 * S * P_i                      (residual at pose i)
       ATE_rmse = sqrt( 1/N * SUM || trans(F_i) ||^2 )
       (also report mean and median; rotation ATE from angle(F_i) if wanted)

RPE:   over pose pairs (i, i+D), D fixed in seconds, meters, or frames:
       E_i = ( Q_i^-1 * Q_{i+D} )^-1 * ( P_i^-1 * P_{i+D} )
       RPE_trans = rmse of || trans(E_i) ||        RPE_rot = rmse of angle(rot(E_i))
       No global alignment needed: relative motions cancel the frame offset.

ATE measures global consistency: after the best rigid fit, how far is each estimated pose from where it should be? Loop closures improve it dramatically because they bend the whole trajectory back into shape. RPE measures local accuracy: over each 1-second or 10-meter chunk, how wrong was the estimated motion? It is the drift-rate metric, and it barely notices loop closures. The classic pattern: a loop-closing system posts a great ATE with mediocre RPE (globally consistent, locally jittery), while a good odometry-only stack posts excellent RPE and an ATE that grows with every meter of drift. If you feed a controller (Chapter 9), RPE jitter is what shakes the robot; if you draw a floor plan, ATE is what warps it.

Drift as a Percentage, and Per Axis

For odometry, the most communicable numbers are normalized: translational drift as a percentage of distance traveled and rotational drift in degrees per meter. This is exactly what the KITTI odometry benchmark computes: RPE evaluated over sub-trajectories of 100 to 800 m, averaged, reported as % and deg/m. Calibrated expectations: a competitive stereo or lidar odometry sits below 1% translation drift, KITTI leaders below 0.5%, and rotation around 0.002 to 0.005 deg/m (roughly 2 to 5 degrees per kilometer). If your stack drifts 5% you have a bug or a calibration problem, not a tuning problem.

Since drift percentages only mean something against a baseline, calibrate your expectations with typical good numbers by system class:

Also look at errors per axis before averaging them away. With an IMU in the loop, gravity makes roll and pitch observable, so real drift concentrates in yaw and translation (the four unobservable directions of VIO). A ballooning z error on a ground robot usually means bad extrinsics or a degenerate scene (Chapter 3's long corridors), and you will only see it if you plot x, y, z separately.

Timestamps, the Silent Killer

ATE and RPE both require matched pose pairs, and estimate and ground truth almost never share timestamps. Tools associate by nearest neighbor within a tolerance (evo's default --t_max_diff is 0.01 s) and can apply a constant --t_offset for unsynchronized clocks. Do the arithmetic on what sloppiness costs: at 1 m/s, a 100 ms clock offset injects 10 cm of pure fiction into ATE, which is larger than the true error of every serious system in this book. If your numbers look mysteriously bad, audit time first (Chapter 2's clock-sync material pays its rent here).

ATE / APE

Absolute trajectory error: per-pose residual against ground truth after a best global alignment. evo generalizes it as APE (absolute pose error), which can also score rotation or the full pose.

RPE

Relative pose error: mismatch of motion increments over a fixed step in time, distance, or frames. The drift metric. Alignment-free by construction.

Umeyama alignment

Closed-form least-squares fit of rotation, translation, and optional scale between two matched point sets. What --align runs under the hood.

Sim(3)

SE(3) plus a scale factor, 7 degrees of freedom. Required alignment group for monocular results; forgives global scale error by design.

Association

Matching estimated and reference poses by timestamp before comparison. Wrong tolerance or a clock offset corrupts every downstream statistic.

8.3The Tooling: evo and Friends

Nobody computes these by hand. The community standard is evo, a Python package that reads every common trajectory format, aligns, computes APE and RPE with full statistics, plots, and packages results into zip files you can compare across runs. The command vocabulary is small:

pip install evo --upgrade

evo_ape tum  gt.txt est.txt -va --plot --save_results run1_ape.zip   # -a align, -v verbose
evo_ape tum  gt.txt est_mono.txt -vas          # -s adds scale: Sim(3), monocular only
evo_rpe tum  gt.txt est.txt --delta 1 --delta_unit m -va             # drift per meter
evo_traj tum est1.txt est2.txt --ref gt.txt -p --plot_mode xz        # overlay trajectories
evo_res run*_ape.zip -p --save_table results.csv                     # compare result zips

The evo_traj plot tabs (xyz and rpy against time) are where per-axis drift becomes visible, and evo_res turns a directory of result zips into the comparison table and box plots for your report. Flags worth knowing: --align_origin (match only the first pose, useful when you want to see raw drift), --n_to_align (align on a subsequence), and --t_max_diff / --t_offset from the previous section. Exact flags move between versions, so trust evo_ape --help over any blog post, including this one.

For publication-grade odometry evaluation, rpg_trajectory_evaluation from the Zurich RPG group implements the methodology of Zhang and Scaramuzza's IROS 2018 tutorial: relative error computed over a spread of sub-trajectory lengths (say 10, 40, 90, 160, 250 m), presented as box plots of error against segment length, plus multi-run aggregation from a yaml config. That relative-error-versus-distance curve is the single most informative odometry plot in the field, because it shows how error grows, not just its average.

Half of real evaluation work is format wrangling, so learn the formats once:

TUM   : timestamp tx ty tz qx qy qz qw     one pose per line, seconds, gravity-free choice of frame
KITTI : r11 r12 r13 tx r21 r22 r23 ty r31 r32 r33 tz    3x4 row-major, NO timestamps (index-matched!)
EuRoC : csv, nanosecond timestamps, position then quaternion with w FIRST

python3 -m venv ~/evo-env && source ~/evo-env/bin/activate
pip install evo --upgrade
git clone https://github.com/MichaelGrupp/evo.git
cd evo/test/data

# 1. absolute error, SE(3) Umeyama alignment, verbose, with plots and a result zip
evo_ape tum fr2_desk_groundtruth.txt fr2_desk_ORB.txt -va --plot --save_results ape.zip

# 2. relative error per second of motion (the local drift rate)
evo_rpe tum fr2_desk_groundtruth.txt fr2_desk_ORB.txt --delta 1 --delta_unit s -va --save_results rpe.zip

# 3. overlay estimate on reference, inspect per-axis tabs
evo_traj tum fr2_desk_ORB.txt --ref fr2_desk_groundtruth.txt -pa --plot_mode xyz

# 4. compare the two result zips side by side
evo_res ape.zip rpe.zip -p

8.4Ground Truth, and Where It Comes From

"Ground truth" is a marketing term. What datasets actually have is a reference measurement whose error is 10 to 100 times smaller than the thing being evaluated, and every source has a price, a working volume, and a failure mode you should be able to recite:

The pseudo-ground-truth route deserves respect because it is the only one that scales to real environments. The recipe: map the space once with a survey-grade terrestrial scanner (a Leica BLK360 or similar, millimeter-accurate point cloud), tie it to surveyed control points, then register each sensor scan of your run into that prior map with ICP, at no real-time constraint. The Newer College dataset did exactly this to get centimeter-level reference poses for a full handheld trajectory, outdoors, where no mocap could follow. The Hilti challenge did the surveyed-control-points version on construction sites.

RTK (Real-Time Kinematic)

GNSS positioning using carrier-phase corrections from a nearby base station. A solid "fix" gives 1 to 2 cm; a "float" solution quietly degrades to decimeters, so log the fix status alongside the poses.

NTRIP

Corrections streamed over the internet from a network of reference stations, so you can have RTK without owning a base station. Coverage and latency vary by region and provider.

Control point

A surveyed marker with known coordinates. Ties your map, your reference scan, and the building drawings into one frame; the anchor of every pseudo-ground-truth pipeline.

Prism

The retroreflector a robotic total station locks onto and tracks. Gives millimeter positions of one point on the robot, so orientation must come from elsewhere or from multiple prisms.

Pseudo ground truth

A reference trajectory produced by a much better offline pipeline (survey map plus unconstrained ICP), blessed by human review and honest about its own 2 to 10 cm error.

Know the circularity trap in your bones. If your reference trajectory came from a lidar-ICP pipeline and your system under test is a lidar-ICP pipeline, shared systematic errors cancel and your scores flatter you. The reference must be independent in mechanism, not just in software version: survey instruments for a SLAM stack, or at minimum a far richer sensor suite run without real-time limits.

8.5The Datasets Everyone Reports On

Datasets are where the field agrees to disagree in public. Each one encodes a sensor suite, an environment, and a set of failure modes it punishes, and knowing which is which tells you how to read any results table:

KITTI odometry is the automotive standard: 22 driving sequences from Karlsruhe, 11 with public ground truth and 11 held out for the leaderboard, scored by the segment-based drift metric of Section 8.2. Its 10 Hz cameras and abundant moving traffic make it a stress test for outlier rejection, and its long straight roads made monocular scale drift famous.

EuRoC MAV (Burri et al., IJRR 2016) is eleven drone flights, graded easy to difficult, with superb hardware synchronization. It has been the standard VIO table for a decade: every visual-inertial paper you read in Chapter 4 reports centimeter ATEs on MH_01 through V2_03. The difficult Vicon-room sequences still break systems with motion blur and near-darkness.

TUM RGB-D (Sturm et al., IROS 2012) is the dataset that taught the community how to evaluate at all: it shipped ATE, RPE, association scripts, and an online evaluation service alongside the data. Kinect v1 imagery, handheld and wheeled sequences, mocap reference. Still the default for RGB-D systems like Chapter 5's RTAB-Map.

TUM VI covers what EuRoC cannot: long handheld routes through corridors and outdoors where the mocap reference exists only in the room where runs start and end. It therefore measures accumulated drift honestly, and its photometrically calibrated fisheye stereo made it the benchmark of choice for direct methods.

Newer College and its successors matter because the reference covers the whole outdoor trajectory (the ICP-against-survey-scan trick above), so lidar SLAM can finally be graded away from a mocap cage. NCLT is the long-term-autonomy dataset: 27 sessions across 15 months of Michigan weather, the benchmark for whether your map survives seasons. The Hilti SLAM Challenge brought prize money and millimeter-surveyed construction sites, and promptly humbled a generation of systems that were excellent on EuRoC; its sequences are the closest public proxy to "real industrial deployment." M2DGR gives ground robots the multi-sensor treatment with the transitions that actually kill deployments (doorways to sunlight, elevators, night). And the DARPA SubT challenge was scored not by ATE but by reporting artifact positions within 5 m in underground courses, a reminder that in the end the customer metric is "did the map put the thing where the thing was."

Reading a results table now becomes pattern matching. A system that wins EuRoC but skips Hilti is telling you something. So is a lidar paper that reports KITTI (flat, structured, 64-beam) but not Newer College (handheld, sparse, vegetated). Absence of a dataset from a table is data.

8.6Scoring the Map Itself

Trajectory error is a proxy. If the deliverable is the map (Chapter 3's map products), you can and should score the map directly against a reference point cloud from a survey scanner. The two halves of that score pull against each other:

accuracy      = mean over p in map of      dist(p, reference)     "is what I built correct?"
completeness  = fraction of q in reference with dist(q, map) < tau  "did I build all of it?"
chamfer       = mean_{p in map} dist(p, ref)  +  mean_{q in ref} dist(q, map)
F-score(tau)  = harmonic mean of precision and recall at threshold tau

Accuracy alone rewards timid maps (only map what you are sure of); completeness alone rewards reckless ones (spray points everywhere). Chamfer distance and the F-score at a stated threshold (commonly 5 or 10 cm) combine both, and the reconstruction community brought the F-score convention over to neural mapping papers in Chapter 7. Use point-to-plane distances on structured scenes if you can; point-to-point penalizes sampling density differences that are not errors.

For 2D occupancy grids the comparison is cell-wise against a reference grid over the cells both maps claim to know: a confusion matrix of occupied/free, or simply whether a planner finds the same routes in both (a map metric your operations team actually cares about). When no reference exists at all, no-reference sharpness measures like mean map entropy and mean plane variance score how crisp the cloud is: compute the local surface distribution around each point and average its entropy; double walls and smeared ceilings show up as high entropy. They catch degradation between runs even though they cannot certify absolute accuracy.

Why do map metrics lag trajectory metrics in adoption? Reference clouds cost a survey scanner and a site day, alignment between map and reference adds its own ambiguity, and for decades the trajectory correlated well enough to stand in for the map. That logic inverts in industry: a surveying customer does not care where the scanner thought it was, only that the wall is where the wall is. Expect map-level evaluation to be the part of your protocol that impresses people.

8.7Benchmarking Your Own Robot Honestly

Public benchmarks rank algorithms. Your robot needs a different instrument: a repeatable protocol that detects regressions and predicts field behavior. A defensible baseline protocol looks like this:

1. Fix the routes. Two or three routes marked with tape on the floor, driven the same direction at the same speed, including at least one full loop where the finish marker is the start marker. A loop gives you a free metric with zero equipment: return-to-start error, measured with a ruler against the tape.
2. Run each route at least five times per configuration. One run is an anecdote. Report the median and the worst run, never the mean alone; SLAM error distributions are heavy-tailed and the mean hides that.
3. Count hard failures separately. A run where tracking was lost, the map forked, or the robot needed rescue is not a large error to average in, it is a different variable. Report "median ATE 6 cm, worst 14 cm, 1 failure in 15 runs", never "mean 40 cm".
4. Verify calibration the same day (Chapter 2's quick checks). Otherwise your A/B test of two algorithms is actually a test of Tuesday's extrinsics against Thursday's.
5. Record everything to bags. Every evaluation run saved as raw sensor data means every future algorithm version can be scored on the exact same inputs, which converts your field time into a permanent private benchmark.

When a number regresses, resist the urge to tune. Ablate: change exactly one thing at a time (feature count, IMU weight, loop-closure threshold) and re-run the full protocol on the recorded bags, so the regression gets localized to a module instead of blamed on the weather. Ablation on recorded data costs CPU time; ablation by re-driving the robot costs days and adds noise.

Simulation earns a place here too, with its limits stated. Sim gives you perfect ground truth, infinite runs, and controllable hazards, which makes it ideal for logic and integration testing (does the loop closure fire, does the map merge). But rendered sensors flatter SLAM: perfect timestamps, no rolling shutter, clean lidar returns, textures with convenient gradients. Treat sim numbers as an upper bound on field performance and never let a sim-only improvement ship without a bag-replay confirmation.

8.8The Seven Deadly Sins of SLAM Evaluation

Each of these appears in published papers and in internal slide decks. Learn to spot them in others' results and, less comfortably, in your own:

1. The Single Golden Run. One trajectory, chosen after the fact, presented as typical. Nondeterministic systems (RANSAC, multi-threaded optimization) can vary 2x between identical runs; a single number is a coin flip you got to re-flip.
2. Scale-Aligned Mono Presented as Metric. A Sim(3)-aligned monocular ATE in the same table as SE(3)-aligned stereo results, alignment group unstated. The mono number has been forgiven the one error class that dominates mono in practice.
3. Grading on the Training Environment. Parameters tuned in the same building, same lighting, same route as the evaluation. The number measures memorization. Hold out an environment the way ML people hold out data.
4. The Vanishing Failure Run. "Runs where tracking was lost were excluded." Excluded from the average, and from the paper, and from the reader's risk model. Failures are the most decision-relevant statistic you have.
5. ATE-Only Reporting. A strong loop closure can pull global error down while local odometry jitters badly, and it is the jitter that shakes the controller and blurs the map. Without RPE alongside ATE, half the behavior is invisible.
6. Tuning on the Test Set. Config files with per-sequence parameters for EuRoC sequences. If your yaml mentions MH_04 by name, the benchmark has stopped measuring generalization.
7. Screenshot-Driven Development. Merging a change because the map "looks cleaner." Looks-cleaner does not survive contact with Section 8.1. Every one of the six sins above starts here, with the belief that eyes are a metric.

8.9Regression Testing SLAM in CI

Everything above becomes routine the day it runs on every commit. The pattern is golden-run replay: a small set of blessed recordings fed through the full SLAM stack in CI, the output trajectory scored with evo, and hard gates on the result. A SLAM regression then fails a pipeline instead of a customer site.

Three kinds of gates earn their keep. Accuracy gates: ATE rmse against the recording's reference trajectory below a threshold, with the threshold set from measured run-to-run variance (blessed median plus three standard deviations over ten replays), not from optimism. Resource gates: wall-clock processing time and peak memory as hard budgets, because a 20% CPU regression is a field failure on your smallest robot even when accuracy is untouched. Liveness gates: the stack produced poses for the whole recording, no gap longer than N seconds, loop closure fired on the loop route. Where does the reference trajectory come from? For a mocap or RTK-covered recording, from the instrument; for everything else, bless a pseudo-ground-truth run (Section 8.4): the full offline pipeline at leisure settings, reviewed once by a human, then frozen. The gate then measures drift from blessed behavior, which is regression testing, and only claims that.

# ci/slam_regress.sh
#!/usr/bin/env bash
set -euo pipefail
DATA=goldens/warehouse_loop_2026_06        # blessed recording, stored in LFS or an artifact store
REF=goldens/warehouse_loop_ref.tum         # blessed reference trajectory (TUM format)
OUT=$(mktemp -d)

# run the stack under test over the frozen data; it writes $OUT/est.tum
/usr/bin/time -v -o "$OUT/time.txt" \
    kiss_icp_pipeline --config ci/pinned_config.yaml "$DATA" --out "$OUT"
mv "$OUT"/*_tum.txt "$OUT/est.tum" 2>/dev/null || true

python3 ci/ate_gate.py "$REF" "$OUT/est.tum" 0.15
python3 ci/resource_gate.py "$OUT/time.txt" --max-rss-mb 900

# ci/ate_gate.py
#!/usr/bin/env python3
import sys
from evo.core import metrics, sync
from evo.tools import file_interface

ref_file, est_file, thresh = sys.argv[1], sys.argv[2], float(sys.argv[3])
ref = file_interface.read_tum_trajectory_file(ref_file)
est = file_interface.read_tum_trajectory_file(est_file)
ref, est = sync.associate_trajectories(ref, est, max_diff=0.02)
est.align(ref, correct_scale=False)                  # SE(3) Umeyama, no scale
ape = metrics.APE(metrics.PoseRelation.translation_part)
ape.process_data((ref, est))
rmse = ape.get_statistic(metrics.StatisticsType.rmse)
print(f"ATE rmse {rmse:.4f} m  (gate {thresh} m, {est.num_poses} poses)")
sys.exit(0 if rmse < thresh else 1)

8.10Reading List

8.11Interview Questions