Sensors, Calibration, and Time

2.1Why a Hardware Chapter

Read enough SLAM post-mortems and a pattern appears. "The map bends at the far end of the warehouse" turns out to be a lidar mounted 0.4° off from what the config claims. "VIO diverges whenever the robot turns fast" turns out to be a 15 ms camera-IMU time offset. "The corridor is 3% too long" turns out to be new casters that changed the effective wheel radius. Almost never is the estimator itself the culprit. The math was fine. The inputs were lies.

This is why senior engineers are boring about hardware. They ask what the shutter is, where the timestamp is taken, when the rig was last calibrated, and only then what algorithm is running. Interviews probe the same instinct. Anyone can name ORB-SLAM3; fewer people can say what a 5 ms sync error does to it, with a number.

Chapter 0 gave you the honest one-line summary of each sensor. Here we go one level down: the physics, the error models the estimators need, and the two rituals (calibration and clock sync) that turn a box of parts into a rig you can trust.

2.2Wheel Odometry

Encoders count shaft ticks. Everything else is bookkeeping. For a differential-drive robot with wheel radius r, encoder resolution N ticks per revolution, and wheel separation b, the pipeline from ticks to pose is short enough to memorize:

Per wheel, ticks to distance:
  Δs_r = 2πr · Δticks_r / N        Δs_l = 2πr · Δticks_l / N

Body motion over one step Δt:
  Δs = (Δs_r + Δs_l) / 2           v = Δs / Δt
  Δθ = (Δs_r − Δs_l) / b           ω = Δθ / Δt

Pose integration (midpoint rule, fine at 50 Hz):
  x ← x + Δs · cos(θ + Δθ/2)
  y ← y + Δs · sin(θ + Δθ/2)
  θ ← θ + Δθ

The midpoint rule is a first-order approximation of the true circular arc. At typical control rates the difference is microscopic, and the arc formula has a divide-by-zero at Δθ = 0 that you would have to special-case anyway.

Systematic vs Stochastic Error

Wheel odometry error comes in two flavors, and only one of them is fixable with a screwdriver. Systematic error is repeatable: the two wheels have slightly unequal diameters, or the effective wheelbase b is not what the CAD says (tires contact the ground over a patch, not a point). These make the robot curve when it should go straight, every time, in the same direction. Stochastic error is the rest: slip, bumps, floor texture. It never repeats and you can only model it as noise.

The classic recipe for the systematic part is UMBmark (Borenstein & Feng, 1996): drive a 4×4 m square clockwise and counterclockwise several times, measure where the robot actually ends up, and decompose the final-position errors into a wheel-diameter correction and a wheelbase correction. It takes an hour and routinely cuts systematic drift by a factor of five or more. The stochastic part goes into the motion model covariance from Chapter 1; the standard practice is to make rotation noise generous, because heading error is the term that turns into unbounded position error.

One warning about platform types. The equations above assume wheels that roll without slipping, which is only ever approximately true. Skid-steer and tracked vehicles turn by slipping, so the effective wheelbase is 1.2 to 1.8 times the physical one and varies with surface; calibrate it empirically and expect it to change on grass. Mecanum platforms slip by design in every direction, and their odometry is poor enough that you should treat it as a velocity hint for the fusion filter, not as a pose source.

2.3The IMU, Honestly

A MEMS gyroscope is a microscopic tuning fork. A proof mass vibrates along one axis; when the chip rotates, the Coriolis effect pushes the mass sideways in proportion to the rotation rate, and capacitive fingers read that deflection. A MEMS accelerometer is simpler still: a proof mass on silicon springs, displaced by specific force. Note the term. An accelerometer at rest does not read zero; it reads 9.81 m/s² upward, the reaction to gravity. Every IMU integration pipeline begins by deciding what to do about that ever-present g, and most IMU bugs begin with getting it wrong.

The Error Model

Estimators do not need the IMU to be good. They need to know precisely how it is bad. The standard model has four parts, and you will meet these exact names in every VIO config file:

• Bias: a slowly wandering offset added to every reading. Different at every power-on (turn-on bias), then drifting during the run.
• Bias random walk (bias instability): how fast the bias wanders. This is the number that separates a $2 IMU from a $20,000 one.
• White noise (noise density): the fuzz on each individual sample. Integrating it gives angle random walk and velocity random walk.
• Scale factor and axis misalignment: the sensor reads 1.002× the true rate, or its axes are 0.1° from orthogonal. Calibratable, temperature-dependent.

Why Double Integration Explodes

Position from an accelerometer means integrating twice, so constant errors grow quadratically and gyro-induced errors grow cubically. The cubic term comes from a nasty coupling: a gyro bias tilts your gravity estimate, and a tilted gravity estimate leaks a slice of that 9.81 m/s² into the horizontal axes as phantom acceleration. Concrete numbers for a cheap, warmed-up MEMS part:

accel bias ≈ 0.02 m/s² (2 mg):
  position error = ½·b·t²   →  1 cm at 1 s,  1 m at 10 s,  36 m at 60 s

gyro bias ≈ 0.02 °/s, leaking gravity into the horizontal:
  position error ≈ (g·b_g/6)·t³  →  0.6 m at 10 s,  123 m at 60 s

So a consumer IMU on its own is a useful position sensor for well under a second, a decent attitude sensor for minutes (gravity keeps pitch and roll observable), and a superb rotation-rate sensor at every instant. Fusion architectures are built around exactly that profile: let the IMU carry you between camera frames or lidar sweeps, and let those exteroceptive sensors reset the IMU's runaway integrals.

The table explains a lot of system design at a glance. A consumer part drifts a few degrees per hour of bias wander, which VIO corrects effortlessly, so phones and drones ship $2 IMUs and spend the savings on cameras. A survey vehicle that must coast through a two-minute GPS outage with centimeter accuracy needs the tactical row, and pays for it.

One more idea before the worked setup. Raw IMU samples arrive at 200 to 1000 Hz, far too fast to give each one its own node in a factor graph. Preintegration (Forster et al.) compresses all the samples between two keyframes into a single relative motion constraint, done on the manifold so the result can be corrected later when the bias estimate improves, without re-integrating. Chapter 1 built the manifold machinery this rests on; Chapters 3 and 4 use it in every lidar-inertial and visual-inertial system they discuss.

Noise density

White noise per sample, quoted per √Hz so it is rate-independent. Units: (rad/s)/√Hz for gyros, (m/s²)/√Hz for accelerometers.

Angle random walk (ARW)

The angle error you accumulate by integrating gyro white noise, growing with √t. Quoted in °/√h.

Bias instability

The floor of the Allan deviation curve: the best stability the bias ever achieves before it wanders off again.

g-sensitivity

Gyro readings polluted by linear acceleration. Matters on vibrating platforms like multirotors; damp your IMU mount.

pip install allantools numpy matplotlib

# log_imu.py, RealSense flavor: writes t,gx,gy,gz,ax,ay,az rows
import pyrealsense2 as rs, csv
pipe = rs.pipeline(); cfg = rs.config()
cfg.enable_stream(rs.stream.gyro);  cfg.enable_stream(rs.stream.accel)
pipe.start(cfg)
with open('imu_static.csv','w',newline='') as f:
    w = csv.writer(f)
    while True:
        frames = pipe.wait_for_frames()
        for fr in frames:
            m = fr.as_motion_frame()
            d = m.get_motion_data()
            w.writerow([m.get_timestamp()/1000.0, fr.profile.stream_type(), d.x, d.y, d.z])

# allan.py: Allan deviation per axis, straight off the CSV
import numpy as np, allantools, matplotlib.pyplot as plt
data = np.loadtxt('gyro_x.csv')          # one axis, rad/s, at rate fs
fs = 200.0
taus, ad, _, _ = allantools.oadev(data, rate=fs, data_type='freq',
                                  taus='decade')
plt.loglog(taus, ad); plt.xlabel('tau [s]'); plt.ylabel('Allan dev')
plt.savefig('allan_gyro_x.png')

N = ad[np.argmin(np.abs(taus - 1.0))]    # noise density  [rad/s/sqrt(Hz)]
B = ad.min() / 0.664                     # bias instability [rad/s]
print(f'noise density ~ {N:.3e}, bias instability ~ {B:.3e}')

accelerometer_noise_density: 2.0e-03   # [m/s^2/sqrt(Hz)]   from accel tau=1s
accelerometer_random_walk:   3.0e-04   # [m/s^3/sqrt(Hz)]   from accel +1/2 slope
gyroscope_noise_density:     1.7e-04   # [rad/s/sqrt(Hz)]   from gyro tau=1s
gyroscope_random_walk:       2.0e-05   # [rad/s^2/sqrt(Hz)] from gyro +1/2 slope

2.4Cameras

The pinhole model is the whole of projective geometry you need for most of this book. A 3D point in the camera frame maps to a pixel through the intrinsics matrix K:

P = (X, Y, Z) in the camera frame lands at pixel (u, v):

  u = fx · X/Z + cx            ⎡ fx   0  cx ⎤
  v = fy · Y/Z + cy        K = ⎢  0  fy  cy ⎥
                               ⎣  0   0   1 ⎦

fx, fy: focal length in pixels     cx, cy: principal point

Real lenses bend this. Radial distortion (coefficients k1, k2, k3) makes straight lines bow outward or inward toward the image edges; tangential distortion (p1, p2) comes from the lens sitting slightly tilted relative to the sensor. Together these form the Brown-Conrady or "plumb bob" model, the default in OpenCV's calibration API. Wide-angle and fisheye lenses beyond about 120° need a different parameterization entirely; the equidistant (Kannala-Brandt) model, pinhole-equi, maps the incidence angle θ to radius r = f·θ and stays well-behaved out to the edges where radtan falls apart.

Rolling vs Global Shutter

A global shutter exposes every pixel simultaneously: one photo, one instant. A rolling shutter reads the sensor row by row over 10 to 30 ms, so the bottom of the image is a photograph of a slightly later world than the top. Yaw at 1 rad/s with a 30 ms readout and the last row has rotated 1.7° relative to the first; at a 600 px focal length that is roughly 18 pixels of shear on a vertical edge. A feature-based SLAM that assumes a single pose per image is now fitting geometry to a smeared impossibility.

Handhelds and slow indoor robots get away with rolling shutter (and Chapter 5's D435i has a rolling RGB sensor for exactly the cost reasons you would guess). Drones and anything that turns fast should pay the extra $30 for global shutter, or run a rolling-shutter-aware estimator that models per-row time, which costs implementation pain instead. The reading list has the reference.

Depth: Mono, Stereo, RGB-D

A single camera cannot measure metric scale; a hallway and a dollhouse of the same proportions produce identical images. Mono SLAM either accepts an arbitrary scale factor or borrows metricity from an IMU. Two cameras with a known baseline B triangulate depth from disparity d, and the error math is worth having in your pocket:

Z = f · B / d                 (d in pixels, B in meters)
σ_Z ≈ Z² · σ_d / (f · B)      (depth error grows with range squared)

D435i-class rig: B = 0.05 m, f = 640 px, σ_d ≈ 0.25 px
  Z = 2 m → σ_Z ≈ 3 cm      Z = 5 m → σ_Z ≈ 20 cm     Z = 10 m → σ_Z ≈ 0.8 m

That Z² is why a 5 cm baseline device is an indoor sensor, full stop, and why autonomous cars mount stereo pairs a meter apart. Widening the baseline buys far-field accuracy but shrinks the near-field overlap between the two views and makes the rig floppier (a baseline that flexes 0.05° has just eaten your calibration).

RGB-D cameras add a dedicated depth mechanism, and the three technologies fail differently:

Exposure Is Part of the Sensor Model

Auto-exposure and auto-gain change image brightness from frame to frame, and indoor lighting flickers at 50 or 60 Hz on top of that. Feature methods (ORB and friends, Chapter 4) shrug this off because descriptors are built for some illumination invariance. Direct methods like DSO, which minimize raw intensity differences, are hurt badly: a global brightness step between frames looks exactly like the whole world moved. The fixes are to lock exposure and gain when the environment allows, or to calibrate the camera photometrically (response curve and vignette, as in the TUM monoVO dataset) so the estimator can model brightness changes instead of being fooled by them.

2.5LiDAR

Nearly every lidar you will touch is pulsed time-of-flight: fire a nanosecond laser pulse, time the echo, multiply by c/2. The newer alternative, FMCW (frequency-modulated continuous wave, e.g. Aeva), chirps a continuous beam and reads range from the frequency shift, which brings two gifts: per-point radial velocity for free, and immunity to sunlight and to other lidars. As of 2026 FMCW is still mostly an automotive-money technology; robotics budgets buy pulsed ToF.

Mechanically, the market splits three ways. Spinning multi-beam units (Velodyne's original design, now Ouster, Hesai, Robosense) stack 16 to 128 laser channels on a motor: 360° coverage, fixed vertical rings, the geometry every classic algorithm assumes. MEMS and solid-state units steer with a micro-mirror or optical phased array into a forward cone, trading field of view for ruggedness and cost. And Livox units steer with rotating Risley prisms, painting a non-repetitive rosette: the scan pattern never retraces itself, so coverage density accumulates the longer you stare. That last property broke the ring-based feature extractors of the LOAM era and is part of why direct methods like FAST-LIO2 (Chapter 3) took over.

Two features on the datasheet matter more than they look. Multi-return: when the beam clips a dust particle or a leaf edge, part of the energy reflects early and part continues; a multi-return lidar reports both, letting you keep the far (real) surface and flag the near one. Reflectivity (calibrated intensity): a per-point measure of surface reflectance that survives range changes, usable as a grayscale image for place recognition and for lidar-camera calibration later in this chapter.

2.6GNSS, RTK, and the Supporting Cast

Plain GPS is a 2 to 5 meter sensor, and the error budget explains why: satellite clock and ephemeris error contribute roughly 1 to 2 m, ionospheric delay 2 to 5 m (the worst offender, varies with solar mood), troposphere about 0.5 m, receiver noise 0.3 m, and multipath anywhere from zero to many meters depending on how much metal and glass surrounds you. Most of these errors are spatially correlated: two receivers 10 km apart see nearly the same ionosphere.

That correlation is the trick behind RTK (real-time kinematic). A stationary base at a surveyed point measures the shared errors and broadcasts corrections (RTCM messages, often over internet via NTRIP); the rover then works with carrier-phase measurements, resolving the integer wavelength ambiguity to reach 1 cm + 1 ppm accuracy in "fixed" mode. Watch the fix status like a hawk: "float" mode means the ambiguity search has not converged and accuracy silently sags to decimeters. PPK is the same math run offline after the fact, which is how Chapter 8 gets outdoor ground truth without a realtime link. The catch for robots: urban canyons reflect signals, and a multipath-corrupted fix can be wrong by meters while reporting high confidence. Fuse GNSS as one more factor with a robust loss (Chapter 1), never as gospel.

The supporting cast, one line each. Radar: sees through fog, rain, and dust that blind lidar, at the price of sparse, noisy, multipath-prone returns. UWB anchors: 10 to 30 cm indoor positioning from time-of-flight to fixed beacons, great when you own the building, useless when you don't. Event cameras: report per-pixel brightness changes with microsecond latency and no motion blur, spectacular for fast motion and HDR scenes, still exotic on the software side. Magnetometer: absolute heading outdoors, hallucination generator inside any steel building. Barometer: relative altitude to about a meter, drifts with the weather, standard on drones for the vertical axis.

2.7Extrinsic Calibration

An extrinsic is a rigid transform between two frames on the robot: T_body_sensor maps a point expressed in the sensor frame into the body frame, p_body = T_{body_sensor} · p_sensor, as a 4×4 SE(3) matrix (Chapter 1). Half of all extrinsics bugs are convention bugs, so adopt the naming discipline where the frame order is in the variable name and stick to it; if a matrix is called T_cam_imu, it takes IMU-frame points to camera-frame points, and its inverse is T_imu_cam, no exceptions, no "I think this one goes the other way." (Calibration reports annotate exactly this, and people still get it backwards.)

Intrinsics come first, always: you cannot place a camera relative to an IMU while the camera's own focal length is wrong. The standard tool is a planar target. Checkerboards are printable anywhere but must be fully visible in every image and carry a 180° ambiguity; an AprilGrid (a lattice of AprilTags) is better on every axis, because each tag has a unique ID, partial views still contribute corners, and orientation is never ambiguous. OpenCV's calibration module handles the checkerboard workflow end to end; its ArUco module detects AprilTag-style grids when you outgrow checkerboards.

Camera-IMU extrinsics are the hard, load-bearing case for VIO. The reference method (Furgale, Rehder, Siegwart, the Kalibr paper) fits a continuous-time B-spline to the rig trajectory and jointly estimates the spline, the gravity direction, T_cam_imu, and the camera-IMU time offset. Whatever implements it, the physics is the same: it needs motion that excites all axes, and a calibration log recorded by someone gently panning left and right is unobservable, so the tool will happily output garbage with small-looking residuals. The worked setup below gets you the practical 90% (intrinsics, time offset, measured extrinsics) in pure Python.

LiDAR-camera calibration has mostly gone targetless: modern methods align depth edges and reflectivity gradients in the point cloud against intensity edges in the image, refined with mutual information or direct photometric cost. A practical, maintained implementation is koide3/direct_visual_lidar_calibration, which gets you within a degree from a single scene plus a rough initial guess.

When one of the two devices gives you poses rather than raw overlapping data (a tracking camera, a robot arm flange, a motion capture rig), the tool is hand-eye calibration: collect paired relative motions A (device 1) and B (device 2), then solve AX = XB for the fixed transform X between them. Tsai-Lenz (1989) is the classical closed-form solution; every robotics library ships one. The requirement that trips people is rotation diversity: motions about at least two independent axes, or the translation component of X is unobservable.

How Good Is Good

Numbers to hold your work against. Camera intrinsics: reprojection RMS below 0.5 px, below 0.3 px with a good target and lighting; above 1 px means bad corners, a flexing target, or the wrong distortion model. Joint camera-IMU calibration: mean reprojection error under 1 px, gyro and accel residuals consistent with your section 2.3 noise numbers, time offset stable across two independent runs to well under a millisecond. And the sanity number that makes rotation errors visceral: 0.5° of extrinsic rotation error displaces a point at 10 m range by 8.7 cm. Your map at range is only as good as your angles.

pip install opencv-python numpy scipy

# intrinsics.py: 30-50 views of the board, whole field of view,
# near and far, tilted views included
import cv2, glob, numpy as np
CB = (9, 6); SQ = 0.024                    # inner corners, measured meters
objp = np.zeros((CB[0]*CB[1], 3), np.float32)
objp[:, :2] = np.mgrid[0:CB[0], 0:CB[1]].T.reshape(-1, 2) * SQ
obj, img = [], []
for path in glob.glob('views/*.png'):
    gray = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
    ok, corners = cv2.findChessboardCorners(gray, CB)
    if not ok: continue
    corners = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1),
        (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-3))
    obj.append(objp); img.append(corners)
rms, K, dist, rvecs, tvecs = cv2.calibrateCamera(obj, img, gray.shape[::-1], None, None)
print('reprojection RMS [px]:', rms); print('K =', K); print('dist =', dist.ravel())

# timeoffset.py: gyro magnitude vs image-derived rotation rate
import numpy as np
from scipy.signal import correlate
# cam_rate[t]: per-frame rotation angle from solvePnP on the board,
#              divided by frame dt   (len N, at camera rate)
# gyr_rate[t]: |gyro| resampled onto the same uniform time grid
c = correlate(cam_rate - cam_rate.mean(), gyr_rate - gyr_rate.mean(), 'full')
lag = c.argmax() - (len(gyr_rate) - 1)
print(f'camera lags IMU by {lag * dt * 1000:.1f} ms')

2.8Time Is a Sensor Too

Fusion means combining measurements taken at the same instant, so a clock error is a measurement error in disguise. The arithmetic is short and worth committing to memory:

pixel error ≈ fx · ω · Δt_sync
            = 600 px · 1 rad/s · 0.005 s  ≈  3 px

Five milliseconds of camera-IMU misalignment during a 1 rad/s turn shifts every feature by about 3 pixels, on a system trying to converge to 0.2 px reprojection error. The estimator cannot tell this from a bad extrinsic, so it corrupts whatever state absorbs it. This is the single most common reason a VIO stack that works on EuRoC (hardware-synced, Chapter 8) falls over on your own USB rig.

The first discipline is where the timestamp is taken. A frame should carry the time its exposure happened (mid-exposure, by convention), stamped by the driver from the device's own reported capture time. What you often get instead is the time the image finally arrived through USB and got a header slapped on, which floats by 5 to 30 ms with bus load and CPU mood. Jitter this size makes a mockery of the 3-pixel arithmetic above.

The gold standard is hardware. The IMU (or a dedicated board) generates a trigger pulse that causes each exposure, and the camera's strobe output confirms it, so the correspondence is physical rather than statistical. GNSS receivers provide PPS, a pulse at the top of every second, to which lidars and computers discipline their clocks. Between machines, PTP (IEEE 1588, or gPTP/802.1AS in its automotive profile) achieves sub-microsecond sync on hardware-timestamping NICs, versus the milliseconds of ordinary NTP. Milliseconds are fine for logging; they are not fine for fusion.

In software, pairing streams means matching measurements whose stamps are close (nearest-neighbor association with a tolerance, which is a five-line function over two sorted arrays). Understand what such pairing does and does not do: it selects plausible pairs, it does not correct stamps. Feeding it arrival-time-stamped garbage yields confidently matched garbage. Estimator-side time offset estimation (offline cross-correlation as in the 2.7 setup, or online estimation inside the VIO) can absorb one constant offset, but nothing downstream can absorb jitter.

PPS

Pulse per second. An electrical edge at each second boundary, usually from a GNSS receiver, used to discipline other clocks.

PTP / gPTP

Precision Time Protocol over Ethernet. Sub-microsecond clock agreement with hardware timestamping; gPTP is the 802.1AS profile.

Strobe

Camera output pin that fires during exposure. Wire it to a timestamping input and you know exactly when each frame happened.

Mid-exposure stamp

The convention that a frame's time is the middle of its exposure window, not its start, end, or arrival.

2.9Picking a Sensor Suite

Sensor selection is a budget exercise against your failure modes, not a shopping spree. Three worked bills of materials, with rough 2026 prices, each chosen to feed specific chapters of this book.

Budget Indoor Robot, About $700

This is deliberately Chapter 5's rig. Its weaknesses are known and survivable indoors: rolling-shutter RGB (keep turns gentle), 5 cm stereo baseline (useless past ~5 m, fine in rooms), USB timestamps (calibrate the offset as in the 2.7 setup and keep motion moderate).

Outdoor UGV, About $8k

The single biggest line item is honest: outdoor range costs money. If the budget hurts, a Livox Mid-360 at $750 replaces the Ouster and saves five thousand dollars at the cost of range and a rosette pattern your software stack must support (FAST-LIO2 does).

Sub-250 g Drone, About $450 in Sensors

Here weight is the currency, not dollars. No lidar makes the cut at this mass, so this platform lives entirely on Chapter 4's visual-inertial methods, and every gram of sensor must justify itself against flight time.

2.10Reading List

2.11Interview Questions