How would you build a segmentation pipeline for crop health monitoring, using camera imagery and the latest models?

1.1Start From the Sensor, Not the Model

A gimbal-mounted RGB camera alone can classify visible necrosis or wilting, but most stress signals — early water deficit, nutrient deficiency — show up in the near-infrared before they're visible to the eye. Fly RGB plus a modified NIR or dedicated multispectral sensor at a controlled altitude and ground-sample distance, then compute vegetation indices (NDVI, NDRE) alongside the RGB channels as additional input planes rather than as a separate downstream product.

1.2The Pipeline, Stage by Stage

For labeling, don't hand-annotate pixel masks from scratch. SAM2 gives strong zero-shot object masks from a point or box prompt, and running it over every tile turns "draw a mask" into "confirm or correct a proposed mask" — a large annotation-cost cut, especially for the long tail of disease presentations a fixed label set won't cover well. Vegetation indices double as a second, free source of weak labels for stress regions, bootstrapping the first training round before any human labeling happens at all.

1.3Choosing a Model

The trade-off is label efficiency vs. deployability, and the choice comes down to which one is actually scarce:

In practice this isn't either/or: SAM2 and DINOv2 do the heavy lifting during dataset construction and early-pilot iteration, and the model that actually ships to the field is the lightest one that hits the accuracy bar — usually a DeepLabV3+ or U-Net variant with an efficient backbone, distilled or fine-tuned from the foundation-model-labeled dataset, because that's the one that fits the edge power and thermal budget for in-field scouting rather than requiring a batch upload for offline processing.

1.4Turning a Mask Into a Grower-Facing Answer

After inference, per-plant or per-row aggregation matters more than raw pixel accuracy for the output that actually gets used: connected-component analysis over the mask, reprojected onto the field ortho-mosaic with GPS metadata, turns a pixel mask into an actionable map — "these three rows are stressed" — rather than a segmentation image nobody in the field will look at directly.

1.5The RGB-Only Branch: No Multispectral Sensor at All

Most real deployments — a phone in a grower's hand, a fixed-wing without a NIR payload — only have a standard RGB camera. You lose NDVI/NDRE's pre-visible stress signal entirely, but a full pipeline is still viable once it's built specifically around three channels instead of assuming a fourth.

Camera-specific preprocessing

Field lighting, not resolution, is what breaks an RGB-only pipeline first. Convert RGB to HSV or Lab before segmenting — HSV isolates brightness into its own channel (V), so shadow and harsh-sun variation stop leaking into the hue/saturation signal the vegetation mask actually depends on. Apply CLAHE (Contrast Limited Adaptive Histogram Equalization) to recover detail in overexposed leaf faces and under-canopy shadow without the blown-out look global histogram equalization gives. Pull focal length, aperture and sensor size from EXIF metadata to back out a millimeter-per-pixel scale at the known capture distance — without it, "lesion covers 40 pixels" has no physical meaning across photos taken from different heights or different phones.

Two-stage topology: isolate, then localize

Loss function matters more than usual because the target class is small — a rust fleck can be a few dozen pixels on a leaf that's thousands. Combine Dice Loss (rewards mask overlap directly, robust to class imbalance) with Focal Loss (down-weights easy, already-confident background pixels so gradient stays on the hard lesion boundary). Plain cross-entropy on this class ratio mostly learns to predict "healthy" everywhere and still scores deceptively well on pixel accuracy.

Severity, correctly oriented

Lesion counting via connected-component labeling on the disease mask is a second, independent signal: five small, separate lesions can mean an early, spreading infection worth flagging urgently even at a severity percentage that looks mild — severity percentage and lesion count diverge in a way that matters agronomically.

Field noise, RGB-only indices, and quantization

Leaves move in wind, lighting shifts constantly, phones auto-adjust exposure mid-shot. Train with heavy color jitter, synthetic glare/flare, and background substitution (swap the soil/weed background behind the leaf mask) so the model doesn't quietly learn "this particular soil color = healthy." Even without NIR, ExG = 2G − R − B (Excess Green) is a cheap, RGB-only vegetation index — feed it in as an explicit fourth input channel alongside R/G/B rather than relying on the network to rediscover the same relationship from raw color. For offline field use, export to ONNX / CoreML / TFLite with FP16 or INT8 quantization — same edge-deployment reasoning as the Jetson/RB5 constraints elsewhere in this book: target hardware's power and thermal budget decides precision, not model accuracy alone.

import torch
from ultralytics import YOLO

leaf_model = YOLO("leaf_seg.pt")          # Stage A: plant/leaf mask
lesion_model = torch.jit.load("lesion_seg.pt").eval()  # Stage B: SegFormer/DeepLabV3+

def exg(img):  # img: float tensor, C,H,W in [0,1], channels R,G,B
    r, g, b = img[0], img[1], img[2]
    return (2 * g - r - b).clamp(-1, 1)

def score_leaf(rgb_frame):
    leaf_mask = leaf_model(rgb_frame)[0].masks.data[0]        # Stage A
    x0, y0, x1, y1 = bbox_from_mask(leaf_mask)
    roi = rgb_frame[:, y0:y1, x0:x1]
    roi_4ch = torch.cat([roi, exg(roi).unsqueeze(0)], dim=0)  # RGB + ExG

    with torch.no_grad():
        lesion_mask = lesion_model(roi_4ch.unsqueeze(0))[0].argmax(0)  # Stage B

    leaf_px = leaf_mask.sum().item()
    lesion_px = lesion_mask.sum().item()
    severity_pct = 100.0 * lesion_px / max(leaf_px, 1)
    lesion_count = count_connected_components(lesion_mask)
    return severity_pct, lesion_count