Raspberry Pi 5 + Hailo-8L  ·  27 FPS  ·  everything onboard

Seeing to Drive on a Single Board

An attention-free student of Depth Anything V2 that runs on a 13-TOPS integer NPU with int8 equal to float, and the onboard obstacle avoidance, global planning and SLAM it makes possible on a small rover.

Itai Mizlish and Boaz Ben-Moshe  ·  Ariel University  ·  DANIEL: Depth Anything Navigation In Exteriors, Learned  ·  2026

27.1 FPSdepth at 288×512 on the Hailo-8L, batch 1, measured on the Pi
r = 0.99993int8 on the chip vs the same network in float, 500 frames
0.269AbsRel on held-out simulated ground truth (next best on the chip 0.354); on real stereo-verified points, not resolvably different from the zoo DAv2-S
8.2 ± 0.4 / 10scenes reached with no contact (five seeds), with the real Pi + Hailo in the real-time loop

Foundation depth does not reach a small NPU through better quantisation.

The Hailo-8L runs attention in int8, and diffuse attention at a few hundred tokens falls below one quantisation step: our build of Depth Anything V2 and the vendor's both erase obstacles. We distil the 335 M-parameter teacher into a 24.5 M-parameter network with no attention at all, range obstacles from where they meet the floor, and run mapping, hybrid-A* planning and SLAM on the Pi itself.

Demos

The real board, in the loop and on real video.

4K source · every frame on the Hailo-8L

Outdoors: the deployed student, frame by frame on the NPU

A low camera crossing a field toward rocks and posts. Every frame went through the student v5 HEF on the Raspberry Pi 5's Hailo-8L (34.6 ms per frame on the NPU) and was read back; the camera half is the 4K original. Frames from this video (every 10th) were in the student's training set, so it is a demonstration, not a held-out result. Download the 4K version.

Real Pi 5 + Hailo-8L in the loop

The deployed stack drives on its own SLAM pose

Isaac Sim streams the car's camera to the real Pi; the Pi computes depth on the Hailo, maps, plans and replies with steering and throttle, applied only when they arrive. The six scenes one recorded pass completed with no contact, each uncut; that pass scored 6 clean and 7 reached of 10, the five-seed mean is 8.2.

Real robot camera

Depth, point cloud and obstacle scan from one camera

Real robot video with depth computed on the Hailo-8L (student v3 HEF, every frame read back from the chip), the metric point cloud scaled by the floor, and the ground-contact scan with the chosen arc.

All ten scenes

A full real-board pass, failures included

Every scene of the same pass, labelled with its outcome on the car's real footprint: chairs, a cone slalom, a barrel gate, people, mixed clutter and a crate wall, with timeouts where they happened.

Real robot camera frame Depth computed on the Hailo-8L for the same frame
cameraHailo-8L depth
Drag

Camera ↔ depth from the chip

Held-out real frames; depth from the deployed student v5, read back from the Hailo-8L.

Method

Three ideas, each measured on the chip.

01

Why transformer depth breaks on the Hailo-8L

At the chip's 449 tokens, a diffuse softmax weight is about 1/449 ≈ 0.0022, below one uniform int8 step of 1/255 ≈ 0.0039. Measured in float, 91 % of attention weights fall below that step and carry 17–33 % of each query's attention mass. Re-calibration and quantisation fine-tuning do not fix it; removing attention does.

Per-block share of attention weights below one int8 step and the attention mass they carry
02

An attention-free student

A ResNet-34 encoder and a U-Net decoder of convolution, batch norm, ReLU and bilinear upsampling: every operation quantises cleanly. 24.5 M parameters (DAv2-S: 24.8 M), distilled from DAv2-Large with a scale- and shift-invariant loss, then fine-tuned on rendered ground truth. On the chip: 35.1 ms, int8 = float at r 0.99993.

ResNet-34 encoder U-Net decoder
03

Metric space without metric depth

Relative depth has no scale. But where an obstacle meets the floor, the image row alone gives its distance from the camera height. The network only has to say where each column's floor ends; the columns become a 69°, 4 m laser scan, and the floor profile becomes a ruler for a metric point cloud.

Z = fy · h / (vc − cy)

Ground-contact scan hits on a held-out frame and in bird's-eye view against the true prop footprints
  1. cameraRGB frame30 Hz
  2. Hailo-8L NPUStudent depth35.0 ms, int8
  3. Pi CPUGround-contact scan + cloudin parallel with the NPU
  4. Pi CPUHit-count map + SLAMground VO, encoder, gyro
  5. Pi workerHybrid A* + safety cascade110 ms median
  6. Pi CPUPure pursuit → PWMESC brake gate

Results

Every depth network we could run on the Hailo-8L.

400 held-out simulated frames with the renderer's metric depth and 100 held-out real robot frames. Every Hailo model runs on the chip and is read back; per-frame least-squares scale and shift in disparity. Obstacle pixels are those that land on a prop's collider, which removes the renderer's floor speckle.

Accuracy against measured on-chip speed

Hover or focus a point. Dashed lines: float GPU models that cannot run on the Pi at this rate.

ModelRuns onAbsRel allAbsRel obst.δ1r (real)Speed
Depth on real robot frames computed on the Hailo-8L by every model, against the float teacher
Real robot frames, depth as computed on the Hailo-8L (except the GPU teacher). The int8 transformer builds lose the scene; the student keeps the teacher's structure at 27 FPS. Dark areas lie outside a model's input crop.

Closing the loop: a real-time error budget.

Ten frozen scenes in a Gaussian-splat reconstruction of a real yard, eleven obstacle types with honest colliders, at least five seeds per arm, contacts scored on the car's real 0.557 × 0.294 m footprint. Clean goals of 10:

Oracle: exact props, lockstep
9.8
Real time, exact obstacles
9.2
Hailo depth, simulator pose
7.9 ± 1.1
Hailo depth, own SLAM pose
7.7 ± 0.7
Own pose, real Pi + Hailo
8.2 ± 0.4
Camera-only odometry
2.6

Execution costs about 0.6, perception 1.3, self-localisation 0.2 clean goals (0.9 goals reached). Ten seeds for the deployed arms, five for the others; the own-pose arms use a simulated wheel encoder and gyro.

Clean success per scene for the main configurations
Per scene: the misses concentrate on the cone slalom, the gate, the chair cluster, the crate wall and the person-and-cones scene.
Every hybrid-A* plan in one episode with the SLAM estimate against the truth
Onboard SLAM and planning on the robot's own pose: every plan the Pi made in one slalom episode. The car reached the goal despite 0.47 m peak drift.
A real robot frame lifted to a metric point cloud on the Pi
One real frame to metric 3D on the Pi: chip depth, scale from the floor profile, flying pixels removed.
Per-frame stage timing on the Pi and hybrid-A* search times
The budget on the Pi: NPU 35 ms overlapped with 47–49 ms of CPU; 16.5 Hz on 720p video with SLAM, 27 Hz without; 28 h of real-board runs without throttling.

Honestly

What is not solved yet.

Resources

Everything to reproduce the numbers.

Real world · new

One real drive: our chip network is not resolvably different from the vendor's depth model.

One drive by hand through a sunlit park, recording everything: the colour camera, the RealSense D435's on-chip stereo depth and both of its infrared images, and the depth our student computed on the Hailo-8L, frame by frame (219 s). There is no ground truth on this drive, so every real number below is agreement with an independent reference; questions that need ground truth go to the hardware-in-the-loop simulator.

DepthObstaclesSLAMSimulator

Four key frames of the video: the drive, the obstacle cells from above, SLAM against the reference, the end card
Both panels follow the drive from the same reference poses (RTAB-Map's position and heading), behind the car's real 3-D model, so only the depth differs (the drive at 8× speed). About 143 s in, the video pauses for 5 s on the drive's largest straight-ahead disagreement between the onboard scans (+41 % on that frame, +23 % over its 0.4 s; median +8 % over the 157 frames with a flag straight ahead): at a bench 1.00 m away by the stereo's metric depth, the stereo scan reads 1.25 m mostly on its seat (13 of 19 rays, overhang rule), the network's 1.77 m on the ground (vertical-face rule). The four stills above show the key frames; watch the video full-screen in landscape.
How to read the videoBoth the 3-D panels and the top-down obstacle comparison place the clouds with the camera mount measured from the stereo floor (3.8° down). Each map is that source's own depth turned into a colour point cloud (left: the RealSense stereo; right: our network, at the scale the car itself uses), growing as the car drives; where the depths disagree, the maps differ. The cone ahead of the car is its camera's view with the live frame on it; cyan lines are this frame's pseudo-lidar rays (the deployed geometry). The line under the maps is the median range gap between the two scans for obstacles beyond 1 m; the timeline carries each frame's gap. The paused ranges are scan ranges in that deployed geometry. People are blurred, and kept out of the point clouds.
  • Depth: on 1,000 points where three stereo matchers agree, ours scores AbsRel 0.262 and the Hailo Model Zoo's Depth Anything V2-S 0.234: not resolvably different (95 % CI −0.060 to +0.118).
  • Obstacles: the simulator's advantage is not shown (an exploratory lean: after correction p = 0.046 or 0.14, depending on which tests are grouped; details below).
  • Range: beyond 1 m, where the onboard rule flags an obstacle base in each depth, the network's flagged range is within a median 25 % of the RealSense's (signed −7 %: a little nearer); most flags sit on grass within 5 cm of the fitted floor, not raised obstacles.
  • SLAM: VGGT-SLAM 2.0 agrees with the independent reference (skill +0.80); ours +0.10, the commands alone +0.08 (same scale-fitting alignment; one run: a 0.02 gap, not a lead). As deployed, depth barely enters our trajectory.
  • With ground truth (simulator): with exact depth our SLAM needs a better scale than its per-frame floor fit: for exact depth on one simulated loop, the true scale cuts the error 6.8× after loop closure (6.3–7.4× per session), and 6.5× (95 % CI 5.5–7.7) on seven further start seeds; smoothing the floor fit does not reliably help; for our network a better scale is not enough: no affine scale tried closes the gap.

Depth on 1,000 stereo-verified points

Reference: three different stereo matchers on the same infrared pair must agree within 2 % + 1 % per metre, between 0.3 and 3 m. 1,000 such points, stratified across five distance bins from 309 frames, are frozen once; every monocular model is scored on exactly these points, relative models with their scale (and shift) fitted on other points of the same frame. Stereo rows are agreement among matchers, not accuracy (method details below).

Depth sourceRuns onAbsRel · per frameδ1AbsRel · one scale per driveRaw metric: pred / true
Depth Anything 3-Large · GPUGPU0.1010.900.418–
Depth Anything 3-Mono-Large · GPUGPU0.1070.880.359–
Depth Anything 3-Base · GPUGPU0.1100.870.408–
Depth Anything 3-Small · GPUGPU0.1150.860.390–
Depth Anything V2-S · GPUGPU0.1600.820.427–
Depth Anything V2-L (teacher) · GPUGPU0.1700.840.422–
Depth Anything V2-B · GPUGPU0.1810.830.426–
Depth Pro · GPUGPU0.2020.650.402×2.0
MoGe-3 · GPUGPU0.2050.620.261×3.5
MoGe-2 · GPUGPU0.2070.630.262×3.3
Depth Anything 3-Metric-Large · GPUGPU0.2220.580.297×2.4
UniDepthV2-L · GPUGPU0.2220.610.305×1.7
DAv2-S, Hailo Model Zoo (the comparison) · chipHailo-8L int80.2340.740.432–
Ours, replayed on the chipHailo-8L int80.2580.740.413–
Ours, float twin · GPUGPU0.2580.750.410–
Ours, live on the chip (distilled student v5)Hailo-8L int80.2620.730.410–
SC-DepthV3, Hailo Model Zoo · chipHailo-8L int80.3920.220.435–
FastDepth, Hailo Model Zoo · chipHailo-8L int80.3940.230.415–
DAv2-S, our int8 build · chipHailo-8L int80.4430.580.439–
Stereo, below: each matcher scored leave-one-out against the other two, on its own points in the infrared frame, holes excluded. Agreement among matchers on the same images, not comparable with the rows above.
RealSense D435 on-chip stereo (no depth on 5% of points)stereo0.0091.00–metric
OpenCV SGBM on the IR pair (no depth on 17% of points)stereo0.0101.00–metric
Fast-FoundationStereo on the IR pair (no depth on 0% of points)stereo0.0400.97–metric
  • Ours and the vendor's model are not resolvably different on these mostly-ground points; in relative terms one drive cannot tell whether the simulator's advantage holds.
  • On obstacles the simulator's advantage is not shown: an exploratory lean that reverses with one scale per drive.
  • Ours beats the other three edge builds and trails its GPU teacher.
  • Metric foundation models misjudge scale from a camera 16 cm above the ground.
The numbers behind these points: intervals, tests, the simulator comparison
  • Ours and the vendor's model are not resolvably different on these mostly-ground points (paired over frames +0.017, against 0.028 between the table's point means; 95 % CI −0.060 to +0.118, p = 0.70; with 10-s blocks of the drive as the unit, from inverting the exact sign-flip test, −0.040 to +0.075; on |log(predicted/true)| +0.009, CI −0.012 to +0.032). The interval bounds any advantage of ours at 0.060 (0.040 with blocks), against the simulator's 0.084 (95 % CI 0.060–0.113 over its 104 runs; paired over the 397 frames both scored, 0.270 vs 0.354, all-pixel AbsRel (the headline 0.269 is over all 399 frames) on held-out simulated frames of the yard the student was fine-tuned on, a different metric), so the block bound lies below that interval; in relative terms the real bound (23 %) overlaps the simulator's 18–30 %, but the real estimate is −8 % (95 % CI −30 to +23 %): one drive cannot decide; on the set rebuilt without Fast-FoundationStereo the absolute bound does not hold either (ours −0.061, block p = 0.078, carried by the zoo model's blow-ups; on |log| not resolved, CI −0.030 to +0.017).
  • On obstacles the simulator's advantage is not shown. In simulation ours led on obstacle pixels within 3 m (AbsRel 0.197 vs 0.334; 213 vs 193 frames, unpaired). On points at least 0.15 m above each frame's floor plane (trunks, benches, bushes, people; 319 points, 87 frames; a stratum added after the main result, at a review's request), scored on |log(predicted/true)| because a few blow-ups rule the mean there, ours leans ahead of the zoo model by −0.165 (frames as units p = 0.012; with 10-s blocks exact p = 0.046, which is 0.14 corrected over the five tests on this metric and 0.046 over the three competitor tests, so the stratum is exploratory), and by −0.123 on the D435 + SGBM consensus (blocks p = 0.055). The median error barely differs (0.232 vs 0.240; 0.369 vs 0.346 on the D435 + SGBM set; untested), and with one scale per drive ours is worse on the D435 + SGBM obstacle set (+0.334, blocks p = 0.040; 0.058 corrected, equally exploratory). Depth Anything 3-L is better than ours on obstacles by 0.184 in log error, V2-L by 0.153 (not on the two-matcher set); MoGe-3 and Depth Pro lead by 0.21 and 0.23 in AbsRel, not resolvably with blocks.
  • Against the rest. Ours beats the other three edge builds (our int8 DAv2-S, SC-DepthV3, FastDepth; block p ≤ 0.014, the last two get a median scale only) and trails its GPU teacher by 0.091 (paired 0.085, CI 0.047–0.141). Aligned per frame, the GPU metric models score 0.20–0.22, better than ours but not resolvably. Live on the car it is 0.004 worse than its own chip replay (CI 0.001–0.006).
  • Metric foundation models misjudge scale from 16 cm. Raw, all overestimate distance, by 1.7× (UniDepthV2) to 3.5× (MoGe-3), plausibly because they were trained on human-height cameras. With one scale for the whole drive, MoGe-3 and MoGe-2 (0.26) have a lower error than every relative network (0.36–0.44; not paired-tested); ours is at 0.410, against the zoo DAv2-S −0.040 paired per frame (CI −0.106 to +0.061; the table's means weight points, not frames). Every relative Depth Anything network but our int8 DAv2-S build loses 0.15–0.32 under one scale per drive (ours 0.15): their output scale moves from frame to frame, so our robot takes a per-frame scale from the camera's known height, which this benchmark does not score.

Obstacles and range: the same poses, two depths

  • Beyond 1 m, where the onboard rule flags an obstacle base, the network's flagged range is within a median 25 % of the stereo's, a little nearer (signed −7 %); 85 % of the stereo's flags sit within 5 cm of its fitted floor (the grass surface), so this compares where the rule fires, not raised obstacles.
  • Seen from above, the two depths mark mostly different obstacle cells: 16 % of the cells either marks are marked by both (network 77 %, stereo 39 %).
The numbers behind these points: rays, cells, calibration
  • Range from the same poses. Where the onboard rule flags an obstacle base in both depths beyond 1 m, the network's range is within 25 % of the stereo's (median over 931 frames of each frame's ray-by-ray gap; 29 % pooled over 28,616 rays); the median signed gap is −7 %, the network's flags a little nearer. An exploratory diagnostic read the scan's detectors and classified each flagged run by the full-resolution stereo at its pixels: of the 26,142 stereo flags the gap uses, 85 % lie under 5 cm above the stereo's fitted floor, 1 % on holes, 7 % on raised points (the rest from the overhang detector). Both ranges come from ground-contact geometry and the camera height, with the onboard stack's nominal intrinsics; re-scored with the factory principal point (5.6 px lower) the gap is 26 % over 982 frames (signed −13 %), so its size does not come from the nominal intrinsics. The mount, measured from the stereo floor, is pitched 3.8° down and rolled −1.4° at 0.168 m (the onboard geometry assumes 0°, 0°, 0.158 m); imposed on the onboard geometry the gap stays at 26 %, but the stereo then marks 97 % of image columns blocked (63 % as deployed), so this is a sensitivity check, not a calibrated range error. Inside 1 m both mark grass as obstacles, in different places: the stereo marks 63 % of image columns blocked, the network 41 %. Neither is right on grass, which this car drives straight over.
  • Free of the rule, pixel by pixel at the measured horizon (post hoc): the network places raised structure nearer than the stereo beyond 1.5 m. This cannot separate the network's depth from its floor-fit scale (in the simulator, exact depth's floor fit erred the other way).
    stereo distancedist.raised ≥ 5 cm, network / stereo (95 % CI)raised, net ÷ stereoblocksground
    0.3–0.6 m1.21 (1.01–2.15)80.99 (0.96–1.00)
    0.6–1.0 m0.97 (0.82–1.25)110.91 (0.85–0.97)
    1.0–1.5 m1.08 (0.77–1.45)90.87 (0.78–0.95)
    1.5–2.0 m0.78 (0.66–0.92)130.80 (0.68–0.89)
    2.0–3.0 m0.69 (0.57–0.79)160.73 (0.59–0.86)

    Median of per-frame medians, 10-s block bootstrap; bold where the interval excludes 1, grey where it does not (per cell, no multiplicity correction); intervals on the desktop page.

  • What the ground-flagging costs (post hoc). Replayed open loop along the drive with the same measured mount, the planner stops on 37 % (23–50 %) of frames with the network's depth and 83 % (75–90 %) with the stereo's (34 % as deployed; 95 % block CIs); ground- and hole-classed flags are implicated in at most 84–87 % of those stops (an upper bound: columns whose first flag is ground are emptied, not re-scanned). Decision counts, not a safety measure.
  • A stricter floor rule does not help (pre-registered). Of 11 stricter variants (floor threshold up to 15 cm, run length up to 12 rows), none passes its simulator detection gate: each loses at least 22 percentage points of detection recall on some approach to a cone, box or barrel. On the drive's first half (network depth, measured mount), the variants up to 10 cm cut ground- or hole-classed flags from 74 % to 57–72 % but not the stops (26–27 % against 27 %); the 15 cm variants reach the camera height and all but switch off the height-band detector.
  • Nor does a stricter vertical-face test (pre-registered). All 12 settings (slope threshold 0.10⁠–⁠0.30, run 16⁠–⁠32 rows) have identical simulator recall, so the gate cannot choose. On the held‑out second half, the one selected on the first half (slope 0.10, 32 rows) cuts the stops from 47 % to 17 % (−30 percentage points, 95 % CI −53 to −9) but keeps only 78 % (CI 62–87 %) of the stereo-verified raised flags, below the pre-set 80 %: it fails validation. The simulated navigation suite did not reject it (+0.6 contact‑free goals per seed, 95 % CI −1.1 to +2.3, 5 seeds), but only in the restored simulator, where the released code itself scores 1.4 goals per seed below the published suite (under investigation).
  • Seen from above. Of the 10 cm cells in which either map holds a point at least 0.15 m above the ground, the stereo marks 39 % and the network 77 % (both 16 %): the network finds 40 % of the stereo's cells, and 20 % of its own are also the stereo's. Both clouds are placed with the measured camera mount (the stereo by its floor normal, the network at the measured horizon and height); a first version assumed a level camera, so the stereo ground rose into the obstacle band and the shares read stereo 86 %, network 39 %, both 25 % (retracted; an intermediate fix that corrected only the stereo read 49 / 70 / 19 %, stereo / network / both). Descriptive, not a comparison of obstacle detection: neither map is ground truth, and the network flags grass bands near the path.

stereo 39 % · network 77 %

marked by both: 16 %

stereo only: 23 %

network only: 61 %

the drive

10 cm cells holding a point ≥ 0.15 m above the ground, from the same reference poses, both clouds placed with the measured camera mount. Descriptive: the stereo cells are not ground truth.

Top-down map of the park: obstacle cells marked by the stereo only, the network only, or both

Where each method thinks the car went

The yardstick (white) is an independent metric system: RTAB-Map on the RealSense's own infrared image and stereo depth, one graph over 97 % of the recorded frames (four loop closures, 47 odometry resets). Skill = 1 − RMS / RMS of standing still: agreement, not accuracy, from one drive (†: monocular, its scale given by the alignment). Click a method to show or hide it.

Hover a track: dots mark every method's position at that same moment.

MethodTrackedSkill, scale fittedSkill, rigidRMS vs referencePath / reference
  • As deployed, depth barely enters our trajectory: the gaps between our arms are the same odometry on different frame sets.
  • Independent systems agree with the reference; several learned baselines lose tracking on sunlit grass.
  • Loop closure needs the right drive: places passed twice in the same direction.
The numbers behind these points: reference checks, gates, runs
  • How skill is scored. Methods run three times show mean ± sd. Monocular methods get a similarity alignment, under which skill cannot fall below 0; scored the same way, ours as deployed reaches +0.10 and the commands alone +0.08. The reference's three re-runs differ from it by 0.64–0.79 m RMS and from each other by 0.33–0.49 m; scored against any of the four runs, no skill moves by more than 0.06 and no ranking among full-coverage methods changes. Skill on a partial overlap does not rank against skill over the whole drive.
  • As deployed, depth barely enters our trajectory. Our stack measured the drive 2.3× too short: a plausibility gate in its camera odometry rejected any motion faster than 1.08 m/s (1.5× the simulated car's 0.72 m/s top speed), throwing away 23 % of frames, exactly the fast ones (the odometry's own 90th-percentile speed on this drive is 2.9 m/s). A physical bound fixes the scale (path within 8 % of RTAB-Map's, the only check of the reference's scale that shares none of its inputs) but not the shape: skill falls to −0.38…−0.62, worse than the commands alone (−0.31). The camera's uncorrected mount (measured 3.8° down, roll −1.4°) is the prime suspect, not yet isolated. With either gate, switching the scan match off changes neither depth arm by more than 0.015 (deployed gate: network −0.097 vs −0.111, RealSense +0.043 vs +0.029; physical gate: −0.615 vs −0.618, −0.381 vs −0.382), so the gaps between the arms are the same odometry on different frame sets, not the depth: this drive cannot rank the two depths inside our SLAM. Our replay is deterministic (three runs, identical scored tracks); these are single-drive numbers. The commands' own path comes from a speed model calibrated in the simulator (0.27× the reference), and with the physical gate our stack trails them under the scale-fitting alignment too (+0.01 and +0.02 vs +0.08). A gyro heading from an IMU is the planned, untested fix.
  • Independent systems agree with the reference. VGGT-SLAM 2.0 (2026, monocular) scores +0.80 (bit-identical over three runs) over 99 % of the drive (117 s in keyframe gaps filled by interpolation; +0.79 scored at its own keyframes only); Glob3R +0.72. ORB-SLAM3 on the RealSense stereo pair scores +0.879 ± 0.013 over three runs, but only on the 13–19 % of the drive it tracks, and it shares the reference's images. DROID-SLAM scores +0.27 ± 0.19 over three runs, so it does not rank against DPV-SLAM (+0.39) or DPVO (+0.29). Monocular ORB-SLAM3 and MASt3R-SLAM lose tracking within seconds on sunlit grass.
  • Loop closure needs the right drive. Of 205 places this car passed twice, 178 were driven the opposite way; a forward 69° camera sees different things each way. As deployed ours closes none (one with the physical gate). The next recordings drive one route twice in the same direction from a taped mark, with AprilTags for true positions.

Scale matters most (simulator, one loop): exact depth with the true scale 0.074 m; camera arms without it vs 0.42–0.84 m

In the hardware-in-the-loop simulator the car drove a loop twice under three lighting presets, carrying a simulated RealSense infrared pair (SGBM), a planar lidar ring and exact depth. Every source went through our SLAM; trajectory error (ATE) is measured against the true poses. Our student is in-distribution here: it was fine-tuned on exact depth rendered in this same yard.

Front end and scanexact depthSGBM stereoours, floatours, Hailo chip
Commands only, no scan matching (noon)0.107
Lidar ring as the scan (consistency check)0.083 ± 0.013
Deployed front end: ground visual odometry0.671 ± 0.0030.543 ± 0.0780.532 ± 0.0620.561 ± 0.077
Deployed pseudo-scan (floor contact)0.844 ± 0.0340.813 ± 0.1070.650 ± 0.3270.489 ± 0.278
Horizontal-plane scan, per-frame floor-fit scale0.501 ± 0.0690.356, 0.609†0.433 ± 0.0900.523 ± 0.166
… the floor-fit scale smoothed over ~20 frames0.415 ± 0.1290.827, 0.612†0.732 ± 0.0570.469 ± 0.228
… the true metric scale0.074 ± 0.0030.195 ± 0.048––

ATE in metres after loop closure, mean ± sd over three sessions (noon, evening, dusk) of one seed and one trajectory. On a phone the SGBM and chip columns are hidden.

Notes on this table: sessions, exclusions, the chip column

The spread is lighting plus render nondeterminism, not seeds or routes. The commands row is the front end at noon; 0.161 once closures matched on exact-depth scans are added (2 of 2 false). The lidar ring is the same renderer's depth. † dusk excluded: SGBM gave no usable scan and the arm fell back to the commands; its two remaining sessions are listed (noon, evening). The simulated car obeys its commands without slip, so every camera arm shows how much a depth source corrupts a near-exact prior. The network has no true scale, hence no entry. The last two columns are the same network: its int8 output from the Pi's Hailo-8L on the same frames agrees with the float twin at a median r 0.9999 (noon), 0.9998 (evening) and 0.9988 (dusk) per frame, but the floor-fit scale the SLAM takes from it differs by a median −1.4 to +0.6 % (dusk 5–95 %: −8 to +5 %), which moves the per-session ATE by a median 0.13 m (up to 0.48 m, 12 pairs): this SLAM turns scale differences of a few per cent into tenths of a metre, so gaps between camera arms smaller than that are not resolved.

Exact depth needs a better scale than its per-frame floor fit; for our network a better scale is not enough: the true scale cuts exact depth's error 6.8×, and 6.5× (95 % CI 5.5–7.7) on seven further start seeds, and smoothing the floor fit does not recover it; for our network no affine scale tried closes the gap.

The numbers behind this: per session, the front end, the scale error

True scale, on more seeds and lighting. Giving exact depth its true scale instead of the per-frame floor-fit scale from camera height cuts the error 6.8× on the original loop (6.3–7.4× per session; 3.1× for the front end alone). On seven further start seeds, fixed in advance, it is 6.5× (95 % CI 5.5–7.7): 0.620 to 0.094 m, every seed lower; three seeds excluded (one displaced at spawn, two with blank depth renders). Re-rendered at evening and dusk (the same drives; exact depth ignores lighting), the true scale lowers the error in every session: evening 6.8× (5.7–8.1) and dusk 6.2× (5.4–7.1). Driven the other way round (each place seen from the opposite direction; seven of ten seeds valid) it is 6.3× (95 % CI 5.7–6.9), lower in every seed.

Our network against exact depth. Under the same floor-fit scale our network's error over exact depth's is 0.75× at noon (95 % CI 0.49–1.14, at most 14 % worse), 0.66× at evening (0.57–0.78, lower) and 0.97× at dusk (0.70–1.36, at most 36 % worse), and 0.98× driven the other way (95 % CI 0.68–1.42, at most 42 % worse): never distinguishably worse. For exact depth the bottleneck is the scale; for our network, fitting the scale to exact depth never closes the gap.

Fitting the scale to exact depth. Fitted per frame (scale and shift) over the whole image, the network's error falls only 1.5× (95 % CI 1.1–2.3; exact depth's falls 6.5×): at noon, on the terrain it was fine-tuned on, it stays 3.2× exact depth's true-scale error (95 % CI 2.4–4.2), at evening and dusk 8.1× (5.6–11.6) and 9.0× (7.8–10.4), driven the other way 7.0× (5.9–8.3). Fitted on the scan's own rows instead, the network's error stays 5.3× at noon (95 % CI 4.1–6.7), and at evening and dusk the network's own floor fit beats both fits. None of the three affine scales tried (the floor fit, the two fits to exact depth) closes the gap. The closer fit on the scan's rows (rms 0.09–0.12 m−1, against 0.19–0.23 image-wide) is no better, and at noon and dusk worse, so this SLAM's error does not track how well the scale fits: beyond any per-frame affine scale tried, what limits it is the network's depth itself, not its scale (its disparity is also not affine to exact inverse depth, 1/Z).

Smoothing and other scale sources. Smoothing the floor-fit scale in time does not recover the true scale's gain: on exact depth the fit puts a true 2 m at a median 1.07×, and at 1–2 m that error drifts slowly along the route rather than jittering, so a filter cannot remove it (slow drift is one explanation; the fit's distance-dependent bias is another, untested). Every camera arm except true-scale exact depth is worse than the commands alone. Of the ways to get a correct scale, only stereo was tested here; an IMU or a metric model is untested.

A stretch of the same park as a 3D Gaussian splat, the camera rising under the trees: trained on the colour images with the RealSense depth as a geometry prior within 3 m. A picture of the place, not part of the comparison; its floaters and smeared trunks are the method's real output.
Method details, calibration, corrections and recorder defects
  • The depth reference. The three matchers (the D435's chip, OpenCV SGBM, Fast-FoundationStereo) share the images and the rectification, so a common-mode error there would not show. Fast-FoundationStereo is distilled from a backbone with Depth Anything V2 features, but all three must agree, so it can only veto a point; the set rebuilt without it is above. A stereo matcher is scored only against the other two, on its own points in the infrared frame with its holes excluded, so its 1–4 % cannot be set against the networks' numbers. The points are where stereo works by construction; sky, glare and objects closer than its minimum range are not in the set.
  • The factory calibration changed the depth numbers. A first version placed the points with the datasheet's nominal calibration, which misplaces every point by a median 14.5 px. The factory values, read from the camera after the drive, put depth edges 6.7 px from the colour image's edges against 12.0 px (closer on all 20 frames that could be scored). On the same physical points they lower the error of 17 of the 19 networks by 10–47 % (Depth Anything V2 by 44–47 %, ours by 24 %, the zoo model by 13 %; SC-DepthV3 and FastDepth barely change) while the stereo scores do not move. The nominal results are kept in the repository as a record.
  • Two corrections, published the same day. The first version of this section compared against our own stack fed RealSense depth, which shares our odometry, so its agreement was not evidence. It also projected several SLAM systems onto the wrong plane (they report the camera's optical frame), which made them look collapsed. Both are fixed above.
  • Three recorder defects, found and worked around. The capture tool's text header byte-shifted every stereo frame, recovered exactly from the lossless files. The RealSense's stereo timestamps drift about 8 % against real time, so frames are paired by image content (residual median 12.5 ms, p90 31.7 ms); 49 of 388 benchmark frames are excluded (26 flagged sync-suspect, 23 without an infrared frame). A power cut lost the last seconds of every stream.
  • Baselines were checked first. Most reproduced their published error on TUM fr1/desk before the park drive; Glob3R and ORB-SLAM3 stereo were not checked there (fr1 has no stereo pair), and monocular ORB-SLAM3 failed 1 of 4 runs. FastDepth's outputs on the chip are poor outdoors (an indoor-trained model).

Citation

BibTeX

@misc{itaim2026seeing,
  title  = {Seeing to Drive on a Single Board: Attention-Free Depth for Onboard Robot Navigation},
  author = {Mizlish, Itai and Ben-Moshe, Boaz},
  year   = {2026},
  url    = {https://itaim18.github.io/seeing-to-drive/}
}

Built on Depth Anything V2 and Depth Anything 3, the Hailo Model Zoo, NVIDIA Isaac Sim / Isaac Lab and its neural-reconstruction rendering. Weights and data CC-BY-NC-4.0 (the teacher's licence).