Snapping a tap to a real corner: building a LiDAR tape measure for iPhone
When you tap the screen to start a measurement, you are not tapping the corner of the table. You are tapping a pixel, and that pixel sits on a ray that leaves the camera and hits something . Often the something is the floor 40 inches behind the table, because your thumb landed a hair past the edge. This post is how the tape measure in our iPhone app deals with that: two kinds of evidence (depth…
When tapping the screen to begin a measurement, users are not tapping the table's corner; they are tapping a pixel, which lies on a ray that strikes something. Usually, this "something" is the floor 40 inches behind the table due to the user's tap being slightly off the table's edge. This post explains how the tape measure within our iPhone application handles this situation: employing two sources of information (depth and image), some geometric calculations, and a filter that prevents the point from fluctuating.
Depth alone proves insufficient as it is too imprecise. On a Pro iPhone, ARKit provides a depth map for every frame (sceneDepth) and an additional smoothedSceneDepth if the device supports it. However, the depth map is coarse, with a reconstructed mesh of a few centimeters per triangle, which smooths over the exact measurements we aim to obtain: the edges.
The snapper simultaneously considers both the depth information and the camera image, within a narrow window surrounding the reticle, for each frame. The process begins with identifying surfaces in a window of depth. The depth samples around the reticle are back-projected into world space and searched for up to three planes using RANSAC.
Several factors matter more than anticipated in this step: Inliers within 3 mm (0.4% of the distance) are considered valid, while LiDAR noise is disregarded. A larger threshold is used for greater distances, preventing walls from being split into slabs at significant intervals. Near-parallel slabs are merged, addressing potential inaccuracies caused by LiDAR noise.
The median distance is selected for each plane, rather than sorting them, using a Quickselect algorithm with a median-of-three pivot for efficiency. A seeded random generator is implemented to ensure RANSAC's randomness, allowing the same depth window to consistently yield the same planes. The plane fitting is performed using a straightforward least squares method, calculating the normal vector as the eigenvector of a 3x3 covariance matrix using Jacobi sweeps.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.