The Off-Key Bard
Technology

How Does Your Phone Know How Far Away Something Is?

Or: how a flat camera builds a 3D map of the world. A photograph is blind to distance. Your phone restores it by cross-examining light, lenses, and motion.

Or: How a Flat Camera Builds a 3D Map of the World

Point a phone at a person and the camera locks focus onto their face. Switch to Portrait Mode and the background dissolves into a soft blur while the subject stays crisp. Open an augmented-reality app and a virtual couch settles convincingly onto the actual floor, staying anchored there as you walk around the room. None of these features come from an ordinary photograph, because an ordinary photograph doesn't contain distance. A camera sensor records brightness and color at every pixel, and a single pixel tells you nothing about how far away the object that produced it actually was. A pedestrian twenty feet away and a building two hundred feet away both end up as pixels on the exact same flat surface. Depth has to be reconstructed — inferred, measured, or estimated — because the camera itself never captured it in the first place.

Focus Was Already a Distance Problem

Long before Portrait Mode or AR existed, cameras needed a rough sense of distance just to focus. A lens has to sit at a specific position to form a sharp image of a subject at a given distance, and different distances require different lens positions. Older systems solved this by brute-force trial and error, called contrast-detection autofocus: move the lens, check whether the image got sharper or blurrier, and keep adjusting until sharpness peaks. It works, but it can require searching back and forth, which is why older cameras sometimes visibly "hunted" for focus.

A faster technique, phase-detection autofocus, compares light arriving at the sensor from different parts of the lens. If a subject is out of focus, those two light paths don't align, and critically, the size and direction of that misalignment tells the system not just that focus is wrong, but which way to move the lens to fix it — no guessing required. Many modern phone sensors build this capability directly into their pixel structure, so the camera effectively gathers rough distance information as a side effect of simply trying to focus, well before any dedicated depth feature gets involved.

Two Eyes, or Two Cameras, See the World Differently

The far more direct way to estimate depth borrows straight from human vision. Close one eye, then switch to the other, and a nearby object visibly jumps relative to the background — your eyes sit a few centimeters apart, so each one sees the world from a slightly different vantage point, and your brain uses that difference as a built-in depth cue. Phones with two or more rear cameras exploit the identical principle. Each camera sees the same scene from a marginally different position, and the amount a given object shifts between those two images — called disparity — depends directly on how far away it is. A nearby face shifts a lot between the two camera views; a distant mountain barely shifts at all. Roughly speaking, more disparity means closer, less disparity means farther, and if the system knows the exact physical distance between its lenses, that shift can be converted into an actual depth estimate through fairly straightforward geometry.

This approach has real limits, though. As a subject gets farther away, the disparity between the two camera views shrinks until it becomes too small to measure reliably, which is why stereo depth tends to work best at near and medium range and gets shakier at a distance. It also needs something to actually compare: a plain white wall offers almost no distinguishing detail, so the software can't confidently tell which patch in one image corresponds to which patch in the other. A detailed, textured surface gives the matching algorithm something to grab onto; a featureless one doesn't.

Sometimes One Camera Is Enough — If It Moves

Stereo depth doesn't strictly require two physical cameras running at once. Close one eye and sway your head gently side to side, and nearby objects still appear to shift relative to distant ones — a phenomenon called motion parallax. A single moving camera can exploit exactly the same effect, comparing frames captured from slightly different positions as the device itself moves through space to infer the three-dimensional structure of a scene, an approach sometimes called structure from motion. Google's own documentation for ARCore's Depth API describes essentially this technique in production: the system compares multiple images taken from different angles as a person naturally moves their phone, using the shift between frames to build a depth estimate for every pixel, while also leaning on machine learning to help when movement is minimal. It also notes the practical tradeoffs plainly — depth is only valid once the device has actually started moving, low-texture surfaces like blank walls still produce imprecise results, and accuracy tends to be best in the roughly half-meter-to-five-meter range. This is exactly why an AR app will sometimes prompt you to "move your phone around" before placing an object — the system genuinely needs several different viewpoints before it can estimate the room's geometry with any confidence.

Actively Measuring Distance With Light

Some phones skip inference entirely and measure distance directly, using a method broadly called time-of-flight sensing: the device emits a pulse of infrared light, waits for it to bounce off a surface and return, and calculates distance from how long that round trip took, since light travels at a known, constant speed. The timing involved is almost absurdly precise — light covers roughly a foot in about a nanosecond — which is why practical consumer depth sensors rely on specialized electronics to extract usable distance data out of such minuscule intervals. NOAA's overview of lidar, the same general family of ranging technology used for large-scale terrain mapping, describes the underlying principle in its own context: a pulsed laser is sent out, the light that bounces back off a surface is captured by a sensor, and that returning signal becomes the basis for a range measurement. Phones apply the identical physical idea at drastically shorter range and finer resolution, usually labeled LiDAR or simply time-of-flight, and it carries a real practical advantage — because the system supplies its own infrared illumination rather than depending on ambient light, it keeps working reasonably well in low-light conditions where ordinary camera-based autofocus and depth estimation both struggle.

A related but distinct active technique, structured light, projects a known pattern — often a dense grid of infrared dots — onto a subject and measures how that pattern visibly distorts across bumps, curves, and contours, the same way a projected grid would warp noticeably across someone's nose, cheekbones, and brow. Apple's own explanation of how Face ID works describes almost exactly this process applied to short-range facial recognition: the TrueDepth camera projects and analyzes thousands of invisible dots to build an actual depth map of the user's face, captures a complementary infrared image, and feeds both into a neural engine that compares the result against the enrolled face data. Crucially, the system always requires genuine depth information — a flat photograph, however detailed, simply doesn't contain it, which is precisely why printing out a picture of someone's face can't fool a system built this way, even if it might momentarily fool a careless human glance.

Building a Believable Augmented-Reality Illusion

A depth map unlocks far more than Portrait Mode's background blur — it's also the reason augmented reality manages to look convincing instead of obviously fake. Place a virtual chair on your floor without any depth information, and the renderer has no way to know that your real coffee table sits physically closer to the camera than the chair does; the virtual object would simply draw on top of everything, floating unnaturally in front of furniture it should be partially hidden behind. This problem is called occlusion, and humans are remarkably sensitive to getting it wrong — the instant a virtual object fails to tuck correctly behind a real one, the illusion collapses immediately, however good the rendering otherwise looks. Google's Depth API documentation lists occlusion explicitly among its core use cases: compare the real-world depth at a given pixel against the virtual object's intended depth at that same pixel, and whichever is actually closer gets drawn in front. The same depth data also supports more subtle effects the Depth API documentation mentions — virtual snow settling realistically across real furniture, fog rendered at the correct distance, or simulated lighting interacting with real surfaces — all of which depend on the renderer actually knowing the three-dimensional shape of the room it's compositing into.

Maintaining that illusion as someone walks around a room, rather than just for one static shot, requires the device to continuously track its own position and build an understanding of the surrounding space at the same time — a classic robotics problem known as SLAM, simultaneous localization and mapping. Wikipedia's overview of the concept describes the core difficulty directly: a device needs an accurate map to know where it is, but it needs to know where it is in order to build that map correctly, so the two problems get solved together, typically using probabilistic methods that let each new sensor reading refine both estimates at once. Cameras contribute recognizable visual landmarks — the corner of a television, the edge of a window — and as the device moves, tracking how those landmarks shift across frames reveals both the device's own motion and the surrounding geometry; recognizing a landmark seen earlier lets the system correct for any small errors that crept in since then, a process called loop closure. This is the same underlying idea whether it's a phone keeping a virtual couch anchored to real carpet, a vacuum robot building a map of an apartment it's never seen, or a drone avoiding a wall it's approaching — track yourself, map the space, and keep refining both as new information arrives.

Nothing Here Works Alone

What ties every one of these techniques together is that none of them is treated as sufficient by itself. Stereo disparity struggles with distant or textureless surfaces; active infrared sensors can get overwhelmed by strong sunlight or confused by mirrors and glass, which redirect light in ways the system doesn't expect; motion-based depth needs the device to actually be moving; and a single still image, estimated through machine learning alone, can occasionally be fooled by a cleverly staged scene the same way a human eye can be fooled by forced-perspective photography. So phones combine evidence rather than trusting any one source. A coarse time-of-flight measurement can anchor absolute distance while the high-resolution camera image sharpens the exact boundary of an object's edge. A rough machine-learning depth guess can be corrected the moment an active sensor or stereo pair provides a conflicting, more reliable reading. None of the individual methods needs to be perfect, because the system was never designed to rely on just one.

The Bard's Take

A camera genuinely cannot see distance — it only ever records the color and brightness of light striking a flat sensor, and every depth feature on a phone exists specifically to compensate for that blindness. Sometimes the fix comes from comparing two slightly different viewpoints, the same trick two eyes use automatically. Sometimes it comes from watching how a scene shifts as the device itself moves through space. Sometimes the phone actively sends light out into the world and times its return, trading inference for direct physical measurement. And sometimes a trained model simply makes an extremely well-informed guess based on how three-dimensional scenes usually look when flattened into two dimensions.

No single one of those approaches is reliable in every situation, which is exactly why a modern phone never leans on just one. It cross-examines several imperfect sources of evidence — stereo geometry, active light timing, structured infrared patterns, motion over time, and learned visual expectation — and lets software reconcile whatever they collectively suggest. The couch that measures out to eight feet away, the virtual chair that stays anchored to your actual floor, the background that blurs convincingly behind a sharp subject — none of it comes from the camera "seeing" depth directly. It comes from a phone building a temporary, surprisingly sophisticated three-dimensional model of the room, out of clues that were never three-dimensional to begin with.

Sources

← Back to the Desk — Technology