Brown engineers build brain-inspired camera for low-visibility environments

Drawing power measured in only tens of milliwatts, Engineers at Brown University have built a brain-inspired imaging system that uses a dynamic vision sensor and a deep spiking neural network to reconstruct images and track moving objects through thick fog or turbid water.

The technology draws inspiration from the human visual system.

Mirroring Retina with Brain-Inspired Hardware

The biological retina focuses on changes in the visual field—such as edges, motion, and flicker—and sends neural spikes to the brain to reconstruct the world over time, rather than capturing a full picture all at once like a video camera. The Brown University research team mirrored this architecture by pairing a dynamic vision sensor with a deep spiking neural network.

The dynamic vision sensor detects light-intensity changes in individual pixels. A pixel fires only when its brightness changes past a set threshold, letting the sensor spot moving targets even when scattering media like fog block most of the light. The human eye is quite good at motion detection, but there are ways in which a camera, specifically a dynamic vision sensor, can be better, including that it’s sensitive to near-infrared light, which the human eye cannot see, said Arto Nurmikko, a professor of engineering at Brown and the study’s senior author, as reported by Brown University.

The scattering background of fog changes much more slowly than a moving target. The sensor largely ignores the fog and focuses on light that has interacted with the object. The raw output is recorded as a cloud of spikes, which the deep spiking neural network then processes to reconstruct images of randomly moving, normally unrecognizable objects and track their trajectories in fractions of a second.

Testing Objects Through Fog Chambers and Water Tanks

Researchers projected moving alpha-numeric characters and silhouettes of various bird species through an obscuring fog chamber and a tank of turbid water to put the system to the test. The experiments demonstrated that the setup could successfully image and track randomly moving objects that remained entirely hidden from standard camera systems and human observers.

The system also proves notably energy-efficient. The sensor draws only tens of milliwatts, while the spiking network consumes a fraction of the energy required by a conventional neural network because it computes solely on sparse spikes rather than dense video frames.

Current Limitations and Potential Real-World Uses

An object must be in motion relative to the camera to be detected because the sensor relies on detecting changes, meaning the approach does have specific operational limits. The current system recovers an object’s silhouette rather than a full grayscale image, and the sensor loses sensitivity under very dim lighting conditions.

The research team is investigating a light-intensifier front end for low-light environments to address these constraints. They are also developing depth-resolved approaches to extend the method into three-dimensional targets by measuring light travel time or by pairing two event cameras to mimic a pair of eyes.

The research received financial support from the Office of Naval Research under grant N00014-25-1-2061, Intel Laboratories under grant CG70982727 2021, and the National Science Foundation under grant 232600. According to Zhang, a researcher involved with the project, This could be useful anywhere where light-scattering by the surrounding medium is a serious problem — self-driving cars, search-and-rescue drones and underwater navigation are a few examples.