High spatiotemporal resolution can offer high precision for vision applications, which is particularly useful to capture the nuances of visual features, such as for augmented reality. Unfortunately, capturing and processing high spatiotemporal visual frames generates energy-expensive memory traffic. On the other hand, low resolution frames can reduce pixel memory throughput, but also reduce the opportunities of high-precision visual sensing. However, our intuition is that not all parts of the scene need to be captured at a uniform resolution. Selectively and opportunistically reducing resolution for different regions of image frames can yield high-precision visual computing at energy-efficient memory data rates.

To this end, we develop a visual sensing pipeline architecture that flexibly allows application developers to dynamically adapt the spatial resolution and update rate of different “rhythmic pixel regions” in the scene. We develop a system that ingests pixel streams from commercial image sensors with their standard raster-scan pixel read-out patterns, but only encodes relevant pixels prior to storing them in the memory. We also present streaming hardware to decode the stored rhythmic pixel region stream into traditional frame-based representations to feed into standard computer vision algorithms.

We integrate our encoding and decoding hardware modules into existing video pipelines around Xilinx's ZCU102. On top of this, we develop runtime support allowing developers to flexibly specify the region labels. With rhythmic pixel regions, the system can achieve 50% of 4K throughput, while maintaining around the same accuracy as 4K.

Venkatesh Kodukula, Alexander Shearer, Van Nguyen, Srinivas Lingutla, Robert LiKamWa. 2021. Rhythmic pixel regions: Multi-resolution visual sensing system towards high-precision visual computing at low power ASPLOS '21 Proceedings of the 26th ACM International Conference on Architectural Support for Programming